Hypothetical Document Embeddings
如果我们正在使用基于相似度搜索的索引，比如矢量存储，那么直接在原始问题上进行搜索可能效果不佳，因为它们的嵌入可能与相关文档的嵌入不够相似。相反，让模型生成一个假想的相关文档，然后利用该文档进行相似度搜索可能会有所帮助。这就是虚拟文档嵌入（Hypothetical Document Embedding，或者称为HyDE）背后的关键思想。
让我们看看如何通过虚拟文件在 LangChain 的 YouTube 视频中为我们的问答机器人执行搜索。
Setup
Install dependencies
# %pip install -qU langchain langchain-openai
Set environment variables
我们在这个例子中将使用 OpenAI。
import getpass
import os
os.environ["OPENAI_API_KEY"] = getpass.getpass()
# Optional, uncomment to trace runs with LangSmith. Sign up here: https://smith.langchain.com.
# os.environ["LANGCHAIN_TRACING_V2"] = "true"
# os.environ["LANGCHAIN_API_KEY"] = getpass.getpass()
Hypothetical document generation
最终生成相关的假设性文档归结为试图回答用户问题。 因为我们正在为LangChain YouTube 视频设计一个问答机器人，我们将提供一些关于LangChain 的基本背景，并提示模型使用更加学究风格，以便获得更真实的假设性文档。
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
system = """You are an expert about a set of software for building LLM-powered applications called LangChain, LangGraph, LangServe, and LangSmith.
LangChain is a Python framework that provides a large set of integrations that can easily be composed to build LLM applications.
LangGraph is a Python package built on top of LangChain that makes it easy to build stateful, multi-actor LLM applications.
LangServe is a Python package built on top of LangChain that makes it easy to deploy a LangChain application as a REST API.
LangSmith is a platform that makes it easy to trace and test LLM applications.
Answer the user question as best you can. Answer as though you were writing a tutorial that addressed the user question."""
prompt = ChatPromptTemplate.from_messages(
[
("system", system),
("human", "{question}"),
]
)
llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0)
qa_no_context = prompt | llm | StrOutputParser()
answer = qa_no_context.invoke(
{
"question": "how to use multi-modal models in a chain and turn chain into a rest api"
}
)
print(answer)
To use multi-modal models in a chain and turn the chain into a REST API, you can leverage the capabilities of LangChain, LangGraph, and LangServe. Here's a step-by-step guide on how to achieve this:
1. **Building a Multi-Modal Model with LangChain**:
- Start by defining your multi-modal model using LangChain. LangChain provides integrations with various deep learning frameworks like TensorFlow, PyTorch, and Hugging Face Transformers, making it easy to incorporate different modalities such as text, images, and audio.
- You can create separate components for each modality and then combine them in a chain to build a multi-modal model.
2. **Building a Stateful, Multi-Actor Application with LangGraph**:
- Once you have your multi-modal model defined in LangChain, you can use LangGraph to build a stateful, multi-actor application around it.
- LangGraph allows you to define actors that interact with each other and maintain state, which is useful for handling multi-modal inputs and outputs in a chain.
3. **Deploying the Chain as a REST API with LangServe**:
- After building your multi-modal model and application using LangChain and LangGraph, you can deploy the chain as a REST API using LangServe.
- LangServe simplifies the process of exposing your LangChain application as a REST API, allowing you to easily interact with your multi-modal model through HTTP requests.
4. **Testing and Tracing with LangSmith**:
- To ensure the reliability and performance of your multi-modal model and REST API, you can use LangSmith for testing and tracing.
- LangSmith provides tools for tracing the execution of your LLM applications and running tests to validate their functionality.
By following these steps and leveraging the capabilities of LangChain, LangGraph, LangServe, and LangSmith, you can effectively use multi-modal models in a chain and turn the chain into a REST API.
Returning the hypothetical document and original question
为了增加我们的召回率，我们可能希望根据假设文档和原始问题来检索文档。我们可以这样轻松地同时返回两者：
from langchain_core.runnables import RunnablePassthrough
hyde_chain = RunnablePassthrough.assign(hypothetical_document=qa_no_context)
hyde_chain.invoke(
{
"question": "how to use multi-modal models in a chain and turn chain into a rest api"
}
)
{'question': 'how to use multi-modal models in a chain and turn chain into a rest api',
'hypothetical_document': "To use multi-modal models in a chain and turn the chain into a REST API, you can leverage the capabilities of LangChain, LangGraph, and LangServe. Here's a step-by-step guide on how to achieve this:\n\n1. **Set up your multi-modal models**: First, you need to create or import your multi-modal models. These models can include text, image, audio, or any other type of data that you want to process in your LLM application.\n\n2. **Build your LangGraph application**: Use LangGraph to build a stateful, multi-actor LLM application that incorporates your multi-modal models. LangGraph allows you to define the flow of data and interactions between different components of your application.\n\n3. **Integrate your models in LangChain**: LangChain provides integrations for various types of models and data sources. You can easily integrate your multi-modal models into your LangGraph application using LangChain's capabilities.\n\n4. **Deploy your LangChain application as a REST API using LangServe**: Once you have built your multi-modal LLM application using LangGraph and LangChain, you can deploy it as a REST API using LangServe. LangServe simplifies the process of exposing your LangChain application as a web service, making it accessible to other applications and users.\n\n5. **Test and trace your application using LangSmith**: Finally, you can use LangSmith to trace and test your multi-modal LLM application. LangSmith provides tools for monitoring the performance of your application, debugging any issues, and ensuring that it functions as expected.\n\nBy following these steps and leveraging the capabilities of LangChain, LangGraph, LangServe, and LangSmith, you can effectively use multi-modal models in a chain and turn the chain into a REST API."}
Using function-calling to get structured output
如果我们将这种技术与其他查询分析技术结合起来，我们可能会使用函数调用来获取结构化查询对象。我们可以像这样为HyDE使用函数调用：
from langchain_core.output_parsers.openai_tools import PydanticToolsParser
from langchain_core.pydantic_v1 import BaseModel, Field
class Query(BaseModel):
answer: str = Field(
...,
description="Answer the user question as best you can. Answer as though you were writing a tutorial that addressed the user question.",
)
system = """You are an expert about a set of software for building LLM-powered applications called LangChain, LangGraph, LangServe, and LangSmith.
LangChain is a Python framework that provides a large set of integrations that can easily be composed to build LLM applications.
LangGraph is a Python package built on top of LangChain that makes it easy to build stateful, multi-actor LLM applications.
LangServe is a Python package built on top of LangChain that makes it easy to deploy a LangChain application as a REST API.
LangSmith is a platform that makes it easy to trace and test LLM applications."""
prompt = ChatPromptTemplate.from_messages(
[
("system", system),
("human", "{question}"),
]
)
llm_with_tools = llm.bind_tools([Query])
hyde_chain = prompt | llm_with_tools | PydanticToolsParser(tools=[Query])
hyde_chain.invoke(
{
"question": "how to use multi-modal models in a chain and turn chain into a rest api"
}
)
[Query(answer='To use multi-modal models in a chain and turn the chain into a REST API, you can follow these steps:\n\n1. Use LangChain to build your multi-modal model by integrating different modalities such as text, image, and audio.\n2. Utilize LangGraph, a Python package built on top of LangChain, to create a stateful, multi-actor LLM application that can handle interactions between different modalities.\n3. Once your multi-modal model is built using LangChain and LangGraph, you can deploy it as a REST API using LangServe, another Python package that simplifies the process of creating REST APIs from LangChain applications.\n4. Use LangSmith to trace and test your multi-modal model to ensure its functionality and performance.\n\nBy following these steps, you can effectively use multi-modal models in a chain and turn the chain into a REST API.')]