待翻译：Automating Web Research

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Key Links Web Researcher Repo New LangChain Retriever and Documentation Hosted Streamlit App Motivation Web research is one of the killer LLM applications: Greg Kamradt highlighted it as one of his top desired AI tools and OSS repos like gpt-researcher are growing in popularity. We decided to take a stab at it, initially setting out like many others to build a web research agent. But, we landed somewhere different: a fairly simple retriever proved to be effective and easily configurable (e.g., to run in private mode as popularized by projects like PrivateGPT) . In this blog we talk about our exploration and thought process, how we built it, and the next steps. Exploration Abovementioned projects like gpt-researcher and AI search engines (perplexity.ai) offer an early glimpse into how web research may be re-imagined. Like many, we first devised an agent that could be given a prompt, a set of tools, and then would set forth to scour the web autonomously! For this, it clearly needed tools to: Search and return pages Scrape the full content of the pages returned Extract relevant information from the pages With those tools, the agent could approximate what a human does: search a topic, choose selected links, skim the link for useful pieces of information, and return to the search in an iterative exploration. We made an agent, gave it these tools ... but found it slowly fumbled thought the iterative search process, much like a human! Improvements We noticed a central advantage that AIs can uniquely exploit: kick off many searches in parallel and, in turn, "read" many pages in parallel. Of course, this risks inefficiency if the first article in a sequential search has all of the necessary information. But for complex questions that warrant an AI researcher, this risk is somewhat mitigated. We added some basic tools to support this process. With a heap of information collected in parallel from a set of pages, it seemed reasonable to fetch the most relevant chunks from each page and load them into the context window of an LLM for synthesis. Of course, at this point we realized that our agent was morphing into a retriever! (NOTE: we still think that agentic properties can further benefit this retriever, as discussed at the end.) Retrieval What exactly would this retriever do under the hood? Our thinking was: Use an LLM to generate multiple relevant search queries (one LLM call) Execute a search for each query Choose the top K links per query (multiple search calls in parallel) Load the information from all chosen links (scrape pages in parallel) Index those documents into a vectorstore Find the most relevant documents for each original generated search query Collectively, these steps fall into the flow used for retrieval augmented generation: