翻訳待ち：Timescale Vector x LangChain: Making PostgreSQL A Better Vector Database for AI Applications

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Partner Timescale Vector x LangChain: Making PostgreSQL A Better Vector Database for AI Applications September 24, 2023 15 min Go back to blog Create agents Editor's Note: This post was written in collaboration with the Timescale Vector team. Their integration with LangChain supports PostgreSQL as your vector database for faster similarity search, time-based context retrieval for RAG, and self-querying capabilities. And they're offering a free 90 day trial! Introducing the Timescale Vector integration for LangChain. Timescale Vector enables LangChain developers to build better AI applications with PostgreSQL as their vector database: with faster vector similarity search, efficient time-based search filtering, and the operational simplicity of a single, easy-to-use cloud PostgreSQL database for not only vector embeddings, but an AI application’s relational and time-series data too. PostgreSQL is the world’s most loved database, according to the Stack Overflow 2023 Developer Survey. And for a good reason: it’s been battle-hardened by production use for over three decades, it’s robust and reliable, and it has a rich ecosystem of tools, drivers, and connectors. And while pgvector, the open-source extension for vector data on PostgreSQL, is a wonderful extension (and is offered as part of Timescale Vector), it is just one piece of the puzzle in providing a production-grade experience for AI application developers on PostgreSQL. After speaking with numerous developers at nimble startups and established industry giants, we saw the need to enhance pgvector to cater to the performance and operational needs of developers building AI applications. Here’s the TL;DR on how Timescale Vector helps you build better AI applications with LangChain: Faster similarity search on millions of vectors: Thanks to the introduction of a new search index inspired by the DiskANN algorithm, Timescale Vector achieves 243% faster search speed at ~99 % recall than Weaviate, a specialized database, and outperforms all existing PostgreSQL search indexes by between 39.39% and 1,590.33% on a dataset of one million OpenAI embeddings. Plus, enabling product quantization yields a 10x index space savings compared to pgvector. Timescale Vector also offers pgvector’s Hierarchical Navigable Small Worlds (HNSW) and Inverted File Flat (IVFFlat) indexing algorithms. Similarity search with efficient time-based filtering: Timescale Vector optimizes time-based vector search, leveraging the automatic time-based partitioning and indexing of Timescale’s hypertables to efficiently find recent embeddings, constrain vector search by a time range or document age, and store and retrieve large language model (LLM) response and chat history with ease. Time-based semantic search also enables you to use Retrieval Augmented Generation (RAG) with time-based context retrieval to give users more useful LLM responses. Simplified AI infra stack: By combining vector embeddings, relational data, and time-series data in one PostgreSQL database, Timescale Vector eliminates the operational complexity that comes with managing multiple database systems at scale. Simplified metadata handling and multi-attribute filtering: You can leverage all PostgreSQL data types to store and filter metadata, and JOIN vector search results with relational data for more contextually relevant responses. In future releases, Timescale Vector will also support rich multi-attribute filtering, enabling even faster similarity searches when filtering on metadata. On top of these innovations for vector workloads, Timescale Vector provides a robust, production-ready PostgreSQL platform with flexible pricing, enterprise-grade security, and free expert support. In the rest of this post, we’ll dive deeper (with code!) into the unique capabilities Timescale Vector enables for developers wanting to use PostgreSQL as their vector database with LangChain: Faster similarity search with DiskANN, HNSW and IVFFlat index types. Efficient similarity search when filtering vectors by time. Retrieval Augmented Generation (RAG) with time-based context retrieval. Advanced self-querying capabilities. (If you’d prefer to jump into the code, explore this tutorial). 🎉 LangChain Users Get 3 Months Free of Timescale Vector Timescale is giving LangChain users an extended 90-day trial of Timescale Vector. This makes it easy to test and develop your applications with Timescale Vector, as you won’t be charged for any cloud PostgreSQL databases you spin up during your trial period. Try Timescale Vector for free today. Faster Vector Similarity Search in PostgreSQL Timescale Vector speeds up Approximate Nearest Neighbor (ANN) search on large scale vector datasets, enhancing pgvector with a state-of-the-art ANN index inspired by the DiskANN algorithm. Timescale Vector also offers pgvector’s HNSW and IVFFlat indexing algorithms as well, giving developers the flexibility to choose the right index for their use case. Our performance benchmarks using the ANN benchmarks suite show that Timescale Vector achieves between 39.43% and 1,590.33% faster search speed at ~99 % recall than all existing PostgreSQL search indexes and 243.77% faster search speed than specialized vector databases like Weaviate, on a dataset of one million OpenAI embeddings. You can read more about the performance benchmark methodology and results here. Caption: Timescale Vector’s new index outperforms specialized vector database Weaviate by 243% and all existing PostgreSQL index types when performing approximate nearest neighbor searches at 99% recall on 1 million OpenAI embeddings. Using Timescale Vector’s DiskANN, HNSW, or IVFFLAT indexes in LangChain is incredibly straightforward. Simply create a Timescale Vector vector store as shown below: from langchain.vectorstores.timescalevector import TimescaleVector # Create a Timescale Vector instance from the collection of documents db = TimescaleVector.from_documents( embedding=embeddings, documents=docs, collection_name=COLLECTION_NAME, service_url=SERVICE_URL, ) And then run: # create an index # by default this will create a Timescale Vector (DiskANN) index db.create_index() This will create a timescale-vector index with the default parameters. We should point out that the term “index” is a bit overloaded. For many vector databases, an index is the thing that stores your data (in relational databases this is often called a table), but in the PostgreSQL world an index is something that speeds up search, and we are using the latter meaning here. We can also specify the exact parameters for index creation in the create_index command as follows: # create an timescale vector index (DiskANN) with specified parameters db.create_index(index_type="tsv", max_alpha=1.0, num_neighbors=50) Advantages to this Timescale Vector’s new DiskANN-inspired vector search index include the following: Faster vector search at high accuracy in PostgreSQL. Optimized for running on disks, not only in memory use. Quantization optimization compatible with PostgreSQL, reducing the vector size and consequently shrinking the index size (by 10x in some cases!) and expediting searches. Efficient hybrid search or filtering additional dimensions. For more on DiskANN and how Timescale Vector’s new index works, see this blog post. Pgvector is packaged as part of Timescale Vector, so you can also access pgvector’s HNSW and IVFFLAT indexing algorithms in your LangChain applications. The ability to conveniently create database indexes from your LangChain application code makes it easy to create different indexes and compare their performance. # Create an HNSW index. # Note: you don't need to specify m and ef_construction parameters as we set smart defaults. db.create_index(index_type="hnsw", m=16, ef_construction=64) # Create an IVFFLAT index # Note:you don't need to specify num_lists and num_records parameters as we set smart defaults. db.create_index(index_type="ivfflat", num_lists=20, num_records=1000) Add Efficient Time-Based Search Functionality to Your LangChain AI Application Timescale Vector optimizes time-based vector search, leveraging the automatic time-based partitioning and indexing of Timescale’s hypertables to efficiently search vectors by time and similarity. Time is often an important metadata component for vector embeddings. Sources of embeddings, like documents, images, and web pages, often have a timestamp associated with them, for example, their creation date, publishing date, or the date they were last updated, to name but a few. We can take advantage of this time metadata in our collections of vector embeddings to enrich the quality and applicability of search results by retrieving vectors that are not just semantically similar but also pertinent to a specific time frame. Here are some examples where time-based retrieval of vectors can improve your LangChain applications: Chat history: Storing and retrieving LLM response history. For example, chatbot chat history. Finding recent embeddings: Finding the most recent embeddings similar to a query vector. For example, finding the most recent news, documents, or social media posts related to elections. Search within a time range: Constraining similarity search to only vectors within a relevant time range. For example, asking time-based questions about a knowledge base (“What new features were added between January and March 2023?”). Let’s look at an example of performing time-based searches on a git log dataset. In a git log, each entry has a timestamp, an author, and some information about the commit. To illustrate how to use TimescaleVector's time-based vector search functionality, we'll ask questions about the git log history for TimescaleDB. Each git commit entry has a timestamp associated with it, as well as a message and other metadata (e.g., author). Load text and extract metadata First, we load in the git log using LangChain’s JSON Loader. # Load data from JSON file and extract metadata loader = JSONLoader( file_path=FILE_PATH, jq_schema='.commit_history[]', text_content=False, metadata_func=extract_metadata ) documents = loader.load() Notice how we provide a function named extract_metadata as an argument to the JSONLoader. This function enables us to store not just the contents of the JSON in vectorized form but metadata about an embedding. It is in this function that we’ll specify the timestamp of the git log entry to be used in our time-based vector search. Create time-based identifiers for Documents For time-based search in LangChain, Timescale Vector uses the ‘datetime’ portion of a UUID v1 to place vectors in the correct time partition. Timescale Vector’s Python client library provides a simple-to-use function named uuid_from_time to create a UUID v1 from a Python datetime object, which you can then pass to the Timescale Vector vector store constructor as we’ll see in the code snippet further down. Here’s how we use the uuid_from_time helper functions: from timescale_vector import client # Function to take in a date string in the past and return a uuid v1 def create_uuid(date_string: str): if date_string is None: return None time_format = '%a %b %d %H:%M:%S %Y %z' datetime_obj = datetime.strptime(date_string, time_format) uuid = client.uuid_from_time(datetime_obj) return str(uuid) Here’s the extract_metdata() function we pass to the JSONLoader specifying the fields we want in the metadata for each vector embedding in our vector collection: # Metadata extraction function to extract metadata from a JSON record def extract_metadata(record: dict, metadata: dict) -> dict: record_name, record_email = split_name(record["author"]) metadata["id"] = create_uuid(record["date"]) metadata["date"] = create_date(record["date"]) metadata["author_name"] = record_name metadata["author_email"] = record_email metada [truncated for AI cost control]