AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。
HFresh is Weaviate's disk-based vector index for applications that prioritize lower memory use over peak query throughput. That tradeoff is useful for small applications with limited resources as well as large datasets. Weaviate Cloud's Free Tier, for example, uses HFresh by default through its Cost Optimized profile. Understanding HNSW's memory bottleneck in similarity search When it comes to finding similar vectors quickly, HNSW (Hierarchical Navigable Small World) has become the gold standard. It's fast and accurate, but as datasets grow from millions to billions of vectors, HNSW reveals a fundamental constraint: its graph and vector cache are kept in memory. HNSW is a graph-based index that organizes vectors into a hierarchical structure. At the top layer, you have a sparse graph with long-distance connections that help you quickly navigate to the right neighborhood. As you descend through the layers, the graphs become denser with more local connections, eventually guiding you to the most similar vectors at the bottom layer. The question isn't whether HNSW is good. It absolutely is. If you need the lowest possible latency and highest throughput, HNSW is hard to beat. But many applications prioritize lower memory use and larger scale over peak query performance. This is where disk-based indexes become interesting. What if you could trade some latency for much lower memory use and the ability to scale to larger datasets? Introducing HFresh HFresh is a modern disk-based vector index designed for high recall, strong update performance, and controlled query I/O at large scale. It builds on ideas introduced by the SPFresh research paper, adapting the design to use battle-tested components already present in Weaviate. At a high level, HFresh belongs to the family of partition-based vector indices. Instead of connecting every vector to neighbors in a global graph like HNSW, HFresh divides vectors into many small regions called postings. Each posting contains vectors that are close to each other in vector space and is stored on disk in an LSM store. To make this layout efficient, HFresh uses a two-stage search strategy. First, a compact in-memory centroid index identifies which regions of the vector space are relevant to a query. Then, only corresponding postings are fetched from disk and searched in detail. By limiting disk reads to a small subset of the dataset, HFresh is designed to keep I/O bounded and latency predictable, even as the dataset grows into billions. This structure is designed to support very large datasets while keeping performance predictable. Freshness without rebuilds The key idea behind SPFresh, and inherited by HFresh, is that most updates only affect a small region of the vector space. In traditional partition-based indexes, updates can accumulate and partitions can drift, eventually requiring a full rebuild to restore recall and latency, a process that can take hours or days at scale. SPFresh shows that this is often unnecessary. In a well-structured partitioned index, inserting or deleting a vector typically only affects a small neighborhood of the vector space. Instead of rebuilding everything, you can maintain index quality through incremental rebalancing, using a small set of local operations. Splitting oversized postings Merging undersized ones Reassigning vectors when boundaries shift These operations run mostly asynchronously in the background, continuously repairing small local imbalances before they accumulate into global problems. The result is an index that stays fresh and well-balanced over time, without disruptive rebuild cycles. How HFresh builds on SPFresh HFresh takes the core idea behind SPFresh and adapts it to fit Weaviate's architecture. The goal was not to reproduce the paper component by component, but to keep what makes the design so interesting: local maintenance instead of rebuilds controlled query I/O a clear separation between the in-memory routing layer and disk-based postings From there, we made a series of pragmatic choices. Instead of introducing entirely new ANN machinery, we reused battle-tested pieces already present in Weaviate and shaped them to serve this new layout. This means keeping the overall SPFresh philosophy, while rethinking some of its components to better match Weaviate's strengths around indexing, filtering, compression, and updates. HNSW as the centroid index A key design choice in HFresh is the use of HNSW as the centroid index, rather than SPTAG. SPTAG is the ANN index used in Microsoft's original SPANN design and carried over into SPFresh. It combines a partitioning tree with a graph, allowing queries to quickly navigate toward the closest centroids and, from there, the right posting lists. For Weaviate, however, HNSW was the more natural fit: it already plays that role very well. HNSW is the most widely used vector index in Weaviate and one of the most battle-tested parts of the system. We understand its behavior well in production, across a wide range of workloads and dataset sizes. Reusing it for centroid search lets HFresh build on infrastructure that is already proven, rather than introducing a brand-new mechanism. HNSW is also a strong fit for what the centroid index actually needs to do. In HFresh, the centroid layer is responsible for routing queries to the right postings quickly and accurately. That means it must remain compact, have low latency, and support frequent updates as postings evolve over time. Centroids are not static: splits, merges, and reassignments continuously reshape the partitioning of the vector space, so the centroid index must be able to absorb many insertions and deletions without requiring expensive rebuilds. Another advantage is that the centroid index can itself be quantized. Because the centroid layer is only used to identify promising regions of the vector space, HFresh can compress those vectors aggressively enough to reduce memory usage while maintaining the accuracy needed for strong recall. In fact, HFresh uses HNSW with RQ8, reducing memory usage of centroids by 4x. Using HNSW here also means that improvements to HNSW automatically benefit HFresh. A good example is ACORN, which improves filtered search by making graph traversal more efficient when only part of the dataset matches a filter. Since HFresh relies on HNSW as the centroid layer, those improvements are not isolated: they strengthen HFresh's query path. Quantization HFresh uses Rotational Quantization in two places, with two different compression levels. The reason is that the two stages of search have different jobs. The centroid index routes the query to the right postings, so it needs enough precision to avoid sending the query to the wrong part of the vector space. The postings, on the other hand, are used to generate candidates. Their approximate scores are not the final ranking, because HFresh later rescores the best candidates using the original uncompressed vectors. Rotational Quantization works by rotating vectors into a representation that is easier to compress, then reducing the precision of each dimension. HFresh uses: RQ8 for centroids, reducing centroid vector memory by 4x RQ1 for postings, reducing stored vector data by up to 32x compared to 32-bit floats RQ8 for the centroid index The first stage of HFresh search uses an HNSW index over centroids. This in-memory index identifies which postings are likely to contain the nearest neighbors for a query. HFresh uses RQ8 here because routing mistakes are expensive. If the centroid search misses the right region, the later posting scan may never see the true nearest neighbors. RQ8 gives a strong compromise: it makes centroid-vector payloads approximately 4x smaller before overhead while keeping enough precision for accurate routing. This works especially well because the centroid index is much smaller than the full vector dataset. HFresh can afford a higher-precision compression format for centroids, while still keeping the in-memory layer compact. RQ1 for postings After HFresh has selected the most promising postings, it scans the vectors stored inside them. This stage has a different tradeoff. Postings live on disk, so their size directly affects storage cost and query I/O. HFresh stores posting vectors with RQ1, where each dimension is represented with a single bit. Compared to 32-bit float vectors, that is up to a 32x reduction in vector storage. That aggressive compression makes each posting smaller and cheaper to read. It also makes the first-pass distance computation fast enough to scan many candidates. HFresh does not rely on RQ1 scores for the final ranking. Instead, RQ1 is used to build a candidate set. HFresh then fetches the original uncompressed vectors for the top candidates and recomputes exact distances during rescoring. This keeps disk reads small without giving up final ranking quality. Background operations The reason HFresh can stay balanced over time without rebuilds is that maintenance is built directly into the index. Instead of letting imbalances accumulate and then fixing everything in one large offline job, HFresh breaks down maintenance into small background tasks that can be processed continuously. In practice, most foreground writes stay simple: vectors are appended quickly, and follow-up work is pushed to background queues. Those tasks are persisted in dedicated on-disk queues, so they survive restarts and can be drained incrementally by the scheduler. HFresh organizes this work into three main background task types: split, merge, and reassign. Split: When a posting grows too large, HFresh splits it. The posting is loaded, stale entries are garbage-collected, its vectors are divided into two balanced groups using the Balanced K-Means algorithm, and two new centroids are created to replace the old one. This keeps postings from growing too much, which would make disk reads heavier and routing less precise. Merge: The opposite problem is postings that become too small. That can happen after deletes or simply as the data distribution evolves. In those cases, HFresh looks for a nearby posting that can absorb the smaller one without becoming too large itself. If it finds a good candidate, it merges the two and removes the extra centroid. This prevents the index from fragmenting into too many tiny postings. Reassign: After a split or a merge, some vectors may no longer belong in the posting where they are currently stored. That's where reassignment comes in. The SPFresh paper describes a protocol called LIRE (Lightweight Incremental Rebalancing) which, after a split, checks whether some vectors would now fit better under one of the new centroids, or even under a neighboring posting. After a merge, it checks whether some vectors that were absorbed should really be moved elsewhere instead. These reassignments let HFresh progressively correct mistakes instead of trying to get everything perfect in one step. Together, these background operations form a continuous balancing loop. Inserts make local changes, and the index quietly tidies itself up afterward. The result is an index that stays fresh over time without disruptive rebuild cycles. Filtered search Filtered vector search adds another constraint: results must be similar to the query vector while also satisfying the filter. In an e-commerce search, for example, that might mean finding similar products from specific brands, within a price range, and currently in stock. HFresh uses an allow list, represented as a bitmap, to track the document IDs that satisfy the filter. It then chooses between two search strategies based on the number of matching vectors. For highly selective filters, running the full HFresh pipeline can cost more than searching the matching subset directly. When the allow list contains fewer than 5,000 IDs, HFresh bypasses centroid routing and postin [truncated for AI cost control]