待翻譯：Object Storage + WAL: Lakebase Postgres for the agentic era

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Object Storage + WAL: Lakebase Postgres for the agentic era | Databricks Blog Skip to main content Agents that interact with a traditional OLTP database often create bottlenecks at the storage layer. New deployments, copies, restores, and replicas all mean moving around large volumes of data which is time-consuming and expensive. The polar opposite is true for object storage. Amazon S3, for example, is cheap, performant, almost invisible to operate. It creates a scalable, cost-effective storage layer for agent memory. Which brings us to the question: Can object storage sit underneath a transactional database and make it easier for agents to work with? This question is what started Lakebase Postgres. The answer does not just depend on how fast your object store is, but rather where you place the source of truth. Two OLTP models The usual mental model for OLTP is data-centric. Data is organized into tables with rows and columns, each representing an entity. Storage is the place where the current state lives, and the database's job is to store and retrieve it. But there is a second model: transaction-centric. Here the database is a journal of transactions. Each entry is an operation, and storage is a timeline of those operations rather than a snapshot of the present. The current state is one thing you can derive from the timeline. For years the data-centric model was the only one that mattered in practice, because what the operations team asked of a database were reads and writes against the present. Over the past few years, that has dramatically changed. The operations that agent workloads ask for are almost all operations on transaction history: Give me an isolated copy of production to work in Put it back the way it was before my last three statements Show me what this table looked like before the migration Run twenty of these at once, and delete nineteen of them in an hour These are all queries about the timeline. A database that only stores the present delivers copies and backups, which are slow and expensive. However, Postgres already contains this timeline: it is called the write-ahead log (WAL). The writing in the WAL Postgres’ WAL records every modification before it reaches the data files. It originally existed so Postgres could recover: if the server died between the log write and the data file write, a WAL replay closed the gap. But WAL contents are interesting far beyond recovery. Take a table and an insert: Before that change reaches the users table on disk, Postgres appends it to the WAL. The log is binary, but pg_waldump will render it. The records for this insert look roughly like this: These are four records, and one transaction. Note how each has a log sequence number (LSN), a monotonically increasing identifier. The heap and btree lines also name the exact 8 KB page that changed. The log does not say "a row was added." It says which page, in which relation, at which point in the timeline. Read that as a recovery mechanism and it is a list of work to redo after a crash. But if you read it as a transaction journal, it is something else: A complete, ordered, byte-level account of every page the database has ever changed, with a unique name on every entry. That name, the LSN, is the part that matters most. It means the timeline is already addressable. Nothing needs to be added to Postgres to make "the database as of a point in time" a well-defined thing. It only needs a storage layer that keeps the log around and can answer questions against it. The log becomes the source of truth In a conventional Postgres deployment, the WAL is a means to an end. The data files are the database, the log protects them, and the log is trimmed once its records are safely applied. Storage is simply a disk attached to the machine running Postgres, and everything about the database's identity is tied to that machine. Now, let’s invert it. Make the log the database, and the data files a derived, cached representation of it. Then you can keep the full timeline, and you no longer have to move data to copy or rewind the database. History becomes addressable, so a database “copy” becomes a pointer instead of a second set of files. This makes deployments, restores, and replicas cheap enough to treat like code. That’s what we did in Lakebase Postgres. Concretely, we split the system into two layers: The compute layer The compute layer runs standard Postgres. It parses SQL, plans and executes queries, enforces MVCC, manages locks and indexes. Nothing in the query engine is rewritten. What changes is what the compute node is responsible for: it exists to execute work, not to preserve data. It has RAM for shared buffers and local NVMe as a page cache, and it can start, stop, scale, or die at any moment without putting durability at risk. The storage layer The storage layer owns correctness, durability, and history. It outlives any individual compute node, and it is built from three components with distinct jobs: Safekeepers replicate the WAL. When the compute node generates WAL records, it streams them to several safekeepers, and a transaction is committed once a quorum acknowledges the record through a Paxos-based protocol. Durability is a property of replication and consensus rather than of one machine's fsync. The pageserver turns WAL into pages. It combines base pages with committed WAL records to materialize the version of a page that a given query needs, and it persists those materialized versions into object storage asynchronously. Object storage holds long-term, immutable history. Materialized page versions and historical states arekept as an append-only record rather than a mutable filesystem. The write path What does the write path look like? A commit in this system follows these steps: Postgres applies changes in memory. Buffers are updated, indexes are modified, WAL records are generated exactly as usual. Instead of flushing WAL to a local filesystem, the compute node streams it over the network to the safekeepers. The transaction is committed once a quorum of safekeepers has acknowledged the record. That is the point where the client hears success. Page materialization happens afterward, in the storage layer, off the transaction's critical path. A commit never waits for pages to be written or uploaded. This design might get an obvious objection: that step 2 adds a network hop to the commit path. But any Postgres deployment that takes durability seriously is already running synchronous replication, which is also a network hop. Externalizing the WAL replaces one network round trip with another rather than adding one. The read path Every read request from a compute node carries a page identifier and an LSN, and the storage layer returns the page as it existed at that LSN. This GetPage@LSN is a central operation in this architecture. Serving it takes a preference order: First comes RAM for Postgres shared buffers, exactly as in any Postgres. Then comes the local NVMe which is still fast, still local. If the page is not in memory, the compute node checks its local disk cache Only on a local miss does the request cross the network into the pageserver. The pageserver then checks whether it already has that page version materialized. If not, it finds the most recent image of the page at or before the requested LSN, collects the WAL records on top of it, replays them, and returns the reconstructed page. The returned page is then cached in RAM and on NVMe, so the next read of it is local again. A primary node asks for the latest version of every page, so in steady state it behaves like any Postgres reading from a warm cache. But nothing in the protocol requires "latest." Ask for a page at an LSN from four hours ago and you get that page from four hours ago. The useful consequence is that the distinction between live data and historical backups disappears. There is one storage system. Old page versions are not a separate artifact kept somewhere else in a different format; they are the same immutable files, still addressable. Non-overwriting storage In other words, the pageserver never updates a file in place. Files are created, merged, and deleted, but never modified. This is a perfect fit for object storage, which does not offer random updates, and that makes history cheap enough to keep. Data is organized into two kinds of layer files: An image layer holds a snapshot of every key in a key range at one LSN A delta layer holds all the changes in a key and LSN range. Keys that were not modified are not stored. Incoming WAL is written out as delta layers. Image layers are produced in the background, for two reasons: they shorten the replay chain a read has to walk, and they make old deltas collectable. Without them, reconstructing a page could require walking back arbitrarily far. So GetPage@LSN becomes a search: start at the requested key and LSN, walk down through the layers collecting WAL records for that page, and stop at the first image of it. To keep that search short, delta and image layers are reshuffled by background compaction, and layers that fall outside the retention window are garbage collected. How to find the right layer quickly The search described above sounds simple, but it is not. It is worth spending some time on, since it determines whether the whole design is viable. A read names a key and an LSN. The storage system has to find the nearest layer that covers that key at or before that LSN. That is a geometric problem, and it is not obvious how to solve it across tens of millions of layers. A linear scan is far too slow, and the obvious spatial structures do not fit: R-trees answer containment queries rather than "the first layer below this point," and segment trees scale with the size of the coordinate space rather than with the number of layers. There are several approaches to this design, but what worked was to solve the easy problem first, then make the data structure remember its own past. Step one: Solve it for a single LSN For one fixed LSN, we work out which layer answers each key. That answer only changes at a handful of points across the key space, so we record those points and store them in a binary search tree. That tree is the layer coverage for that LSN, and it answers any read at that LSN with a single lookup. This works, but only for one LSN. Coverage changes every time a layer is added, and there are millions of LSNs, so we cannot build and keep a separate tree for each one. Step two: Make the tree persistent Persistent as in, “keep the old versions available”. We build the coverage incrementally, inserting layers in LSN order from the bottom up. Inserting one layer only touches the nodes along a single path from the root downward. Instead of overwriting those nodes, the system copies them and leaves the originals untouched. The new copies point at the old, unchanged subtrees on either side. Two things follow from that: The insert costs a handful of new nodes rather than a whole new tree, because everything off the path is shared The old root still describes the tree exactly as it was before the insert, so it remains a valid coverage for the earlier LSN We do that for every layer, in order, and we end up with a single structure that contains every intermediate root, each one the coverage at a different LSN. We get all of those trees for close to the price of one. A historical read then costs the same as a current one: the system picks the root for the LSN you want, and does the same single lookup. That is the trick, in summary: Latest-only reads are one tree lookup Historical reads use an older root, so they cost the same Building those roots stays cheap as layers accumulate, so a long history does not make lookups slower Where object storage actually sits This is where the current argument about Postgres and object storage tends to go wrong [truncated for AI cost control]