待翻译：Open Table Formats Explained: Iceberg vs. Delta vs. Hudi

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Open Table Formats Explained: Iceberg vs. Delta vs. Hudi | Databricks Blog Skip to main content Open table formats bring ACID transactions and schema evolution to data lakes; catalog-coordinated commits now let Delta Lake and Apache Iceberg share a single governance model. Metadata trees and transaction logs enable data skipping and time travel, cutting query costs while keeping every prior table version reliably queryable. Interoperability features like Delta Lake UniForm and Unity Catalog reduce vendor lock-in, letting teams query the same data natively as Delta Lake or Iceberg across Spark, Trino, and others. Open table formats are metadata layers that sit on top of data files in object storage, adding ACID transactions, schema evolution, and time travel to data stored in a data lake. Apache Iceberg, Delta Lake, and Apache Hudi are the three main open table formats in production use today, and each turns a collection of Parquet or ORC files into a table that behaves like a database: readers see consistent results, writers can update and delete rows safely, and every change is tracked so earlier versions remain queryable. This overview explains how the main open table formats work, how they compare on ACID transaction support and schema evolution, and how they relate to the data lakehouse architecture — drawing on storage-layer innovations like catalog-coordinated transactions, row lineage, and unified metadata to show where Delta Lake and Apache Iceberg are converging. What Is a Data Lake, and Why Do Open Table Formats Matter? A data lake is a centralized repository built on low-cost object storage — Amazon S3, Azure Data Lake Storage, or Google Cloud Storage — that holds structured, semi-structured, and unstructured data in raw, native form. Organizations adopted data lakes because object storage scales cheaply and separates storage from compute, letting any query engine read the same data. Object storage was never built to guarantee consistency, though: it has no native concept of a table, schema, or transaction. A data lakehouse layers table-like structure, governance, and performance onto that raw storage, combining the low cost of a data lake with the reliability of a data warehouse. The bridge between the two is the open table format: it turns loose files in object storage into governed, queryable tables without copying data into a proprietary warehouse. Before open table formats existed, running analytics on traditional data lakes caused sustained problems: concurrent writers could corrupt data mid-write, updates and deletes meant rewriting entire partitions, and there was no dependable way to know which files represented a table's current, correct state. Open table formats manage metadata for data files in object storage, tracking exactly which files belong to a table — the standardization that let data lakes finally support database-like features such as record-level updates. The Open Table Formats and File Formats You Need to Know Apache Iceberg Apache Iceberg, originally developed at Netflix and now an Apache Software Foundation project, was designed to make huge, slow-changing tables fast to query and safe to evolve. Apache Iceberg uses a tree structure for efficient metadata management: manifest files and manifest lists track every data file in a table, letting query engines prune irrelevant data before a scan starts. Iceberg tables support schema evolution and partition evolution without rewriting underlying files. Delta Lake Delta Lake, created by Databricks and released as open source, brought ACID transactions to Apache Spark workloads through a write-ahead transaction log. Delta Lake originated at Databricks and integrates with Spark natively, though it now supports a broad set of engines through independent connectors. Delta Lake tables record every write as an ordered, atomic log entry, giving readers a consistent view even while new data is being written. Apache Hudi Apache Hudi, short for Hadoop Upserts Deletes and Incrementals, is built around fast, frequent record-level updates. Apache Hudi optimizes for frequent updates and streaming data by maintaining indexes that locate the exact file holding a given record, enabling efficient record-level updates without a full table scan. That design makes Hudi a common choice for change-data-capture pipelines and near-real-time ingestion. Parquet and ORC Parquet and ORC are columnar file formats, not table formats: they define how individual data files are organized, not how files become a governed table. Iceberg, Delta Lake, and Hudi are all built on Parquet files — Iceberg and Hudi also support ORC — using file-level statistics Parquet stores to prune data before a query engine reads it. That distinction, file format versus table format, clears up most enterprise confusion about where each layer's responsibilities begin. Iceberg vs. Delta Lake vs. Hudi: Quick Comparison The three main open table formats now share more capabilities than they differ on, but the table below highlights where design history still shows through. CapabilityApache IcebergDelta LakeApache Hudi ACID transactionsYes, via catalog-coordinated commitsYes, via transaction log and catalog commitsYes, via timeline-based commits Schema evolutionFull — add, drop, rename, reorder columnsFull, including column mappingFull, schema-on-write and schema-on-read Partition evolutionYes, without rewriting existing filesLimited; typically requires redefinitionYes, via evolving indexing strategies OriginNetflix / multi-engine analyticsDatabricks / Apache SparkUber / streaming ingestion Update/delete performanceDeletion vectors and row lineage (v3)Deletion vectors and row trackingNative record-level indexes Multi-engine supportBroad — Spark, Trino, Flink, SnowflakeBroad via Delta Kernel and UniFormSpark, Flink, Presto, Trino Inside Apache Iceberg: Tables and Metadata The Iceberg metadata tree An Iceberg table is defined by a metadata tree, not a single file: a metadata file points to a manifest list, which points to manifest files listing the actual data files that make up a snapshot. This layered structure lets a query engine prune irrelevant manifests and data files using stored column statistics without opening a single file, improving query performance on tables with millions of files. Snapshots and time travel Every write to an Iceberg table creates a new snapshot — an immutable record of which data files existed at that moment — and the metadata tree keeps a history of previous snapshots. This enables time travel: engines can query a table as it existed at a specific snapshot ID or timestamp, supporting auditing, reproducible ML training sets, and rollback to the last stable state after a bad write. Partition evolution Iceberg decouples a table's physical partitioning from its query patterns through partition evolution, letting teams change how new data is partitioned without rewriting existing files or breaking queries against the old scheme. Hidden partitioning means analysts do not need to reference physical partition columns directly to get partition pruning. Iceberg Table Internals and Maintenance Because every write produces a new snapshot with its own manifest list, an actively written Iceberg table can accumulate thousands of small manifest and data files if left unmanaged. Query engines still have to open and evaluate each relevant manifest file, so manifest sprawl erodes the query performance gains the metadata tree was built to deliver. The standard remedy is scheduled compaction: a maintenance job that rewrites small data files into fewer, larger ones and consolidates manifests, run nightly or hourly depending on ingestion volume. Pairing compaction with regular snapshot expiration — removing metadata past a retention window — keeps storage and metadata size under control without limiting how far back time travel can reach. Delta Lake and ACID Transactions The Delta Lake transaction log Delta Lake tables store an ordered, append-only transaction log — a sequence of JSON entries recording every add, remove, and metadata change — alongside periodic checkpoint files that summarize the log for faster reads. Historically, the file system itself acted as commit coordinator, meaning any client with file-level access could write to a Delta table directly, without going through a governing catalog. How ACID transactions behave in Delta Lake ACID transactions ensure data consistency during concurrent writes by requiring every writer to check the current log version, generate a new entry, and commit it only if no conflicting change happened in between; if a conflict is detected, the writer retries. ACID compliance prevents data corruption because a reader never sees a Delta Lake table in a partially written state, and ACID transactions enable complex data operations without conflicts across overlapping partitions. From Spark-native to multi-engine support Because Delta Lake's original commit model depended on file system access, third-party engines outside Apache Spark had to reach tables through static file paths rather than a governing catalog — leaving those accesses ungoverned and able to silently break schema relationships. Databricks addressed this with catalog commits, an open standard letting a catalog such as Unity Catalog act as commit coordinator, so every read, write, and discovery request is authorized centrally. Catalog commits are now generally available, aligning Delta Lake with the catalog-oriented approach Iceberg has used from the start and unlocking multi-table transactions. File Format Fundamentals: How Parquet Enables Data Skipping A Parquet file organizes tabular data by column rather than by row, grouping values from the same column into contiguous blocks called row groups. Columnar storage lets a query engine read only the columns a query references, skipping the rest — a major reason Parquet outperforms row-oriented formats for scan-heavy workloads. Every row group carries statistics — minimum and maximum values, null counts, and value distributions per column — written into the file's metadata footer. These statistics let an engine determine, without decompressing any data, whether a row group could possibly match a query's filter. Open table formats extend this principle a level higher: Iceberg's manifest files and Delta Lake's transaction log both cache Parquet-level statistics at the metadata layer, so an engine can skip entire data files before listing them from object storage. This two-tier data skipping is a major contributor to improved query performance on large tables. Databricks has also extended the model with the Variant data type — now part of Parquet, Delta Lake, and Iceberg — storing semi-structured payloads in typed binary form instead of raw JSON, so engines extract nested fields without expensive parsing. Read now Data Versioning, Time Travel, and Incremental Processing Data versioning is an open table format's ability to retain a record of every previous table state rather than overwriting data in place, and time travel reads any of those previous versions by version number, snapshot ID, or timestamp. Open table formats allow for time travel and versioning of datasets by design, since every write already creates a new, independently addressable snapshot or log entry. Incremental processing reads only the rows that changed since a table was last processed rather than rescanning an entire dataset — the pattern behind Change Data Capture (CDC), where pipelines consume just the inserts, updates, and deletes applied to a source table. Row lineage and deletion vectors, introduced to Delta Lake and brought to Iceberg through Iceberg v3, made this cheaper: row lineage tracks which rows changed since a table was last scanned, and deletion vectors represent deleted rows as a compact bitmap instead of rewriting data files. [truncated for AI cost control]