Introducing ExtractBench: The Most Comprehensive Benchmark for Data Extraction from Enterprise Documents

Why we built ExtractBench What ExtractBench evaluates How the ground truth was built Results Reproduce it yourself Enterprise agents increasingly act on critical business data, and much of that data lives in unstructured documents. Document extraction turns those documents into structured data that agents can process at scale. But as agents gain autonomy and humans step out of the loop, document extraction has to meet a higher bar than it did when a human reader was there to catch mistakes. That means developers need a benchmark that measures extraction the way production runs it. So we built ExtractBench. ExtractBench is an open benchmark: the dataset, eval harness, and methodology are all public and reproducible. It is the most comprehensive of its kind, testing 370 enterprise documents (4,869 pages) across 8 business domains and 67 document types, each type with its own schema. It is also the only benchmark that jointly evaluates long-record completeness, real scans and handwriting, word- and page-level grounding, and measured cost.We evaluated 14 systems on it: frontier VLMs, coding agents, and specialized extraction APIs. And it arrives with a companion: Agentic Plus, a new LlamaExtract tier that debuts at the top of the board. We're excited for you to put both to the test. Dataset: HuggingFace Code & evaluation: GitHub Paper: arXiv Why we built ExtractBench We built ExtractBench because we couldn't find an existing benchmark we trusted, or one that reflected the questions our enterprise customers ask when deploying document extraction to production: Does quality hold on a 100-page filing, or only on the first 10 pages? Can the system read a scan of a 1950s typewritten document ? When it returns a value, can it show where it came from? What does it cost to extract a million pages a month? Existing benchmarks fall into two generations. The older ones (SROIE, DocILE, RealKIE) score how well a model recovers a fixed set of fields, which is real work but says nothing about how a system handles a schema it hasn't seen. The newer schema-guided ones do take a new schema at inference time, but each covers just a slice of the workload.None of them scores visual grounding, includes a scanned or handwritten page, or reports what a page costs to run. What ExtractBench evaluates The corpus is real enterprise paperwork: SEC filings and fund-holdings schedules, government procurement and customs forms, healthcare remittance, auto valuations, bankruptcy filings, and energy-sector regulatory forms. ExtractBench tags each document along five independent axes, and every score is reported per axis, so a low score can be traced to its actual cause. Task challenge — how difficult the extraction job itself is: thousands of repeated records, a few values buried in a long filing, or hundreds of fields on one dense form. Each one fails in its own way. Perception — how the page arrived: born-digital, scanned, handwritten, rotated. The same field on the same form is a different problem scanned than born-digital. A system that reasons perfectly still misreads a smudged 5 as an 8. Table structure — merged headers, tables that run across pages, tables past a thousand rows. Values can be read correctly and still be assembled into the wrong records. Document length — short, medium, long. A system can read every page correctly and still stop before the end of a long schedule. Business domain — 8 domains and 67 document types. Domain measures coverage rather than difficulty. A system that wins on financial filings tells you nothing about customs forms. Scroll for all benchmarks → T1 · Long-list completeness Schema-guided Fixed ontology ExtractBench Ctx EB Extend LA Micro1 LX VAREX RealKIE DocILE VRDU Leg KIE Kleister CUAD T1 · Long-list completeness T1.a Single long table T1.b Cross-page continuation T1.c Repeated complex region T1.d Pivoted / matrix T1.e Packed / multi-row cell Scroll for all benchmarks → T2 · Needle-in-haystack Schema-guided Fixed ontology ExtractBench Ctx EB Extend LA Micro1 LX VAREX RealKIE DocILE VRDU Leg KIE Kleister CUAD T2 · Needle-in-haystack T2.a Sparse in narrative T2.b Paraphrased match T2.c Dedup across modalities T2.d Cross-reference / reconciliation Scroll for all benchmarks → T3 · Dense documents Schema-guided Fixed ontology ExtractBench Ctx EB Extend LA Micro1 LX VAREX RealKIE DocILE VRDU Leg KIE Kleister CUAD T3 · Dense documents T3.a Dense form T3.b Receipt / invoice T3.c Classify then extract T3.d Filer–reviewer separation Scroll for all benchmarks → Perception challenges Schema-guided Fixed ontology ExtractBench Ctx EB Extend LA Micro1 LX VAREX RealKIE DocILE VRDU Leg KIE Kleister CUAD Perception challenges P1 Rotated / image-only P2 Scanned P3 Handwriting Scroll for all benchmarks → Table structure Schema-guided Fixed ontology ExtractBench Ctx EB Extend LA Micro1 LX VAREX RealKIE DocILE VRDU Leg KIE Kleister CUAD Table structure S1 Merged headers S2 Header not at top / pivoted S3 Cross-page table S4 Enormous table S5 Table within a cell Scroll for all benchmarks → Grounding & output trust Schema-guided Fixed ontology ExtractBench Ctx EB Extend LA Micro1 LX VAREX RealKIE DocILE VRDU Leg KIE Kleister CUAD Grounding & output trust G1 Value-level box G2 1:N cardinality G3 Deep nesting G4 Checkbox / boolean box Covered & scored Partial or incidental Absent How the ground truth was built A benchmark is only as good as its answer key. It has to be right, and it can't favor any one system. So no extractor's output is trusted on its own, ours included. We built the key three ways: Real documents: systems from different model families run the same schema. Values they all agree on become candidate truth; where they disagree, a person looks at the document and decides. Synthetic long lists: They are built data-first, so every value and box is known before the PDF is rendered and completeness can be scored exactly at thousands of rows. Forms: 169 regulatory and tax forms, where a reviewer checks every field by hand and places the bounding box for 84% of fields Every document's ground truth comes from one of these three documented pipelines and every system gets identical inputs. Results We tested 14 systems across three categories: frontier VLMs called through structured-output APIs, alongside four self-hosted open-weight models; coding agents working in isolated sandboxes with local tools only; and specialized extraction APIs, including LlamaExtract in all three modes(Cost-Effective, Agentic, and Agentic Plus ) Agentic Plus is our new highest-accuracy tier, built for the broadest range of document extraction challenges. It debuts at the top of the board: 95.6 % value F1 and the best grounding scores at both levels, at 8.1¢ a page Mean value accuracy across all 370 documents. Top result 95.6% LlamaExtract Agentic Plus · 8.1¢ LlamaExtract · Ours Agentic Plus 95.6% 8.1¢ Agentic 89.5% 3.1¢ Cost Effective 86.8% 1.0¢ Coding Agents Codex (GPT-5.5) 93.6% 27.8¢ Claude Code (Opus 4.8) 87.1% 16.2¢ Specialized APIs Reducto Deep Extract 90.4% 34.4¢ Extend (Max Context) 86.3% 10.0¢ Datalab (Accurate + Balanced) 64.5% 3.5¢ General-Purpose VLMs Qwen3.6 35B 87.3% — Gemini 3.5 Flash 79.8% 1.0¢ Lift (9B OSS) 77.3% — GPT-5.4 Nano 74.9% 0.21¢ Gemma4 26B 66.2% — NuExtract3 47.9% — Dense multi-domain business docs. 72 documents · 22 types · ≤10 pages. Top result 96.6% LlamaExtract Agentic Plus · 8.3¢ LlamaExtract · Ours Agentic Plus 96.6% 8.3¢ Agentic 92.0% 3.1¢ Cost Effective 90.8% 1.0¢ Coding Agents Codex (GPT-5.5) 95.7% 27.8¢ Claude Code (Opus 4.8) 90.1% 21.4¢ Specialized APIs Reducto Deep Extract 94.2% 38.0¢ Extend (Max Context) 92.0% 10.0¢ Datalab (Accurate + Balanced) 62.8% 3.5¢ General-Purpose VLMs Qwen3.6 35B 93.1% — Gemini 3.5 Flash 87.9% 1.2¢ Lift (9B OSS) 87.2% — Gemma4 26B 80.5% — GPT-5.4 Nano 77.4% 0.25¢ NuExtract3 54.4% — Multi-section reports. 70 documents · 26 types · 11–50 pages. Top result 93.3% LlamaExtract Agentic Plus · 7.7¢ LlamaExtract · Ours Agentic Plus 93.3% 7.7¢ Agentic 85.4% 3.1¢ Cost Effective 80.1% 1.0¢ Coding Agents Codex (GPT-5.5) 91.2% 27.8¢ Claude Code (Opus 4.8) 79.2% 8.2¢ Specialized APIs Reducto Deep Extract 80.5% 23.1¢ Extend (Max Context) 78.8% 10.0¢ Datalab (Accurate + Balanced) 73.8% 3.5¢ General-Purpose VLMs Qwen3.6 35B 84.8% — GPT-5.4 Nano 76.4% 0.12¢ Gemini 3.5 Flash 69.8% 0.69¢ Lift (9B OSS) 62.6% — Gemma4 26B 40.5% — NuExtract3 39.3% — Repeated records at extreme scale — up to 26,725 rows. 17 docs · 12 types. Top result 94.4% LlamaExtract Agentic Plus · 7.5¢ LlamaExtract · Ours Agentic Plus 94.4% 7.5¢ Agentic 78.6% 3.1¢ Cost Effective 69.2% 1.0¢ Coding Agents Claude Code (Opus 4.8) 88.1% 3.4¢ Codex (GPT-5.5) 78.9% 27.8¢ Specialized APIs Reducto Deep Extract 92.0% 50.6¢ Extend (Max Context) 51.3% 10.0¢ Datalab (Accurate + Balanced) 40.5% 3.5¢ General-Purpose VLMs GPT-5.4 Nano 35.8% 0.05¢ Gemini 3.5 Flash 27.9% 0.24¢ Qwen3.6 35B 26.8% — Lift (9B OSS) 25.3% — Gemma4 26B 12.2% — NuExtract3 8.9% — Short documents make everyone look good. Eight of the fourteen systems score above 90% on documents under ten pages. At that length the board is nearly flat, and the only real difference between systems is price. Length is where systems separate. On documents longer than fifty pages, every commercial VLM falls below 35% recall while still retaining high precision. The values that come back are right; most of the document never comes back at all. Agentic Plus is the only system that barely moves, holding 94.4% on the longest documents. Every coding agent has a perception blind spot. The two frontier coding agents fail in opposite directions: one reads scans and handwriting well but stumbles on rotated pages, the other handles rotation and falls apart on scans and handwriting. Agentic Plus is the only system that stays above 93% on all three. Grounding remains an open challenge. Systems are increasingly good at naming the source page for extracted values, but reliably connecting each value to its exact supporting evidence is not completely solved, by anyone. VLMs and coding agents return no evidence by default, so they score zero at both grounding levels. Among the systems that do return boxes, Agentic Plus leads at both levels. Grounding is the clearest area for improvement across the field, and this benchmark provides a concrete baseline to track that progress. Quality only matters at a price. Enterprises typically run document extraction at large scale: at a million pages a month, each cent per page is another $10,000. So a couple points of accuracy bought at four times the cost is a bad trade. Quality vs Cost Quality against cost, ranked The 9 evaluated systems that carry a measured price, ordered by overall value F1 — best first. Two patterns stand out: Price doesn't predict accuracy. Cost and accuracy barely track each other across the 10 systems: the most expensive system isn't the most accurate, and the accuracy leader charges less than a quarter as much. LlamaExtract delivers the most accuracy per dollar. Our new Agentic Plus tier posts the top score at 8.1¢ a page; the closest peer in accuracy runs at more than three times the rate. At the other end, Cost-Effective comes within a point of Claude Code at a sixteenth of the rate. Reproduce it yourself The dataset, harness, and paper are open, and the schemas are frozen. Run your own system against it: git clone TODO-github-url cd ExtractBench uv sync uv run extract-bench run Dataset: HuggingFace Code & evaluation: GitHub Paper: arXiv Related articles