Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks

Datalab has released OmniExtractBench, an open benchmark for structured document extraction. It tests how accurately a system fills a JSON schema from a PDF. The benchmark pools 620 documents from 4 existing benchmarks. One deterministic scorer grades all of them and explains each decision. The release lands while extraction vendors publish their own leaderboards. Datalab argues those leaderboards are hard to compare or audit. OmniExtractBench is its attempt at a shared yardstick. Is it deployable? Yes, the scorer installs from PyPI as omni-extract-bench (v0.1.7, Python 3.11+, SciPy only) under Apache 2.0. Rerunning vendors requires your own API keys and paid credits. What is OmniExtractBench? OmniExtractBench is a structured extraction benchmark built by Datalab. Each task gives a system a PDF and a JSON schema. The system returns JSON, which is scored value by value against a gold file. The code is on GitHub, and the data is on Hugging Face under CC BY 4.0. The 4 flaws it targets Datalab’s launch post names 4 recurring problems with existing extraction benchmarks: Bias: documents and scoring can favor the vendor that built the benchmark. Opaque harnesses: a low score may reflect a broken harness, not a weak model. Unclear scoring: readers cannot tell why a given document scored low. Narrow document variety: some suites hold only dense tables, others only clean, unscanned files. Where the 620 documents come from SuiteDocumentsUpstream publisherContent ExtractBench329LlamaIndexForms, filings, decks Internal202Datalab (synthetic)Dense scalar schemas, small documents LongExtractBench47micro1 (commissioned by Reducto)Very large tables LongArray-Extract42ExtendLarge tables with repeated scalars Regulatory filing forms are the largest category, at 88 documents. 128 documents are a single page. At the other end, 33 documents over 100 pages hold 40% of all pages. Datalab’s own synthetic suite is the second largest share. How the scorer works The scorer flattens prediction and gold JSON into addresses, which are paths to single values. It normalizes each value first, so “03/31/2024” matches “2024-03-31”. Tables are the hard part. Compared by position, one missed row shifts every row after it. We reran the scorer on a 100-row table missing its first row. Positional comparison scored 0%, while OmniExtractBench scored 99%. The fix is content-based pairing with the Hungarian algorithm. ExtractBench and LongArray-Extract already align rows this way. OmniExtractBench adds a verdict layer on top. 6 verdicts per value matched: paired, and the values agree. misread: paired, but the values differ. unfound: gold has a value, the prediction does not. fabricated: the schema allows it, gold is silent, the prediction fills it. invented_item: part of a predicted row that pairs with nothing. invented_field: an address the schema never declared. Accuracy is matched values over all verdicts. Precision divides matched values by predicted values. Recall divides them by gold values. The full rules are in the metric spec. The null rule Empty strings, None and whitespace count as omissions, so those addresses are dropped. Strings like “NA” or “-” remain real answers. This blocks a quiet exploit: padding a schema with empty optional fields to earn free matches. In our test, padded null fields added 0 verdicts. How it compares with other extraction benchmarks BenchmarkPublisherDocumentsRow alignmentPer-value explanationScorer licenseData license OmniExtractBenchDatalab620, from 4 sourcesHungarian, by contentYes, 6 verdict typesApache 2.0CC BY 4.0 ExtractBenchLlamaIndex370HungarianPer-field diffs, HTML reportApache 2.0Apache 2.0 LongArray-ExtractExtend45, syntheticHungarianPer-document scoreNot statedCC BY 4.0 LongExtractBenchmicro1225, of which 50 publicBy row keyNot statedMITCC BY 4.0, labels only Sources linked on each benchmark name. Checked September 27, 2026. Who misses fields and who invents values Datalab scored 10 system configurations on the full corpus. Its accurate mode led at 93.85 accuracy. Datalab balanced (93.48) and Reducto deep_extract v2 (93.47) are effectively tied. Precision and recall then show how each system fails. Balanced: Datalab (both modes) and Reducto keep precision and recall within 0.6 points. Leans to misses: GPT 5.6-sol posts 95.11 precision but 84.99 recall. It loses 11.88% to unfound values. Gemini and Claude show the same pattern, less sharply. Leans to invented values: LlamaExtract has 93.13 recall but 86.57 precision, losing 9.03% to fabricated values. Extend loses 4.01% to invented items. Low on both: Mistral OCR 4.1 and Azure Content Understanding trail on both metrics, with recall lower still. How each system falls short of 100%, by verdict type. Source: Datalab. Run it yourself Copy CodeCopiedUse a different Browser uv pip install omni-extract-bench oeb score --pred pred.json --gt gold.json --schema schema.json --verdicts Install the [benchmark] extra and run oeb benchmark to rerun vendors. Runs are resumable, and each provider needs its own credentials. Datalab also suggests testing its playground on your own documents. Key Takeaways OmniExtractBench pools 620 documents from LlamaIndex, micro1, Extend and Datalab suites. A deterministic scorer gives every value 1 of 6 auditable verdicts. Content-based row pairing stops one missed row from zeroing a table. Dropping null addresses stops schema padding from inflating scores. Datalab accurate leads at 93.85; Datalab balanced and Reducto tie near 93.5. FAQ What does OmniExtractBench measure? It measures how accurately a system extracts values from a PDF into a JSON schema, scored per value. Is OmniExtractBench open source? Yes. The scorer is Apache 2.0 on GitHub and PyPI. The dataset is CC BY 4.0 on Hugging Face. Who built OmniExtractBench? Datalab built it. Datalab also maintains the open-source Marker and Surya document tools. Thanks to the Datalab team for the resources behind this article. Datalab supported and sponsored this content. The post Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks appeared first on MarkTechPost.