待翻譯：Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning | Databricks Blog Skip to main content OfficeQA Pro V2 is a new benchmark for enterprise grounded reasoning, built using our internal synthetic data pipeline from roughly 1,400 U.S. Treasury PDFs spanning 233 years and approximately 120,000 pages. We leverage synthetic data generation to build OfficeQA Pro V2, allowing us to scale diverse, verified questions and answers. Combining these synthetic data techniques with our understanding of enterprise workflows enables us to rapidly build new benchmarks to make progress on the tasks our customers care about, like grounded reasoning. An optimized agent harness can dramatically improve performance. Out-of-the-box agents averaged only 26.0% accuracy on OfficeQA Pro V2, while Databricks’ Genie delivered a 92% relative improvement on average across matched models and achieved up to 60% accuracy using the same models. Despite these gains, significant headroom remains in grounded reasoning. Today, we are releasing OfficeQA Pro V2, a new benchmark designed to evaluate whether AI agents can generalize to unfamiliar, enterprise-style grounded-reasoning tasks. Seven months ago, we introduced the OfficeQA benchmark to measure how well AI systems answer analytical questions using evidence from large document collections, an extremely common and important enterprise task that we found agents struggled with. Since its introduction, OfficeQA, and its frontier subset, OfficeQA Pro, has become an important measure for frontier model and agent capabilities, driving progress in document retrieval, parsing, and analytical reasoning. But this progress raises a fundamental question: do these improvements reflect broader advances in grounded reasoning, or progress specific to one corpus and task distribution? This distinction matters in enterprise settings, where agents rarely operate on a single, stable document collection. Our new benchmark, OfficeQA Pro V2, is designed to test that generalization directly. We initially developed OfficeQA Pro V2 as the benchmark for the inaugural Databricks Grounded Reasoning Cup, a live competition in which 11 academic teams, supported by OpenAI, Anthropic, and Google DeepMind, built agents that were evaluated on a previously unseen corpus and set of grounded-reasoning tasks. The benchmark contains 90 questions grounded in approximately 120,000 pages from the U.S. Treasury’s Accounts of Receipts and Expenditures, which was released as a cohesive dataset by the U.S. Treasury for the first time in conjunction with the 250th anniversary of the United States. Now, we are excited to release the benchmark to AI practitioners more broadly to use for their own development. Accuracy on OfficeQA Pro V2 using model-provider harnesses (Claude Code for Anthropic models and Codex for OpenAI models) versus the Databricks Genie. Across the four matched models, Genie improved mean accuracy by 15.3 percentage points, from 37.5% to 52.8%. OfficeQA Pro V2 remains challenging for today’s AI systems. Out-of-the-box frontier agents using Claude Code with Claude Opus 4.8 and Claude Fable 5, or Codex with GPT-5.5 and GPT-5.6 Sol, achieved an average accuracy of 37.5%. Agents developed specifically for the Grounded Reasoning Cup performed better, averaging 41.1%, with the winning team reaching 63.3%. We also evaluated Databricks Genie, Databricks’ AI agent designed to answer complex questions over enterprise data, using the same underlying models. Genie, which also uses Databricks’ ai_parse to pre-parse the document corpus, improved accuracy by an average of 24.0 percentage points over the default harnesses, with the strongest configuration reaching 60%. Together, these results show that grounded reasoning remains far from solved, but that the right agent harness can unlock significant gains from existing frontier models. Below, we describe in more detail how agents perform on OfficeQA Pro V2, how it was built, how it differs from the original OfficeQA benchmark. Agent Performance on OfficeQA Pro V2 Cost versus correctness for model + harness combinations on OfficeQA Pro V2. Models using Databricks Genie dominate the Pareto frontier, achieving higher quality at a lower cost. Per-rollout costs do not include the one-time, upfront cost of parsing the corpus with ai_parse. We first evaluated frontier models using each model provider’s associated harness, with Codex for GPT models and Claude Code for Claude models, using the same deterministic exact-match metric with 0.0% tolerance metric as used for OfficeQA. Across the five models evaluated under both harnesses, these baseline configurations achieved 26.0% accuracy on average. Performance varied substantially by model, and higher cost did not consistently translate into greater accuracy. For example, Sonnet 5 on Claude Code scored 15.6% at $5.01 per rollout, while GPT-5.6 Sol on Codex reached 33.3% at a comparable $4.70. Using these models within Genie produced substantial gains. Across matched model comparisons, Genie improved accuracy by 24.0 percentage points on average (a 92% relative improvement). Genie configurations using GPT-5.6 Luna, GPT-5.6 Terra, and Claude Fable 5 also dominate the cost-quality Pareto frontier, demonstrating that these improvements do not require trading efficiency for accuracy. In the case of Claude Fable 5, switching to Genie improves performance by 14.4 percentage points (+32% relative) while reducing cost by ~9x. From inspecting traces, we find that Fable 5 is prone to looping attempts to parse many documents, which leads to long rollouts costing $37.36 on average. Genie parses documents using ai_parse document, which enables the model to efficiently identify the correct information from each page, thereby saving cost and improving performance. While these harness improvements lead to substantial gains, significant headroom remains in OfficeQA Pro V2. Systems continue to exhibit similar failure modes to those we observed in the original OfficeQA – parsing fidelity, incorrect temporal reconciliation as accounting conventions change over time, and misinterpretation of entity scope or category granularity. Building a New Grounded Reasoning Benchmark To truly test generalization, we needed a new corpus that still reflected the same enterprise-relevant skills OfficeQA was designed to measure: parsing complex documents, retrieving the right evidence, and performing analytical reasoning grounded in that information. We worked with the U.S. Treasury to build the benchmark around a new corpus: the U.S. Federal Accounts of Receipts and Expenditures, which the Treasury was preparing to release as a unified dataset for the first time in conjunction with the 250th anniversary of the United States. The corpus consists of roughly 1,400 PDFs and 120,000 pages containing detailed U.S. accounting records, spanning 1793 through 2024. Like the U.S. Treasury Bulletins used in the original OfficeQA benchmark, this new corpus reflects many of the challenges common across enterprise document collections: dense tables and charts, values revised over time, evolving reporting conventions, and deep institutional knowledge. These complexities also translate into a challenging evaluation: as with the original OfficeQA benchmark, we find that baseline frontier agents struggle to achieve consistently high accuracy on OfficeQA Pro V2. Scaling OfficeQA Pro V2 with Synthetic Data Synthetic Data Pipeline used to build OfficeQA Pro V2. When we created the original OfficeQA benchmark in late 2025, the process was highly manual. Human annotators constructed questions and answers by hand while combing through a corpus comprising 89,000 U.S. Treasury Bulletin pages, often requiring numerous rounds of human review in order to ensure question quality. Although this approach eventually produced a high-quality benchmark, it was time-consuming, costly, and difficult to scale. Since then, we have developed techniques to automate the creation of rigorous benchmarks much more efficiently and reliably. To build OfficeQA Pro V2, we leveraged asynth, our internal library for building synthetic data-generation pipelines. This enabled us to quickly and scalably create a diverse, verifiable benchmark that accurately reflected the challenges of grounded reasoning using a systematic process for generating and vetting questions: Inputs. To promote sample diversity, the synthesizer was seeded with a randomly selected time period, analytical method, target number of source documents, and salient topic drawn from a list developed with our partner, USAFacts, to reflect questions that analysts might naturally ask of the corpus. Synthesis. For each candidate question, a synthesis agent searched the corpus for evidence matching these criteria and generated a sample supported by a traceable chain of grounded reasoning. Verification. Next, samples went through several quality gates to audit for source fidelity, analytical relevance, diversity, and answerability. Independent solver agents attempted the question from scratch, producing alternate solution trajectories, which were reviewed by a verification agent. Only candidates with verifiably correct answers advanced to the next stage. Final Review. As a final quality-control step, samples were manually reviewed alongside solutions produced by additional frontier models. Reviewers examined the question, answer, reasoning, and cited evidence together, retaining only samples whose interpretation and ground truth were unambiguously correct. The resulting questions require the same core grounded-reasoning capabilities measured by the original OfficeQA benchmark: retrieval across multiple source documents, analytical reasoning, and specialized capabilities such as supplemental web search and multimodal interpretation of figures. Example questions from the benchmark, as well as details about its composition, can be seen below. Example Questions 1. Low Difficulty: Compute Chief Justice Earnings: “What is the total nominal dollar value for John Jay's earnings as Chief Justice of the Supreme Court, according to the Account of Receipts and Expenditures of the United States during the back half of CY 1793, and which warrant numbers corresponded to the payments? Return your answer as comma-separated values in enclosed brackets in the order of his earnings value, and the warrant numbers in calendar year order.” This question requires locating a single 1793 Account of Receipts and Expenditures page and reading the four warrant payments recorded to Chief Justice John Jay (highlighted in red) while correctly interpreting that "do." is shorthand for "ditto," indicating the same data category as the preceding row. The question then requires understanding that the “back half of CY 1793” scopes the answer to only the July and November payments. Finally, retrieval of corresponding warrant numbers and a simple sum is required to compute the final answer. 2. Medium Difficulty: Forecast Increase in Medicare Outlays: “Between FY1990 and FY1994 inclusive, Medicare outlays grew rapidly as the program expanded. Fitting an OLS linear regression to the reported annual Medicare outlay figures for those fiscal years, what is the estimated average annual dollar increase in Medicare outlays reported in millions of dollars, rounded to two decimal places?” This question requires retrieving the Medicare outlay figure from the "Outlays by Function" table in five separate annual Combined Statements (FY1990–FY1994, highlighted in red). Importantly, the agent must read each year’s value from the latest revised prior-year column, rather than the first reported figure. Finally, the agent must fit the series with a linear regression model to estimate the average annual increase. 3. High Difficulty: Forecast Increase of Outlays Across Agencies: “For the combined net outlay [truncated for AI cost control]