2026-07-03 04:00 UTCOriginal source2 min readUpdated: 2026-07-03 08:13 UTC

Office Comprehension Benchmark

The Office Comprehension Bench (OCB) is the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats. It comprises two tracks: File Fidelity Q&A (structural/visual perception) and Domain Q&A (expert reasoning across 12 domains). Even the strongest frontier system achieves only 59.3% on Domain Q&A; increasing thinking depth within a tier yields no material improvement, while moving to a higher product tier offers modest gains. The dataset, evaluation tooling, judge prompt, and leaderboard are released.

SourcearXiv Computational LinguisticsAuthor: Firoz Shaik, Mateus Pican\c{c}o Lima Gomes, Tanvir Aumi, Jingci Wang, Milos Milunovic, Filip Basara, Ivana Jovanovic, Vishwas Suryanarayanan, Neha Nandan Kenkare, Weiyao Xie, Zhipeng Han, Zheng Zhang, Waleed Shahid, Jay Rathi, Russell Scherer, Thong Q. Nguyen, Michael Bentley, Tamara Stankovic, Rasika Chakravarthy, Vishal Chowdhary

-->

[Submitted on 29 May 2026]

Title:Office Comprehension Benchmark

View a PDF of the paper titled Office Comprehension Benchmark, by Firoz Shaik and 19 other authors

View PDF

Abstract:We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.docx, .xlsx, .pptx) and their variants. OCB consists of two tracks. File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges. Domain Q&A tests expert-level reasoning grounded in real-world industry documents across 12 professional domains, with queries requiring multi-step analysis and synthesis across documents. Each reference answer is decomposed into atomic, binary-gradable claims, and an ensemble of LLM judges scores responses against each claim independently. Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A; increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains. We release the dataset, evaluation tooling, judge prompt, and a public leaderboard.

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Information Retrieval (cs.IR); Machine Learning (cs.LG)

Cite as: arXiv:2607.01245 [cs.CL]

(or arXiv:2607.01245v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.01245

arXiv-issued DOI via DataCite

Submission history

From: Firoz Shaik [view email] [v1] Fri, 29 May 2026 03:19:32 UTC (1,503 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Office Comprehension Benchmark, by Firoz Shaik and 19 other authors

View PDF

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

cs cs.AI cs.CY cs.IR cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)