Handwritten-edit benchmark: Fable 5 is #1, Opus 4.8 regresses 55% on miscounting

How It Works

Models are presented with scanned pages containing handwritten editorial marks and are asked to identify each edit, its type, location, and the text before and after the edit.

The Task

Each page in the benchmark consists of printed text overlaid with handwritten editorial annotations. Models are tasked with detecting and interpreting all such editorial corrections and outputting them in structured JSON format.

For each correction, the model must identify:

Type of edit: One of insertion, deletion, replacement, punctuation, capitalization, or italicize

Original text: The text as it appeared before the edit

Corrected text: The intended version after applying the edit

Line number: The line on which the edit occurs. Use line 0 for titles or headings, and start counting full lines of body text from line 1

Page: A known identifier for the image (e.g., "001.png"), provided alongside the input and not extracted by the model

Models should infer the intent behind handwritten annotations using both visual and textual cues. Common markup conventions include:

Insertions: Indicated by caret marks (^) or added words between lines

Deletions: Shown using strikethroughs or crossed-out text

Replacements: Circled, underlined, or bracketed text with substitutions nearby

Punctuation edits: Handwritten punctuation added, removed, or modified

Capitalization: Case changes marked explicitly or via notation

Italicize: Text that should be formatted in italics, typically indicated by underlining or special notation

This task combines fine-grained visual recognition with natural language understanding and domain knowledge of editorial conventions. The goal is not just OCR or layout detection, but true interpretation of handwritten edits in context.

Example Input

Sample page from Little Dorrit with editorial marks.