Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

[Submitted on 2 Oct 2026]

Title:Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

View a PDF of the paper titled Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?, by Zongye Lyu

View PDF HTML (experimental)

Abstract:Reinforcement learning with verifiable rewards (RLVR) is credited with gains beyond the domain its rewards check, such as higher MMLU accuracy after training on mathematics. Such accuracies are usually read out of the model's text by an answer extractor. A model that learns to write its answer where the extractor looks can therefore score higher without answering better. We measure how much of the gain reported by extractors copied from public code arises this way. We generate outputs from a base model and its RL model and score them twice: with those extractors, and with one extractor of our own that reads a stated answer whether it is in the answer box \boxed{} or in the text. On four released 7B SimpleRL base-RL pairs, 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours: it is a gain in answer format, not in answers. The base models already state most of their answers in text, and RL moves them into the box, where these extractors look first. Independent LLM readers, in place of our extractor, put the figure at 77%. With a prompt that itself asks for the box, our extractor puts it at 79%. On a 32B pair trained with another recipe, almost none of the reported gain is format gain: its base model boxes its answers when asked, and its RL model rarely does when not. A reported gain is partly a property of the extractor that read it, and re-scoring stored outputs with a second extractor shows how much, without re-running RL.

Comments: 36 pages (10 main), 5 figures, 28 tables

Subjects:

Machine Learning (cs.LG); Computation and Language (cs.CL)

ACM classes: I.2.7

Cite as: arXiv:2610.20639 [cs.LG]

(or arXiv:2610.20639v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.20639

arXiv-issued DOI via DataCite

Submission history

From: Zongye Lyu [view email] [v1] Fri, 2 Oct 2026 12:33:17 UTC (3,633 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?, by Zongye Lyu

View PDF

HTML (experimental)

TeX Source

view license

Ancillary-file links:

Ancillary files (details):

README.md

REDACTIONS.txt

data/items/MANIFEST.json

data/items/aqua_rat.jsonl

data/items/arc_c.jsonl

data/items/bbh.jsonl

data/items/gsm8k.jsonl

data/items/math500.jsonl

data/items/mmlu.jsonl

data/items/mmlu_pro.jsonl

data/items/sat_math.jsonl

data/llama-pilot-items/arc_c.jsonl

data/llama-pilot-items/mmlu.jsonl

data/llama-pilot-items/mmlu_pro.jsonl

harness/adjudicate.py

harness/adjudication_descriptive.py

harness/adjudication_prompt.txt

harness/adjudication_result.py

harness/analyze.py

harness/answer_tags.py

harness/boxing_by_prompt.py

harness/build_generation_manifest.py

harness/correctness_criterion.py

harness/decompose.py

harness/denominator_weight.py

harness/derive_supporting.py

harness/dose_response.py

harness/dose_response_pooled.py

harness/external_matched.py

harness/external_reference.py

harness/extraction_rate_settings.py

harness/extractors.py

harness/first_pilot_generate.py

harness/format_compliance.py

harness/freeze_items.py

harness/gen_model.py

harness/gen_unit.py

harness/larger_pairs.py

harness/llama_pilot.py

harness/llama_pilot_arc_c.py

harness/make_figures_2_3.py

harness/make_figures_4_5.py

harness/math_equiv.py

harness/mechanism_by_prompt.py

harness/mechanism_main_pairs.py

harness/omega_matched.py

harness/omega_sensitivity.py

harness/parser_departures.py

harness/parser_fidelity.py

harness/parser_reach_matrix.py

harness/parser_reach_nonempty.py

harness/poison_test.py

harness/pooled.py

harness/publish_bounds.py

harness/published_vs_reread.py

harness/qwen_repair_impact.py

harness/qwen_strip_string.py

harness/release_loop.py

harness/release_loop_effect.py

harness/running_example.py

harness/running_example_misses.py

harness/share_checks.py

harness/shipped_parsers.py

harness/transcript_differential.py

harness/verify_upstream_pins.py

harness/with_letter_step.py

records/adjudication-precommitment.CORRECTION.md

records/adjudication-precommitment.json

records/first-pilot/abstaining_extractor.py

records/first-pilot/accuracy_where_read.py

records/first-pilot/extraction_rate_criterion.py

records/first-pilot/extractors_on_grid.py

records/first-pilot/format_share.py

records/first-pilot/generations/gen-base.json

records/first-pilot/generations/gen-post.json

records/first-pilot/outputs_without_answer.py

records/first-pilot/permissive_extractor.py

records/first-pilot/prompts.py

records/first-pilot/reconciliation.json

records/first-pilot/result.json

records/first-pilot/results/accuracy-where-read.json

records/first-pilot/results/extraction-rate-criterion.json

records/first-pilot/results/extractors-on-grid.json

records/first-pilot/results/format-share.json

records/first-pilot/results/outputs-without-answer.json

records/first-pilot/results/prompts.json

records/harness-checks/model-roster.json

records/harness-checks/parser-fidelity.json

records/harness-checks/poison-test.json

records/harness-checks/transcript-differential.json

records/larger-pairs-precommitment.json

records/release-survey.json

records/screening/parsers-transcribed.json

records/screening/rlvr-pairs-verified.json

records/upstream-pin-verification.json

requirements.txt

results/MANIFEST.json

results/MANIFEST.md

results/adjudication-descriptive.json

results/adjudication-result.json

results/adjudication/RESULT.md

results/adjudication/batches.json

results/adjudication/human-kit/README.md

results/adjudication/human-kit/answers-to-fill.csv

results/adjudication/human-kit/items.html

results/adjudication/raw/first/0000.attempt0.json

results/adjudication/raw/first/0001.attempt0.json

results/adjudication/raw/first/0002.attempt0.json

results/adjudication/raw/first/0003.attempt0.json

results/adjudication/raw/first/0004.attempt0.json

results/adjudication/raw/first/0005.attempt0.json

results/adjudication/raw/first/0006.attempt0.json

results/adjudication/raw/first/0007.attempt0.json

results/adjudication/raw/first/0008.attempt0.json

results/adjudication/raw/first/0009.attempt0.json

results/adjudication/raw/first/0010.attempt0.json

results/adjudication/raw/first/0011.attempt0.json

results/adjudication/raw/first/0012.attempt0.json

results/adjudication/raw/first/0013.attempt0.json

results/adjudication/raw/first/0014.attempt0.json

results/adjudication/raw/first/0015.attempt0.json

results/adjudication/raw/first/0016.attempt0.json

results/adjudication/raw/first/0017.attempt0.json

results/adjudication/raw/first/0018.attempt0.json

results/adjudication/raw/first/0019.attempt0.json

results/adjudication/raw/first/0020.attempt0.json

results/adjudication/raw/first/0021.attempt0.json

results/adjudication/raw/first/0022.attempt0.json

results/adjudication/raw/first/0023.attempt0.json

results/adjudication/raw/first/0024.attempt0.json

results/adjudication/raw/first/0025.attempt0.json

results/adjudication/raw/first/0026.attempt0.json

results/adjudication/raw/first/0027.attempt0.json

results/adjudication/raw/first/0028.attempt0.json

results/adjudication/raw/first/0029.attempt0.json

results/adjudication/raw/first/0030.attempt0.json

results/adjudication/raw/first/0031.attempt0.json

results/adjudication/raw/first/0032.attempt0.json

results/adjudication/raw/first/0033.attempt0.json

results/adjudication/raw/first/0034.attempt0.json

results/adjudication/raw/first/0035.attempt0.json

results/adjudication/raw/first/0036.attempt0.json

results/adjudication/raw/first/0037.attempt0.json

results/adjudication/raw/first/0038.attempt0.json

results/adjudication/raw/first/0039.attempt0.json

results/adjudication/raw/first/0040.attempt0.json

results/adjudication/raw/first/0041.attempt0.json

results/adjudication/raw/first/0042.attempt0.json

results/adjudication/raw/first/0043.attempt0.json

results/adjudication/raw/first/0044.attempt0.json

results/adjudication/raw/first/0045.attempt0.json

results/adjudication/raw/first/0046.attempt0.json

results/adjudication/raw/first/0047.attempt0.json

results/adjudication/raw/first/0048.attempt0.json

results/adjudication/raw/first/0049.attempt0.json

results/adjudication/raw/first/0050.attempt0.json

results/adjudication/raw/first/0051.attempt0.json

results/adjudication/raw/first/0052.attempt0.json

results/adjudication/raw/first/0053.attempt0.json

results/adjudication/raw/first/0054.attempt0.json

results/adjudication/raw/first/0055.attempt0.json

results/adjudication/raw/first/0056.attempt0.json

results/adjudication/raw/first/0057.attempt0.json

results/adjudication/raw/first/0058.attempt0.json

results/adjudication/raw/first/0059.attempt0.json

results/adjudication/raw/first/0060.attempt0.json

results/adjudication/raw/first/0061.attempt0.json

results/adjudication/raw/first/0062.attempt0.json

results/adjudication/raw/first/0063.attempt0.json

results/adjudication/raw/first/0064.attempt0.json

results/adjudication/raw/first/0065.attempt0.json

results/adjudication/raw/first/0066.attempt0.json

results/adjudication/raw/first/0067.attempt0.json

results/adjudication/raw/first/0068.attempt0.json

results/adjudication/raw/first/0069.attempt0.json

results/adjudication/raw/first/0070.attempt0.json

results/adjudication/raw/first/0071.attempt0.json

results/adjudication/raw/first/0072.attempt0.json

results/adjudication/raw/first/0073.attempt0.json

results/adjudication/raw/first/0074.attempt0.json

results/adjudication/raw/first/0075.attempt0.json

results/adjudication/raw/first/0076.attempt0.json

results/adjudication/raw/first/0077.attempt0.json

results/adjudication/raw/first/0078.attempt0.json

results/adjudication/raw/first/0079.attempt0.json

results/adjudication/raw/first/0080.attempt0.json

results/adjudication/raw/first/0081.attempt0.json

results/adjudication/raw/first/0082.attempt0.json

results/adjudication/raw/first/0083.attempt0.json

results/adjudication/raw/first/0084.attempt0.json

results/adjudication/raw/first/0085.attempt0.json

results/adjudication/raw/first/0086.attempt0.json

results/adjudication/raw/first/0087.attempt0.json

results/adjudication/raw/first/0088.attempt0.json

results/adjudication/raw/first/0089.attempt0.json

results/adjudication/raw/first/0090.attempt0.json

results/adjudication/raw/first/0091.attempt0.json

results/adjudication/raw/first/0092.attempt0.json

results/adjudication/raw/first/0093.attempt0.json

results/adjudication/raw/first/0094.attempt0.json

results/adjudication/raw/first/0095.attempt0.json

results/adjudication/raw/first/0096.attempt0.json

results/adjudication/raw/first/0097.attempt0.json

results/adjudication/raw/first/0098.attempt0.json

results/adjudication/raw/first/0099.attempt0.json

results/adjudication/raw/first/0100.attempt0.json

results/adjudication/raw/first/0101.attempt0.json

results/adjudication/raw/first/0102.attempt0.json

results/adjudication/raw/first/0103.attempt0.json

results/adjudication/raw/first/0104.attempt0.json

results/adjudication/raw/first/0105.attempt0.json

results/adjudication/raw/first/0106.attempt0.json

results/adjudication/raw/first/0107.attempt0.json

results/adjudication/raw/first/0108.attempt0.json

results/adjudication/raw/first/0109.attempt0.json

results/adjudication/raw/first/0110.attempt0.json

results/adjudication/raw/first/0111.attempt0.json

results/adjudication/raw/first/0112.attempt0.json

results/adjudication/raw/first/0113.attempt0.json

results/adjudication/raw/first/0114.attempt0.json

results/adjudication/raw/first/0114.attempt1.json

results/adjudication/raw/first/0115.attempt0.json

results/adjudication/raw/first/0116.attempt0.json

results/adjudication/raw/first/0117.attempt0.json

results/adjudication/raw/first/0118.attempt0.json

results/adjudication/raw/first/0119.attempt0.json

results/adjudication/raw/first/0120.attempt0.json

results/adjudication/raw/first/0121.attempt0.json

results/adjudication/raw/first/0122.attempt0.json

results/adjudication/raw/first/0123.attempt0.json

results/adjudication/raw/first/0124.attempt0.json

results/adjudication/raw/first/0125.attempt0.json

results/adjudication/raw/first/0126.attempt0.json

results/adjudication/raw/first/0127.attempt0.json

results/adjudication/raw/first/0128.attempt0.json

results/adjudication/raw/first/0129.attempt0.json

results/adjudication/raw/first/0130.attempt0.json

results/adjudication/raw/first/0131.attempt0.json

results/adjudication/raw/first/0132.attempt0.json

results/adjudication/raw/first/0133.attempt0.json

results/adjudication/raw/first/0134.attempt0.json

results/adjudication/raw/first/0135.attempt0.json

results/adjudication/raw/first/0136.attempt0.json

results/adjudication/raw/first/0137.attempt0.json

results/adjudication/raw/first/0137.attempt1.json

results/adjudication/raw/first/0138.attempt0.json

results/adjudication/raw/first/0139.attempt0.json

results/adjudication/raw/first/0140.attempt0.json

results/adjudication/raw/first/0141.attempt0.json

results/adjudication/raw/first/0142.attempt0.json

results/adjudication/raw/first/0143.attempt0.json

results/adjudication/raw/first/0143.attempt1.json

results/adjudication/raw/first/0144.attempt0.json

results/adjudication/raw/first/0145.attempt0.json

results/adjudication/raw/first/0146.attempt0.json

results/adjudication/raw/first/0147.attempt0.json

results/adjudication/raw/first/0148.attempt0.json

results/adjudication/raw/first/0149.attempt0.json

results/adjudication/raw/first/0150.attempt0.json

results/adjudication/raw/first/0151.attempt0.json

results/adjudication/raw/first/0152.attempt0.json

results/adjudication/raw/first/0153.attempt0.json

results/adjudication/raw/first/0154.attempt0.json

results/adjudication/raw/first/0155.attempt0.json

results/adjudication/raw/first/0156.attempt0.json

results/adjudication/raw/first/0157.attempt0.json

results/adjudication/raw/first/0158.attempt0.json

results/adjudication/raw/first/0159.attempt0.json

results/adjudication/raw/first/0160.attempt0.json

results/adjudication/raw/first/0161.attempt0.json

results/adjudication/raw/first/0162.attempt0.json

results/adjudication/raw/first/0163.attempt0.json

results/adjudication/raw/first/0164.attempt0.json

results/adjudication/raw/first/0165.attempt0.json

results/adjudication/raw/first/0166.attempt0.json

results/adjudication/raw/first/0167.attempt0.json

results/adjudication/raw/first/0168.attempt0.json

results/adjudication/raw/first/0169.attempt0.json

results/adjudication/raw/first/0170.attempt0.json

results/adjudication/raw/first/0171.attempt0.json

results/adjudication/raw/first/0172.attempt0.json

results/adjudication/raw/first/0173.attempt0.json

results/adjudication/raw/first/0174.attempt0.json

results/adjudication/raw/first/0175.attempt0.json

results/adjudication/raw/first/0176.attempt0.json

results/adjudication/raw/first/0177.attempt0.json

results/adjudication/raw/first/0178.attempt0.json

results/adjudication/raw/first/0179.attempt0.json

results/adjudication/raw/first/0180.attempt0.json

results/adjudication/raw/first/0181.attempt0.json

results/adjudication/raw/first/0182.attempt0.json

results/adjudication/raw/first/0183.attempt0.json

results/adjudication/raw/first/0184.attempt0.json

results/adjudication/raw/first/0185.attempt0.json

results/adjudication/raw/first/0186.attempt0.json

results/adjudication/raw/first/0187.attempt0.json

results/adjudication/raw/first/0188.attempt0.json

results/adjudication/raw/first/0189.attempt0.json

results/adjudication/raw/first/0190.attempt0.json

results/adjudication/raw/first/0191.attempt0.json

results/adjudication/raw/first/0192.attempt0.json

results/adjudication/raw/first/0193.attempt0.json

results/adjudication/raw/first/0194.attempt0.json

results/adjudication/raw/first/0195.attempt0.json

results/adjudication/raw/first/0196.attempt0.json

results/adjudication/raw/first/0197.attempt0.json

results/adjudication/raw/first/0198.attempt0.json

results/adjudication/raw/first/0199.attempt0.json

results/adjudication/raw/first/0200.attempt0.json

results/adjudication/raw/first/0201.attempt0.json

results/adjudication/raw/first/0202.attempt0.json

results/adjudication/raw/first/0203.attempt0.json

results/adjudication/raw/first/0204.attempt0.json

results/adjudication/raw/first/0205.attempt0.json

results/adjudication/raw/first/0206.attempt0.json

results/adjudication/raw/first/0206.attempt1.json

results/adjudication/raw/first/0207.attempt0.json

results/adjudication/raw/first/0208.attempt0.json

results/adjudication/raw/first/0209.attempt0.json

results/adjudication/raw/first/0210.attempt0.json

results/adjudication/raw/first/0211.attempt0.json

results/adjudication/raw/first/0212.attempt0.json

results/adjudication/raw/first/0213.attempt0.json

results/adjudication/raw/first/0214.attempt0.json

results/adjudication/raw/first/0215.attempt0.json

results/adjudication/raw/first/0216.attempt0.json

results/adjudication/raw/first/0217.attempt0.json

results/adjudication/raw/first/0218.attempt0.json

results/adjudication/raw/first/0219.attempt0.json

results/adjudication/raw/first/0220.attempt0.json

results/adjudication/raw/first/0221.attempt0.json

results/adjudication/raw/first/0222.attempt0.json

results/adjudication/raw/first/0223.attempt0.json

results/adjudication/raw/first/0224.attempt0.json

results/adjudication/raw/first/0225.attempt0.json

results/adjudication/raw/first/0226.attempt0.json

results/adjudication/raw/first/0227.attempt0.json

results/adjudication/raw/first/0228.attempt0.json

results/adjudication/raw/first/0229.attempt0.json

results/adjudication/raw/first/0230.attempt0.json

results/adjudication/raw/first/0231.attempt0.json

results/adjudication/raw/first/0231.attempt1.json

results/adjudication/raw/first/0232.attempt0.json

results/adjudication/raw/first/0233.attempt0.json

results/adjudication/raw/first/0234.attempt0.json

results/adjudication/raw/first/0235.attempt0.json

results/adjudication/raw/first/0236.attempt0.json

results/adjudication/raw/first/0237.attempt0.json

results/adjudication/raw/first/0238.attempt0.json

results/adjudication/raw/first/0239.attempt0.json

results/adjudication/raw/first/0240.attempt0.json

results/adjudication/raw/first/0241.attempt0.json

results/adjudication/raw/first/0242.attempt0.json

results/adjudication/raw/first/0243.attempt0.json

results/adjudication/raw/first/0244.attempt0.json

results/adjudication/raw/first/0245.attempt0.json

results/adjudication/raw/first/0246.attempt0.json

results/adjudication/raw/first/0247.attempt0.json

results/adjudication/raw/first/0248.attempt0.json

results/adjudication/raw/first/0249.attempt0.json

results/adjudication/raw/first/0250.attempt0.json

results/adjudication/raw/first/0251.attempt0.json

results/adjudication/raw/first/0252.attempt0.json

results/adjudication/raw/first/0253.attempt0.json

results/adjudication/raw/first/0254.attempt0.json

results/adjudication/raw/first/0255.attempt0.json

results/adjudication/raw/first/0256.attempt0.json

results/adjudication/raw/first/0257.attempt0.json

results/adjudication/raw/first/0258.attempt0.json

results/adjudication/raw/first/0259.attempt0.json

results/adjudication/raw/first/0260.attempt0.json

results/adjudication/raw/first/0261.attempt0.json

results/adjudication/raw/first/0262.attempt0.json

results/adjudication/raw/first/0263.attempt0.json

results/adjudication/raw/first/0264.attempt0.json

results/adjudication/raw/first/0265.attempt0.json

results/adjudication/raw/first/0266.attempt0.json

results/adjudication/raw/first/0267.attempt0.json

results/adjudication/raw/first/0268.attempt0.json

results/adjudication/raw/first/0269.attempt0.json

results/adjudication/raw/first/0270.attempt0.json

results/adjudication/raw/first/0271.attempt0.json

results/adjudication/raw/second/0000.attempt0.json

results/adjudication/raw/second/0001.attempt0.json

results/adjudication/raw/second/0002.attempt0.json

results/adjudication/raw/second/0003.attempt0.json

results/adjudication/raw/second/0004.attempt0.json

results/adjudication/raw/second/0005.attempt0.json

results/adjudication/raw/second/0006.attempt0.json

results/adjudication/raw/second/0007.attempt0.json

results/adjudication/raw/second/0008.attempt0.json

results/adjudication/raw/second/0009.attempt0.json

results/adjudication/raw/second/0010.attempt0.json

results/adjudication/raw/second/0011.attempt0.json

results/adjudication/raw/second/0012.attempt0.json

results/adjudication/raw/second/0013.attempt0.json

results/adjudication/raw/second/0014.attempt0.json

results/adjudication/raw/second/0015.attempt0.json

results/adjudication/raw/second/0016.attempt0.json

results/adjudication/raw/second/0017.attempt0.json

results/adjudication/raw/second/0018.attempt0.json

results/adjudication/raw/second/0019.attempt0.json

results/adjudication/raw/second/0020.attempt0.json

results/adjudication/raw/second/0021.attempt0.json

results/adjudication/raw/second/0022.attempt0.json

results/adjudication/raw/second/0023.attempt0.json

results/adjudication/raw/second/0024.attempt0.json

results/adjudication/raw/second/0025.attempt0.json

results/adjudication/raw/second/0026.attempt0.json

results/adjudication/raw/second/0027.attempt0.json

results/adjudication/raw/second/0028.attempt0.json

results/adjudication/raw/second/0029.attempt0.json

results/adjudication/raw/second/0030.attempt0.json

results/adjudication/raw/second/0031.attempt0.json

results/adjudication/raw/second/0032.attempt0.json

results/adjudication/raw/second/0033.attempt0.json

results/adjudication/raw/second/0034.attempt0.json

results/adjudication/raw/second/0035.attempt0.json

results/adjudication/raw/second/0036.attempt0.json

results/adjudication/raw/second/0037.attempt0.json

results/adjudication/raw/second/0038.attempt0.json

results/adjudication/raw/second/0039.attempt0.json

results/adjudication/raw/second/0040.attempt0.json

results/adjudication/raw/second/0041.attempt0.json

results/adjudication/raw/second/0042.attempt0.json

results/adjudication/raw/second/0043.attempt0.json

results/adjudication/raw/second/0044.attempt0.json

results/adjudication/raw/second/0045.attempt0.json

results/adjudication/raw/second/0046.attempt0.json

results/adjudication/raw/second/0047.attempt0.json

results/adjudication/raw/second/0048.attempt0.json

results/adjudication/raw/second/0049.attempt0.json

results/adjudication/raw/second/0050.attempt0.json

results/adjudication/raw/second/0051.attempt0.json

results/adjudication/raw/second/0052.attempt0.json

results/adjudication/raw/second/0053.attempt0.json

results/adjudication/raw/second/0054.attempt0.json

results/adjudication/raw/second/0055.attempt0.json

results/adjudication/raw/second/0056.attempt0.json

results/adjudication/raw/second/0057.attempt0.json

results/adjudication/raw/second/0058.attempt0.json

results/adjudication/raw/second/0059.attempt0.json

results/adjudication/raw/second/0060.attempt0.json

results/adjudication/raw/second/0061.attempt0.json

results/adjudication/raw/second/0062.attempt0.json

results/adjudication/raw/second/0063.attempt0.json

results/adjudication/raw/second/0064.attempt0.json

results/adjudication/raw/second/0065.attempt0.json

results/adjudication/raw/second/0066.attempt0.json

results/adjudication/raw/second/0067.attempt0.json

results/adjudication/raw/second/0068.attempt0.json

results/adjudication/raw/second/0069.attempt0.json

results/adjudication/raw/second/0070.attempt0.json

results/adjudication/raw/second/0071.attempt0.json

results/adjudication/raw/second/0072.attempt0.json

results/adjudication/raw/second/0073.attempt0.json

results/adjudication/raw/second/0074.attempt0.json

results/adjudication/raw/second/0075.attempt0.json

results/adjudication/raw/second/0076.attempt0.json

results/adjudication/raw/second/0077.attempt0.json

results/adjudication/raw/second/0078.attempt0.json

results/adjudication/raw/second/0079.attempt0.json

results/adjudication/raw/second/0080.attempt0.json

results/adjudication/raw/second/0081.attempt0.json

results/adjudication/raw/second/0082.attempt0.json

results/adjudication/raw/second/0083.attempt0.json

results/adjudication/raw/second/0084.attempt0.json

results/adjudication/raw/second/0085.attempt0.json

results/adjudication/raw/second/0086.attempt0.json

results/adjudication/raw/second/0087.attempt0.json

results/adjudication/raw/second/0088.attempt0.json

results/adjudication/raw/second/0089.attempt0.json

results/adjudication/raw/second/0090.attempt0.json

results/adjudication/raw/second/0091.attempt0.json

results/adjudication/raw/second/0092.attempt0.json

results/adjudication/raw/second/0093.attempt0.json

results/adjudication/raw/second/0094.attempt0.json

results/adjudication/raw/second/0095.attempt0.json

results/adjudication/raw/second/0096.attempt0.json

results/adjudication/raw/second/0097.attempt0.json

results/adjudication/raw/second/0098.attempt0.json

results/adjudication/raw/second/0099.attempt0.json

results/adjudication/raw/second/0100.attempt0.json

results/adjudication/raw/second/0101.attempt0.json

results/adjudication/raw/second/0102.attempt0.json

results/adjudication/raw/second/0103.attempt0.json

results/adjudication/raw/second/0104.attempt0.json

results/adjudication/raw/second/0105.attempt0.json

results/adjudication/raw/second/0106.attempt0.json

results/adjudication/raw/second/0107.attempt0.json

results/adjudication/raw/second/0108.attempt0.json

results/adjudication/raw/second/0109.attempt0.json

results/adjudication/raw/second/0110.attempt0.json

results/adjudication/raw/second/0111.attempt0.json

results/adjudication/raw/second/0112.attempt0.json

results/adjudication/raw/second/0113.attempt0.json

results/adjudication/raw/second/0114.attempt0.json

results/adjudication/raw/second/0115.attempt0.json

results/adjudication/raw/second/0116.attempt0.json

results/adjudication/raw/second/0117.attempt0.json

results/adjudication/raw/second/0118.attempt0.json

results/adjudication/raw/second/0119.attempt0.json

results/adjudication/raw/second/0120.attempt0.json

results/adjudication/raw/second/0121.attempt0.json

results/adjudication/raw/second/0122.attempt0.json

results/adjudication/raw/second/0123.attempt0.json

results/adjudication/raw/second/0124.attempt0.json

results/adjudication/raw/second/0125.attempt0.json

results/adjudication/raw/second/0126.attempt0.json

results/adjudication/raw/second/0127.attempt0.json

results/adjudication/raw/second/0127.attempt1.json

results/adjudication/raw/second/0128.attempt0.json

results/adjudication/raw/second/0129.attempt0.json

results/adjudication/raw/second/0130.attempt0.json

results/adjudication/raw/second/0131.attempt0.json

results/adjudication/raw/second/0132.attempt0.json

results/adjudication/raw/second/0133.attempt0.json

results/adjudication/raw/second/0134.attempt0.json

results/adjudication/raw/second/0135.attempt0.json

results/adjudication/raw/second/0136.attempt0.json

results/adjudication/raw/second/0137.attempt0.json

results/adjudication/raw/second/0138.attempt0.json

results/adjudication/raw/second/0139.attempt0.json

results/adjudication/raw/second/0140.attempt0.json

results/adjudication/raw/second/0141.attempt0.json

results/adjudication/raw/second/0142.attempt0.json

results/adjudication/raw/second/0143.attempt0.json

results/adjudication/raw/second/0144.attempt0.json

results/adjudication/raw/second/0145.attempt0.json

results/adjudication/raw/second/0146.attempt0.json

results/adjudication/raw/second/0147.attempt0.json

results/adjudication/raw/second/0148.attempt0.json

results/adjudication/raw/second/0149.attempt0.json

results/adjudication/raw/second/0150.attempt0.json

results/adjudication/raw/second/0151.attempt0.json

results/adjudication/raw/second/0152.attempt0.json

results/adjudication/raw/second/0153.attempt0.json

results/adjudication/raw/second/0154.attempt0.json

results/adjudication/raw/second/0155.attempt0.json

results/adjudication/raw/second/0156.attempt0.json

results/adjudication/raw/second/0157.attempt0.json

results/adjudication/raw/second/0158.attempt0.json

results/adjudication/raw/second/0159.attempt0.json

results/adjudication/raw/second/0160.attempt0.json

results/adjudication/raw/second/0161.attempt0.json

results/adjudication/raw/second/0162.attempt0.json

results/adjudication/raw/second/0163.attempt0.json

results/adjudication/raw/second/0164.attempt0.json

results/adjudication/raw/second/0165.attempt0.json

results/adjudication/raw/second/0166.attempt0.json

results/adjudication/raw/second/0167.attempt0.json

results/adjudication/raw/second/0168.attempt0.json

results/adjudication/raw/second/0169.attempt0.json

results/adjudication/raw/second/0170.attempt0.json

results/adjudication/raw/second/0171.attempt0.json

results/adjudication/raw/second/0172.attempt0.json

results/adjudication/raw/second/0173.attempt0.json

results/adjudication/raw/second/0173.attempt1.json

results/adjudication/raw/second/0174.attempt0.json

results/adjudication/raw/second/0175.attempt0.json

results/adjudication/raw/second/0176.attempt0.json

results/adjudication/raw/second/0177.attempt0.json

results/adjudication/raw/second/0177.attempt1.json

results/adjudication/raw/second/0178.attempt0.json

results/adjudication/raw/second/0179.attempt0.json

results/adjudication/raw/second/0180.attempt0.json

results/adjudication/raw/second/0181.attempt0.json

results/adjudication/raw/second/0182.attempt0.json

results/adjudication/raw/second/0183.attempt0.json

results/adjudication/raw/second/0184.attempt0.json

results/adjudication/raw/second/0185.attempt0.json

results/adjudication/raw/second/0186.attempt0.json

results/adjudication/raw/second/0187.attempt0.json

results/adjudication/raw/second/0188.attempt0.json

results/adjudication/raw/second/0189.attempt0.json

results/adjudication/raw/second/0190.attempt0.json

results/adjudication/raw/second/0191.attempt0.json

results/adjudication/raw/second/0192.attempt0.json

results/adjudication/raw/second/0193.attempt0.json

results/adjudication/raw/second/0194.attempt0.json

results/adjudication/raw/second/0195.attempt0.json

results/adjudication/raw/second/0196.attempt0.json

results/adjudication/raw/second/0197.attempt0.json

results/adjudication/raw/second/0198.attempt0.json

results/adjudication/raw/second/0199.attempt0.json

results/adjudication/raw/second/0200.attempt0.json

results/adjudication/raw/second/0201.attempt0.json

results/adjudication/raw/second/0202.attempt0.json

results/adjudication/raw/second/0203.attempt0.json

results/adjudication/raw/second/0204.attempt0.json

results/adjudication/raw/second/0205.attempt0.json

results/adjudication/raw/second/0206.attempt0.json

results/adjudication/raw/second/0207.attempt0.json

results/adjudication/raw/second/0208.attempt0.json

results/adjudication/raw/second/0209.attempt0.json

results/adjudication/raw/second/0210.attempt0.json

results/adjudication/raw/second/0211.attempt0.json

results/adjudication/raw/second/0212.attempt0.json

results/adjudication/raw/second/0213.attempt0.json

results/adjudication/raw/second/0214.attempt0.json

results/adjudication/raw/second/0215.attempt0.json

results/adjudication/raw/second/0216.attempt0.json

results/adjudication/raw/second/0217.attempt0.json

results/adjudication/raw/second/0218.attempt0.json

results/adjudication/raw/second/0219.attempt0.json

results/adjudication/raw/second/0220.attempt0.json

results/adjudication/raw/second/0221.attempt0.json

results/adjudication/raw/second/0222.attempt0.json

results/adjudication/raw/second/0223.attempt0.json

results/adjudication/raw/second/0224.attempt0.json

results/adjudication/raw/second/0225.attempt0.json

results/adjudication/raw/second/0226.attempt0.json

results/adjudication/raw/second/0227.attempt0.json

results/adjudication/raw/second/0228.attempt0.json

results/adjudication/raw/second/0228.attempt1.json

results/adjudication/raw/second/0229.attempt0.json

results/adjudication/raw/second/0230.attempt0.json

results/adjudication/raw/second/0231.attempt0.json

results/adjudication/raw/second/0232.attempt0.json

results/adjudication/raw/second/0233.attempt0.json

results/adjudication/raw/second/0234.attempt0.json

results/adjudication/raw/second/0235.attempt0.json

results/adjudication/raw/second/0236.attempt0.json

results/adjudication/raw/second/0237.attempt0.json

results/adjudication/raw/second/0238.attempt0.json

results/adjudication/raw/second/0239.attempt0.json

results/adjudication/raw/second/0240.attempt0.json

results/adjudication/raw/second/0241.attempt0.json

results/adjudication/raw/second/0242.attempt0.json

results/adjudication/raw/second/0243.attempt0.json

results/adjudication/raw/second/0244.attempt0.json

results/adjudication/raw/second/0245.attempt0.json

results/adjudication/raw/second/0246.attempt0.json

results/adjudication/raw/second/0247.attempt0.json

results/adjudication/raw/second/0248.attempt0.json

results/adjudication/raw/second/0248.attempt1.json

results/adjudication/raw/second/0249.attempt0.json

results/adjudication/raw/second/0250.attempt0.json

results/adjudication/raw/second/0251.attempt0.json

results/adjudication/raw/second/0252.attempt0.json

results/adjudication/raw/second/0253.attempt0.json

results/adjudication/raw/second/0254.attempt0.json

results/adjudication/raw/second/0255.attempt0.json

results/adjudication/raw/second/0256.attempt0.json

results/adjudication/raw/second/0257.attempt0.json

results/adjudication/raw/second/0258.attempt0.json

results/adjudication/raw/second/0259.attempt0.json

results/adjudication/raw/second/0260.attempt0.json

results/adjudication/raw/second/0261.attempt0.json

results/adjudication/raw/second/0261.attempt1.json

results/adjudication/raw/second/0261.attempt2.json

results/adjudication/raw/second/0262.attempt0.json

results/adjudication/raw/second/0263.attempt0.json

results/adjudication/raw/second/0264.attempt0.json

results/adjudication/raw/second/0265.attempt0.json

results/adjudication/raw/second/0266.attempt0.json

results/adjudication/raw/second/0267.attempt0.json

results/adjudication/raw/second/0268.attempt0.json

results/adjudication/raw/second/0269.attempt0.json

results/adjudication/raw/second/0270.attempt0.json

results/adjudication/raw/second/0271.attempt0.json

results/adjudication/raw/tie_break/0000.attempt0.json

results/adjudication/raw/tie_break/0001.attempt0.json

results/adjudication/raw/tie_break/0002.attempt0.json

results/adjudication/raw/tie_break/0003.attempt0.json

results/adjudication/raw/tie_break/0004.attempt0.json

results/adjudication/raw/tie_break/0005.attempt0.json

results/adjudication/reader-first.jsonl

results/adjudication/reader-second.jsonl

results/adjudication/reader-tie_break.jsonl

results/adjudication/run-first.log

results/adjudication/run-second.log

results/adjudication/run-tie_break.log

results/adjudication/sample.jsonl

results/analysis-before-qwen-repair.json

results/analysis-grid-and-larger-pairs.json

results/analysis-grid.json

results/analysis-token-limit-rule.json

results/answer-tags.json

results/boxing-by-prompt.json

results/by-condition.json

results/common-cause.json

results/correctness-criterion.json

results/denominator-weight.json

results/dose-response-pooled.json

results/dose-response.json

results/external-agreement-direction.json

results/external-agreement.json

results/external-matched.json

results/external-reference.json

results/extraction-rate-settings.json

results/format-compliance.json

results/larger-pairs.json

results/llama-pilot-arc_c.json

results/llama-pilot.json

results/mechanism-by-prompt.json

results/mechanism-main-pairs.json

results/mechanism.json

results/omega-matched.json

results/omega-sensitivity.json

results/parser-departures.json

results/parser-reach-matrix-nonempty.json

results/parser-reach-matrix.json

results/published-bounds.json

results/published-vs-reread.json

results/qwen-repair-impact.json

results/release-loop.json

results/running-example-misses.json

results/running-example.json

results/share-checks.json

results/timings.json

results/truncation-check.json

results/with-letter-step/INDEX.json

results/with-letter-step/adjudication-result.json

results/with-letter-step/analysis-grid-and-larger-pairs.json

results/with-letter-step/analysis-grid.json

results/with-letter-step/by-condition.json

results/with-letter-step/common-cause.json

results/with-letter-step/denominator-weight.json

results/with-letter-step/dose-response-pooled.json

results/with-letter-step/dose-response.json

results/with-letter-step/external-matched.json

results/with-letter-step/external-reference.json

results/with-letter-step/extraction-rate-settings.json

results/with-letter-step/larger-pairs.json

results/with-letter-step/llama-pilot-arc_c.json

results/with-letter-step/llama-pilot.json

results/with-letter-step/parser-reach-matrix-nonempty.json

results/with-letter-step/parser-reach-matrix.json

results/with-letter-step/published-bounds.json

results/with-letter-step/published-vs-reread.json

results/with-letter-step/running-example-misses.json

results/with-letter-step/share-checks.json

runs/llama-pilot-arc_c/out/arc_c.base.jsonl

runs/llama-pilot-arc_c/out/arc_c.base.meta.json

runs/llama-pilot-arc_c/out/arc_c.rl.jsonl

runs/llama-pilot-arc_c/out/arc_c.rl.meta.json

runs/llama-pilot-mmlu/out/arc_c.base.jsonl

runs/llama-pilot-mmlu/out/arc_c.base.meta.json

runs/llama-pilot-mmlu/out/arc_c.rl.jsonl

runs/llama-pilot-mmlu/out/arc_c.rl.meta.json

runs/llama-pilot-mmlu/out/mmlu.base.jsonl

runs/llama-pilot-mmlu/out/mmlu.base.meta.json

runs/llama-pilot-mmlu/out/mmlu.rl.jsonl

runs/llama-pilot-mmlu/out/mmlu.rl.meta.json

runs/llama-pilot-mmlu/out/mmlu_pro.base.jsonl

runs/llama-pilot-mmlu/out/mmlu_pro.base.meta.json

runs/llama-pilot-mmlu/out/mmlu_pro.rl.jsonl

runs/llama-pilot-mmlu/out/mmlu_pro.rl.meta.json

(744 additional files not shown)

Additional Features

Audio Summary

Current browse context:

cs.LG

new | recent | 2026-10

Change to browse by:

cs cs.CL

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender (What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with