212k AI coding benchmarks: context beats generic prompts

Updated June 2026. This guide was originally based on 1,458 Python benchmarks. Since then, we’ve run 212,000+ benchmarks across Python, Go, JavaScript, and C#—testing chain-of-thought, politeness, personas, context pollution, model selection, verification instructions, multi-turn iteration, language-specific rules, /init optimization, emotional stakes, output compression, and more. The conclusions have evolved significantly—especially across languages. See the capstone v2 synthesis for the full cross-language picture.

This post distills all of that data into a practical guide. Not opinions. Not “best practices” passed around on social media. Numbers from controlled experiments with statistical analysis, run across Claude Haiku, Sonnet, and Opus on 4 programming languages.

If you want the data, everything is open source at claude-benchmark. If you want the conclusions, keep reading.

Finding the cross-language tasks: Tasks for each language follow a naming convention: bug-fix-01 (Python), bug-fix-01-go (Go), bug-fix-01-js (JavaScript), bug-fix-01-cs (C#). All 48 tasks (12 per language) are in the tasks/builtin/ directory. Cross-language experiments use experiments/cross-language.toml and experiments/cot-cross-language.toml.

Cross-Language Update: The original seven rules below were derived from Python-only testing. Cross-language experiments revealed that several findings flip by language. Where this matters, each rule now includes language-specific caveats. The biggest shift: the Baseline Competence Theory—Python’s high baseline means most advice is noise, but Go, JS, and C# benefit from instructions the model doesn’t already internalize.

The Uncomfortable Baseline

Let’s start with the finding that frames everything else: across 1,458 runs, 13 prompt configurations, and 3 models, no configuration consistently beat an empty prompt.

Configuration Tokens Composite Score Delta from Empty

empty (no instructions) 0 92.15 —

micro-quality 74 91.61 -0.54

refactor-aware 105 91.34 -0.81

workflow 152 91.10 -1.04

anti-pattern 125 90.95 -1.19

judge-aligned 148 90.70 -1.44

typical-readable 753 91.08 -1.07

typical-compressed ~650 90.80 -1.35

large-readable ~6,800 91.29 -0.86

large-compressed ~5,900 90.57 -1.58

bare + cot-prefix ~8 94.7* -0.56*

bare + cot-detailed ~25 94.6* -0.73*

• CoT experiment used a different task subset (3 tasks vs. 12), so absolute scores aren’t directly comparable. The relative delta from their own bare baseline is the meaningful number.

The correlation between instruction token count and composite quality score across the Phase 2 profiles was r = -0.95. Nearly perfect negative correlation. Every additional token of instruction marginally decreased output quality.

The most common prompt engineering advice is actively counterproductive for coding tasks. Adding style rules, chain-of-thought instructions, persona definitions, or formatting guidelines to your system prompt does not help Claude write better code on generic tasks. It adds noise to a signal the model already has.

That’s the uncomfortable starting point. Now let’s talk about when and how instructions do help, because the story is more nuanced than “never use a system prompt.”

The Seven Rules

From 1,458 runs and 13 configurations, seven principles emerged. Each is backed by specific data points. I’ve organized them from highest impact to lowest.

Rule 1: Don’t Teach Claude How to Code

Remove any instruction that restates general programming knowledge.

Claude’s training already encodes Python best practices, clean code principles, naming conventions, design patterns, and edge-case awareness. Instructions like “use snake_case for variables,” “write clear docstrings,” or “handle edge cases” are redundant. They don’t make the model try harder—they add tokens that dilute the actual task.

The micro-quality profile tested this directly. It contained just four bullet points of universal coding advice:

Write clear, self-documenting code with descriptive names

Handle edge cases explicitly with informative error messages

Follow task instructions precisely

Keep functions focused and short

These are principles so universal they seem impossible to disagree with. They still reduced quality by 0.54 points. The instructions create a scorecard that can only subtract points, never add them.

Evidence: micro-quality (74 tokens, -0.54), typical-readable (753 tokens with style/naming/formatting rules, -1.07), large-readable (6,800 tokens of comprehensive guidelines, -0.86). All negative. Phase 2, 648 runs.

Don’t include

“Use snake_case for variables”

“Write docstrings for public functions”

“Handle edge cases”

“Follow SOLID principles”

“Use descriptive variable names”

“Keep functions short”

Do include

“Use Pydantic v2 model_validator, not v1 validator”

“Our API returns snake_case JSON, not camelCase”

“Error responses must include a trace_id field”

Things Claude cannot know from training

Rule 2: Don’t Tell Claude How to Think

Skip chain-of-thought instructions for code generation.

“Think step by step” is the most widely repeated prompt engineering advice on the internet. It works on math and logic puzzles. It does not work for coding. In our 270-run experiment, every CoT variant scored lower than the bare baseline on every model.

More explicit reasoning instructions produced worse results than lighter ones. cot-detailed (“analyze requirements step by step, consider edge cases, then implement”) scored lower than cot-prefix (“think step by step”), which scored lower than no instruction at all. The more structure you impose on the model’s reasoning, the more you constrain its natural problem-solving.

Evidence: cot-prefix averaged -0.56 from bare; cot-detailed averaged -0.73. Opus suffered the most (-1.14 for cot-detailed). On refactor-01, Opus dropped 3.5 points (86.8 → 83.3). CoT experiment, 270 runs.

The chess grandmaster analogy holds: giving a grandmaster a “first consider all captures, then consider all checks” checklist doesn’t help them—it fragments the holistic pattern recognition they’re better at. Claude already reasons through implementation. Explicit CoT instructions fragment that reasoning.

Cross-language update: 2,880 cross-language runs revealed CoT helps Go (+5.3) and C# (+7.7) while hurting Python (-0.5). The model’s baseline competence in Python makes CoT redundant, but for languages where the model is less confident, structured reasoning provides scaffolding. If you write Go or C#, add a CoT prefix. If you write Python, skip it.

Rule 3: Tell Claude What It Doesn’t Know

Instructions should encode project-specific knowledge, not programming knowledge.

The one category of information that consistently helped was knowledge Claude cannot have from training: your project’s architecture, your team’s conventions, your deployment constraints, your domain’s terminology.

The typical-readable profile contained build commands (make build, make test), project structure (src/ for code, tests/ mirrors source), and workflow conventions (commit message format, branch naming). These are exactly the kind of project-specific facts that help Claude navigate a real codebase. On their own they still showed an overall negative delta (-1.07), but this is because they were bundled with generic style rules that diluted the signal.

Evidence: Build commands and project structure are the type of knowledge that helps in multi-file, real-world sessions (not captured in this benchmark’s single-file tasks). The negative result on generic coding tasks confirms that generic rules hurt; it doesn’t invalidate project-specific context. See Caveats.

The ideal CLAUDE.md is a project knowledge base, not a coding style guide. Think of it as onboarding documentation for a new team member who already knows how to code but doesn’t know your codebase.

Rule 4: Use Positive Framing, Never Prohibitions

Say “do X” not “don’t do Y.”

The anti-pattern profile framed guidance as prohibitions: “do not leave dead code,” “do not use overly clever one-liners,” “do not create deeply nested code.” The micro-quality profile framed equivalent guidance as positive directives: “write clear, self-documenting code,” “keep functions focused and short.”

Positive framing outperformed negative framing by 0.66 points (91.61 vs 90.95). Negative instructions appear to prime the model toward the failure mode being described rather than away from it. When you say “do not write deeply nested code,” the model’s attention is drawn to deeply nested code as a concept—and that attention leak shows up in the output.

Evidence: micro-quality (positive, 91.61) vs. anti-pattern (negative, 90.95). Both ~100-125 tokens, same intent, different framing. Positive won by 0.66. Phase 2, 648 runs.

Cross-language update: The capstone v2 experiment (20,499 scored runs across 4 languages) tested positive vs. negative framing as an isolated variable. The difference was noise—no statistically significant advantage for either framing direction. The original 0.66-point gap was confounded by content differences between the micro-quality and anti-pattern profiles, not by framing valence itself. Positive framing is still fine default practice, but don’t stress about rewording existing negative rules.

Emotional stakes update: The emotional stakes experiment (15,120 runs) tested a related axis: does adding emotional weight to your framing help? Growth-mindset framing wins (+1.88 vs neutral)—”This is a great opportunity to demonstrate clean code.” But high-pressure framing backfires: life-or-death urgency hurt by -3.33 points. The takeaway: positive framing matters less in valence (do/don’t) and more in emotional register (encouraging vs. threatening).

Negative (worse)

“Do not leave dead code”

“Do not use overly clever one-liners”

“Do not create deeply nested code”

“Do not ignore edge cases”

“Avoid abbreviations”

Positive (better)

“Remove unused code before finishing”

“Prefer clear multi-line logic over one-liners”

“Use guard clauses and early returns”

“Handle every edge case mentioned in the requirements”

“Use descriptive names”

Rule 5: Target Specific Weaknesses, Not General Quality

Instructions hurt on easy tasks and help on hard ones. Write them for the hard parts.

The overall averages hide the most actionable finding in the data. Instructions consistently hurt on tasks Claude already aces (bug fixes, code generation) and helped on tasks where it struggles (refactoring, instruction-following).

The largest improvement across all 1,188 Phase 1+2 runs: the workflow profile lifted Opus’s instruction-following score by +5.80 points. A structured “read the task, plan your approach, verify your solution” checklist helped Opus dramatically on tasks requiring precise specification compliance. And it didn’t just raise the average—it raised the floor from 61.4 to 83.5, eliminating catastrophic low-scoring runs.

Evidence: workflow +5.80 on Opus/instruction tasks. refactor-aware +2.54 on Haiku/refactor. micro-quality +4.41 on Opus/instruction. Meanwhile, these same profiles all showed negative deltas on bug-fix and code-gen tasks. Phase 2, 648 runs.

This means the value of instructions is highly context-dependent. A CLAUDE.md that helps on refactoring tasks may hurt on bug-fix tasks. The right strategy is to use targeted instructions only for the task types where you observe weakness, and omit them otherwise.

Rule 6: Keep the Formatting

Markdown headers, bullet points, and whitespace help smaller models parse instructions.

I originally advocated stripping markdown formatting from CLAUDE.md files to save tokens. The benchmark showed this was wrong for two of three models.

The readable version of the typical profile outperformed the compressed version on Haiku (-1.56 delta) and Sonnet (-0.86 delta). Only Opus was indifferent (+0.19). For the large p

[truncated for AI cost control]