Ponytail Skill for Claude Code: Does It Really Cut Agent Code by 54%?

JetBrains AI

Supercharge your tools with AI-powered features inside many JetBrains products

Follow

Follow:

RSS RSS

Explore More

Agentic AI AI AI Assistant

Ponytail Skill for Claude Code: Does It Really Cut Agent Code by 54%?

Part 3 of a series where we take public “token saver” add-ons for coding agents and run the same paired A/B benchmark against each of them. Part 1 was the caveman skill (advertised −65%, measured −8.5%). Part 2 was rtk (advertised −60–90%, measured +7.6%).

We ran 80 paired tasks to test the ponytail skill for Claude Code. Advertised: −54% code, -22% tokens, -20% cost, -27% time. Measured: −15% code, −10.3% cost and -11% time. Here’s what actually happened.

Real savings, although roughly a quarter to a half of what is advertised, it is the first tool in this series with a statistically solid cost-saving signal. We found no quality difference, though ~80 pairs can only rule out large ones. The catch: the code cut only shows up where there was room to over-build.

Why we ran this

Ponytail skill is designed to make AI agents write less code. Its core premise: a senior developer who has seen everything replaces your fifty lines with one. Ask for a date picker and instead of installing flatpickr and writing a wrapper component, it writes and moves on.

Mechanically it is a ladder the model climbs before writing anything. Does this need to exist at all? Is it already in the codebase? Does the standard library do it? A native platform feature? An installed dependency? Can it be one line? Only then: write the minimum that works. The ladder runs after understanding the problem, not instead of it, and validation, error handling, security and accessibility are explicitly off the chopping block.

Here is what that looks like in practice, from our own run. Both agents were asked to export a three.js scene to a Blender-ready OBJ file; both produced a file the verifier accepted. Both wrote the same fiddly loop to expand instanced meshes, because three.js’s OBJExporter cannot handle them. The difference is everything around that loop. To rotate the scene into Blender’s orientation and write it out, the plain agent builds a wrapper object to hold the rotation and names every intermediate step:

// no skill — 10 statements to rotate the scene and write the file const exportRoot = new THREE.Group(); exportRoot.name = 'blender_export_root'; exportRoot.rotation.x = -Math.PI / 2; exportRoot.add(root); exportRoot.updateMatrixWorld(true);

const exporter = new OBJExporter(); const objString = exporter.parse(exportRoot);

const outputPath = '/root/output/object.obj'; fs.mkdirSync(path.dirname(outputPath), { recursive: true }); fs.writeFileSync(outputPath, objString);

// ponytail — the same job, 5 statements root.rotation.x = -Math.PI / 2; root.updateMatrixWorld(true);

const obj = new OBJExporter().parse(root); fs.mkdirSync('/root/output', { recursive: true }); fs.writeFileSync('/root/output/object.obj', obj);

Nothing was sacrificed there. Ponytail rotated the object it already had instead of building a parent to rotate it for it, and skipped an import while it was at it. Ten statements became five, both files exported the same geometry, and both scored 1.0. That is the effect working exactly as advertised — on one file, on one task.

The headline claim is −54% code, plus −22% tokens, −20% cost and −27% time. What made this one worth testing is that the claim is unusually well documented. The authors rebuilt their benchmark in response to a critique (issue #126) that their original numbers came from a chatty baseline, and they publish the honest version: a real headless Claude Code session editing a real FastAPI + React repo, scored on the git diff it leaves behind. They even document a contamination bug they found in their own harness, where a SessionStart hook was firing on every arm and secretly running ponytail in the baseline.

That is more methodological candour than most tools in this space manage. So the question here is whether the effect survives a benchmark the authors did not choose, on a stronger model, with verifier-scored quality.

Setup

HarnessHarbor 0.18 — Docker sandboxes, task verifiers, paired runs

AgentClaude Code 2.1.201, headless, bypassPermissions, pinned in both arms

Modelclaude-sonnet-5 at medium reasoning effort

BenchmarkSkillsBench, 80 paired tasks, auto-graded 0–1 with partial credit

Arm Astock Claude Code

Arm Bponytail v4.8.4: skill installed and its ruleset injected, byte-identical to the ruleset text its own SessionStart hook generates (the hook’s other first-run output is not reproduced). A close emulation of the shipped plugin’s full mode, with three documented differences (no first-run statusline nudge, no subagent re-injection, ruleset appended after the task rather than before it)

Volume3 paired stages (10-task smoke, same 10 at k=3, full 80), plus self-activation and wiring checks — 251 billed agent trials in the complete evaluation program, USD 246.09. A few SkillsBench tasks are excluded: one that cannot run in a local sandbox, and a handful that fail identically in both arms on our hardware

One detail matters more than it looks. We generated arm B’s injected text by calling ponytail’s own hooks/ponytail-instructions.js rather than writing a summary of it, so the ruleset the model saw is the skill’s own text rather than our paraphrase of it. That covers the ruleset the hook generates, not every side-effect the hook has on a real first run. Every with-ponytail trial is audited afterwards to confirm the ruleset actually reached the model; every baseline trial is audited to confirm it did not. That check is the direct descendant of the contamination bug ponytail found in its own benchmark, and it came back clean: 100% of treatment trials, 0% of baselines.

Finding 1 — Does ponytail skill self-activate in Claude Code?

Before the paid runs we tested the obvious install path: drop the skill in and let Claude Code decide when to use it. Ponytail’s description invites exactly that, telling the model to use it on “ANY coding task: writing, adding, refactoring, fixing, reviewing, or designing code.”

Across all ten sessions it self-activated zero times. Not rarely. Never. The skill sat installed and visible and the model did not once reach for it.

This is not a bug in ponytail, and it is why the tool ships as a plugin with a SessionStart hook that injects the ruleset whether you ask or not. But it does mean the install method decides whether you get anything at all. Copy the SKILL.md into a skills folder and you will very likely measure nothing. Every number below comes from the arm where the ruleset is actually injected.

Finding 2 — the observed code cut is a third of the advertised size

Across 80 paired tasks a typical task shed 15.4% of the code the agent wrote; in total, 10,205 lines became 8,756. That is a substantial observed reduction. At p=0.088, however, it is the softest of our headline numbers. It is also nowhere near 54%.

Two things to say about the gap, both fair to the tool. First, their −54% is a mean across twelve hand-picked feature tickets; ours is a median across 80 tasks nobody chose for this purpose. Means and medians on skewed data are different animals, and their own writeup is explicit that the figure “reaches 94% where an agent over-builds and is near zero where the code is already minimal.”

Second, and this is the more interesting half: our own data points the same way.

Finding 3 — the saving concentrates where there was room to over-build

Split the tasks by how much code the baseline wrote. Ponytail cannot pick its own bucket that way, since the plain agent decides it. Worth saying plainly though: we chose these thresholds after seeing the data, and grouping by the baseline’s own output can stretch a gradient like this on its own. Read the chart as a strong hint about where the effect lives, not as a measured law.

On big builds the cut reaches −31%. On tasks where the plain agent already wrote almost nothing, the typical task moved by zero — though the totals in that group actually rose, 104 lines to 910, and that gap is where the run’s one real surprise turned up.

On seven tasks our counter recorded zero lines for the plain agent and 51 to 230 for ponytail. Reading the transcripts, that gap is mostly about where the code lived rather than how much of it there was. The plain agent piped its solution straight into a Python interpreter as a heredoc, which produced the deliverable and left no script behind. Ponytail wrote the same kind of logic to a file. Our counter treats a saved file as code and an inline heredoc as scratch, so one arm got charged for it and the other did not.

To be clear about what those files are: all seven are ordinary work scripts — edit.py, diff.py, build_model.py — not tests. So this is not ponytail’s “leave one runnable check behind” rule showing up; it simply saved its solution to disk where the plain agent piped the equivalent through an interpreter. We cannot say ponytail wrote more code on those tasks, only that more of its code was persisted.

Does that bias the headline? Slightly, and in both directions. Ponytail alone persisted code on 7 tasks (761 lines); the plain agent alone did on 4 (567 lines). Net, about 190 lines out of 10,205 land against ponytail — under 2%, and too small to lean on either way. We are not claiming the −15.4% is conservative because of it.

Finding 4 — the bill drops, and this time the signal is solid

A typical task cost 10.3% less with ponytail installed: p=0.004 across 80 pairs, cheaper on 46 tasks and dearer on 34. That is the strongest positive cost result in this series so far, and the first that is a solid saving rather than a solid penalty — rtk’s +7.6% was every bit as significant, just pointing the wrong way. Caveman also came out around 10% cheaper once we removed a single pricing-tier outlier, but that was a fragile number resting on one exclusion; this is the first time the cost difference has survived a paired test on a full run.

One honest qualifier, because we would want it applied to a vendor: the median saving is −10.3%, but the spread around it is wide enough that a bootstrap interval on the median just touches zero. The direction is well supported and the per-task test is clear. “Roughly 10% cheaper on this workload” is defensible; “ponytail saves you 10%” is not.

Worth noting what did not move cleanly: the input side. Re-reading its own history fell 8.4% and fresh tokens 3.9%, neither of them significant (p=0.138 and p=0.085). In part 2 we found that an agent’s bill is dominated by that re-reading, which is why a tool compressing command output barely dented it. Ponytail attacks the other side of the ledger, what the model writes, and on this benchmark that is the side the money moved on.

Finding 5 — no quality difference we can detect

The obvious worry about a skill whose whole personality is “write less” is that it gets there by deleting things that mattered. Ponytail claims it never touches validation, error handling, security or accessibility, and reports 100% safety in a separate adversarial tier of its own benchmark.

We cannot speak to that safety claim, and want to be explicit about why: SkillsBench verifiers score whether a task was completed. They are not a security, validation or accessibility suite. Nothing below tests whether ponytail preserves a guard, only whether the work still passes.

Nine tasks scored slightly worse, six slightly better, 65 identical — statistically indistinguishable. That is a null result, not a clean bill of health: this run was never powered to prove equivalence, and the data remain compatible with a small degradation as well as a small improvement. What we can say is that nothing here looks like the obvious failure mode, where writing less quietly stops the tests passing.

One small note on adherence. Ponytail’s ruleset asks the model t

[truncated for AI cost control]