Ponytail Skill for Claude Code: Does It Cut Tokens

Supercharge your tools with AI-powered features inside many JetBrains products

Part 3 of a series where we take public “token saver” add-ons for coding agents and run the same paired A/B benchmark against each of them. Part 1 was the caveman skill (advertised −65%, measured −8.5%). Part 2 was rtk (advertised −60–90%, measured +7.6%).

We ran 80 paired tasks to test the ponytail skill for Claude Code. Advertised: −54% code, -22% tokens, -20% cost, -27% time. Measured: −15% code, −10.3% cost and -11% time. Here’s what actually happened.

Real savings, although roughly a quarter to a half of what is advertised, it is the first tool in this series with a statistically solid cost- saving signal. We found no quality difference, though ~80 pairs can only rule out large ones. The catch: the code cut only shows up where there was room to over-build.

Ponytail skill is designed to make AI agents write less code. Its core premise: a senior developer who has seen everything replaces your fifty lines with one. Ask for a date picker and instead of installing flatpickr and writing a wrapper component, it writes <input type="date"> and moves on.

Mechanically it is a ladder the model climbs before writing anything. Does this need to exist at all? Is it already in the codebase? Does the standard library do it? A native platform feature? An installed dependency? Can it be one line? Only then: write the minimum that works. The ladder runs after understanding the problem, not instead of it, and validation, error handling, security and accessibility are explicitly off the chopping block.

Here is what that looks like in practice, from our own run. Both agents were asked to export a three.js scene to a Blender-ready OBJ file; both produced a file the verifier accepted. Both wrote the same fiddly loop to expand instanced meshes, because three.js’s OBJExporter cannot handle them. The difference is everything around that loop. To rotate the scene into Blender’s orientation and write it out, the plain agent builds a wrapper object to hold the rotation and names every intermediate step:

// no skill — 10 statements to rotate the scene and write the file const exportRoot = new THREE.Group(); exportRoot.name = 'blender_export_root'; exportRoot.rotation.x = -Math.PI / 2; exportRoot.add(root); exportRoot.updateMatrixWorld(true); const exporter = new OBJExporter(); const objString = exporter.parse(exportRoot); const outputPath = '/root/output/object.obj'; fs.mkdirSync(path.dirname(outputPath), { recursive: true }); fs.writeFileSync(outputPath, objString); // ponytail — the same job, 5 statements root.rotation.x = -Math.PI / 2; root.updateMatrixWorld(true); const obj = new OBJExporter().parse(root); fs.mkdirSync('/root/output', { recursive: true }); fs.writeFileSync('/root/output/object.obj', obj); Nothing was sacrificed there. Ponytail rotated the object it already had instead of building a parent to rotate it for it, and skipped an import while it was at it. Ten statements became five, both files exported the same geometry, and both scored 1.0. That is the effect working exactly as advertised — on one file, on one task.

The headline claim is −54% code , plus −22% tokens, −20% cost and −27% time. What made this one worth testing is that the claim is unusually well documented. The authors rebuilt their benchmark in response to a critique ( issue #126 ) that their original numbers came from a chatty baseline, and they publish the honest version: a real headless Claude Code session editing a real FastAPI + React repo, scored on the git diff it leaves behind. They even document a contamination bug they found in their own harness, where a SessionStart hook was firing on every arm and secretly running ponytail in the baseline.

That is more methodological candour than most tools in this space manage. So the question here is whether the effect survives a benchmark the authors did not choose, on a stronger model, with verifier-scored quality.

One detail matters more than it looks. We generated arm B’s injected text by calling ponytail’s own hooks/ponytail-instructions.js rather than writing a summary of it, so the ruleset the model saw is the skill’s own text rather than our paraphrase of it. That covers the ruleset the hook generates, not every side-effect the hook has on a real first run. Every with-ponytail trial is audited afterwards to confirm the ruleset actually reached the model; every baseline trial is audited to confirm it did not. That check is the direct descendant of the contamination bug ponytail found in its own benchmark, and it came back clean: 100% of treatment trials, 0% of baselines.

Before the paid runs we tested the obvious install path: drop the skill in and let Claude Code decide when to use it. Ponytail’s description invites exactly that, telling the model to use it on “ANY coding task: writing, adding, refactoring, fixing, reviewing, or designing code.”

Across all ten sessions it self-activated zero times . Not rarely. Never. The skill sat installed and visible and the model did not once reach for it.

This is not a bug in ponytail, and it is why the tool ships as a plugin with a SessionStart hook that injects the ruleset whether you ask or not. But it does mean the install method decides whether you get anything at all. Copy the SKILL.md into a skills folder and you will very likely measure nothing. Every number below comes from the arm where the ruleset is actually injected.

Across 80 paired tasks a typical task shed 15.4% of the code the agent wrote ; in total, 10,205 lines became 8,756. That is a substantial observed reduction. At p=0.088, however, it is the softest of our headline numbers. It is also nowhere near 54%.

Two things to say about the gap, both fair to the tool. First, their −54% is a mean across twelve hand-picked feature tickets; ours is a median across 80 tasks nobody chose for this purpose. Means and medians on skewed data are different animals, and their own writeup is explicit that the figure “reaches 94% where an agent over-builds and is near zero where the code is already minimal.”

Second, and this is the more interesting half: our own data points the same way.

Split the tasks by how much code the baseline wrote. Ponytail cannot pick its own bucket that way, since the plain agent decides it. Worth saying plainly though: we chose these thresholds after seeing the data, and grouping by the baseline’s own output can stretch a gradient like this on its own. Read the chart as a strong hint about where the effect lives, not as a measured law.

On big builds the cut reaches −31% . On tasks where the plain agent already wrote almost nothing, the typical task moved by zero — though the totals in that group actually rose, 104 lines to 910, and that gap is where the run’s one real surprise turned up.

On seven tasks our counter recorded zero lines for the plain agent and 51 to 230 for ponytail. Reading the transcripts, that gap is mostly about where the code lived rather than how much of it there was. The plain agent piped its solution straight into a Python interpreter as a heredoc, which produced the deliverable and left no script behind. Ponytail wrote the same kind of logic to a file. Our counter treats a saved file as code and an inline heredoc as scratch, so one arm got charged for it and the other did not.

To be clear about what those files are: all seven are ordinary work scripts — edit.py , diff.py , build_model.py — not tests. So this is not ponytail’s “leave one runnable check behind” rule showing up; it simply saved its solution to disk where the plain agent piped the equivalent through an interpreter. We cannot say ponytail wrote more code on those tasks, only that more of its code was persisted.

Does that bias the headline? Slightly, and in both directions. Ponytail alone persisted code on 7 tasks (761 lines); the plain agent alone did on 4 (567 lines). Net, about 190 lines out of 10,205 land against ponytail — under 2%, and too small to lean on either way. We are not claiming the −15.4% is conservative because of it.

A typical task cost 10.3% less with ponytail installed: p=0.004 across 80 pairs, cheaper on 46 tasks and dearer on 34. That is the strongest positive cost result in this series so far, and the first that is a solid saving rather than a solid penalty — rtk’s +7.6% was every bit as significant, just pointing the wrong way. Caveman also came out around 10% cheaper once we removed a single pricing-tier outlier, but that was a fragile number resting on one exclusion; this is the first time the cost difference has survived a paired test on a full run.

One honest qualifier, because we would want it applied to a vendor: the median saving is −10.3%, but the spread around it is wide enough that a bootstrap interval on the median just touches zero. The direction is well supported and the per-task test is clear. “Roughly 10% cheaper on this workload” is defensible; “ponytail saves you 10%” is not.

Worth noting what did not move cleanly: the input side. Re-reading its own history fell 8.4% and fresh tokens 3.9%, neither of them significant (p=0.138 and p=0.085). In part 2 we found that an agent’s bill is dominated by that re-reading, which is why a tool compressing command output barely dented it. Ponytail attacks the other side of the ledger, what the model writes, and on this benchmark that is the side the money moved on.

The obvious worry about a skill whose whole personality is “write less” is that it gets there by deleting things that mattered. Ponytail claims it never touches validation, error handling, security or accessibility, and reports 100% safety in a separate adversarial tier of its own benchmark.

We cannot speak to that safety claim, and want to be explicit about why: SkillsBench verifiers score whether a task was completed. They are not a security, validation or accessibility suite. Nothing below tests whether ponytail preserves a guard, only whether the work still passes.

Nine tasks scored slightly worse, six slightly better, 65 identical — statistically indistinguishable. That is a null result, not a clean bill of health: this run was never powered to prove equivalence, and the data remain compatible with a small degradation as well as a small improvement. What we can say is that nothing here looks like the obvious failure mode, where writing less quietly stops the tests passing.

One small note on adherence. Ponytail’s ruleset asks the model to mark deliberate shortcuts with a ponytail: comment naming the ceiling and the upgrade path. Across 80 trials with the ruleset demonstrably in context, that happened once . The ladder gets followed; the paperwork does not.

Worth showing because it is the trap this whole series exists to avoid. Our ten-task smoke run said ponytail cut code by 3% and made things 9.6% more expensive , with mean task scores collapsing from 0.51 to 0.31. Had we published that, we would have written a very different and completely wrong article.

Ponytail works. Across 80 paired tasks, it cut the typical bill by 10.3% and reduced code written by 15%, with no quality difference we could detect. It is the first tool in this series that clearly saved money. If you install it and forget about it, you should be modestly better off.

Do not expect the advertised 54% everywhere. Ponytail’s benchmark uses tasks with obvious over-building traps. Ours did not. In our runs, code fell 31% on larger builds and barely moved on tasks that were already lean. The more over-building your agent does, the more ponytail can cut.

Got a tool that claims to save tokens? Tell us which one and we will run it through the same benchmark.

Chart style borrowed from dither-kit , reimplemented here as a dependency-free inline widget.

Does ponytail skill work with Claude Code? Yes, but installation method matters. If you copy the SKILL.md into a skills folder and let the model decide when to use it, it will self-activate zero times