Headroom cut 39% of my tokens and raised my Claude bill

there has been a lot of hype around token saving tools for llms like rtk , headroom , and ponytail . every time i read about one i wonder the same thing: do these actually work?

i haven’t seen much a/b testing out of any of them, and the optimizations don’t make much sense to me in an efficient market. i don’t have a good name for this, so i’ll call it the platform envelopment paradox: why wouldn’t anthropic or openai adopt these tricks internally and forward a chunk of the savings to you? headroom, rtk and ponytail all save tokens, and i’d imagine most engineers at those labs spend their days looking for simple ways to save compute. any major token optimization that can be lifted onto the platform probably will be. so i wanted to actually evaluate one in a controlled way. funny enough, jetbrains had already tested rtk and found it increased costs, so i ran the same kind of test on headroom.

most of headroom’s benchmarks compare headroom to itself. run it locally and you get numbers telling you how much money you’re saving, with no way to say how the same non-deterministic entropy hole would have behaved without it.

so i used harbor with the sr swe bench , three arms over the same tasks: plain claude as the baseline, headroom in token mode, and headroom in cache mode 1 . cache mode changes so little that it mostly measures how far two runs of the same setup drift apart on their own, and that number ends up doing a lot of work later. running all of it meant spinning up a pile of vms on an anthropic api key (no subscription pricing), so i sampled 25 test cases picked to favor headroom.

token mode 1 cut total input tokens from 215m to 132m across the arm. that part works exactly as advertised. yay

0 100M 200M baseline cache mode token mode 215M 191M 132M 0 100M 200M baseline cache mode token mode 215M 191M 132M share that missed the cache

0 4% 8% baseline cache mode token mode 1.7% 2.1% 7.4% 0 4% 8% baseline cache mode token mode 1.7% 2.1% 7.4% compression works on the left panel and backfires on the right. token mode sends 39% fewer prompt tokens than baseline, and 4.4x as large a share of them arrives uncached, where they bill at ten times the cache-read rate. great, right? not exactly. token mode came out about $0.37 per task cheaper on average, and more tasks came out pricier with compression on than came out cheaper.

dollars saved per task, compared to baseline

dot is the mean, bar is the median, line is the 95% bootstrap ci

-$1 $0 +$1 +$2 cache mode token mode +$0.730 +$0.367 median +$0.833 median -$0.270 pricier with headroom cheaper with headroom -$1 $0 +$1 +$2 cache mode token mode +$0.730 +$0.367 median +$0.833 median -$0.270 pricier with headroom cheaper with headroom cache mode n 25 · wilcoxon p 0.156 · cohen's dz 0.291 · 17 cheaper / 8 pricier token mode n 25 · wilcoxon p 0.656 · cohen's dz 0.13 · 9 cheaper / 13 pricier running the identical configuration twice, which is what cache mode amounts to, already produces an apparent saving of $0.730 per task. token mode's $0.367 lands inside that band, its median is negative , and more of its tasks came out pricier than cheaper. there is no effect here to measure. $0.37 sounds like a win until you look at cache mode, which “saved” $0.73 per task while changing almost nothing. the apparent saving is half the size of the noise.

so: 39% fewer tokens, no detectable change in cost. that combination should feel wrong. tokens are what you pay for.

all tokens are not billed the same. with provider prompt caching they aren’t even fungible. at published sonnet rates , fresh input runs $3.00 per million and a cache read runs $0.30, a tenth of the price. an agent loop that keeps appending to a stable prefix is close to the ideal case for this: on the baseline arm, roughly 98% of prompt tokens were cache reads.

that 98% puts a ceiling on the whole exercise. the 83m tokens headroom removed were almost all cache reads, and cache reads run $0.30 per million, so a perfect compressor that never busts a single cache entry saves about $25 here, tops. it only gets worse from there.

compression works by rewriting history, and rewriting history changes the prefix the cache is keyed on. cache mode 1 busted the provider cache 14 times across 25 tasks. token mode busted it 123 times, and cache-read reuse per write token fell from 38:1 to 11:1 . compression removed tokens and, in the process, pushed six million other tokens out of the cache and into full-price billing.

the discount it gave up came out within a dollar or two of the savings it produced. you end up trading tokens at $0.30 for the privilege of sending fewer tokens at $3.00.

what compression saved, against what it gave up

dollars across the 25-task token mode run

-$15 $0 +$15 compression saving, as metered cache discount forfeited net +$15.41 -$16.37 -$0.96 -$15 $0 +$15 compression saving, as metered cache discount forfeited net +$15.41 -$16.37 -$0.96 token mode busted the provider cache 123 times against cache mode's 14 , and cache-read reuse per write token fell from 38:1 to 11:1 . that knocked 6,062,098 tokens out of the cache and into full-price billing, worth $16.37 of forfeited discount, against $15.41 of savings headroom credited itself. on a workload that already reuses the cache, shrinking the prompt costs money.

headroom reports its own savings, and it reported real money saved on this run. it overstated by roughly 1.9x, and the mechanism isn’t specific to headroom: any meter that prices removed tokens at the list rate and never subtracts the cache reuse it broke will overstate on a cached workload.

the meter prices every removed token at the model’s list input rate , and nothing in it knows what a cache read is. on this run that worked out to crediting about $2.00 per million tokens removed, while the cache-weighted price actually paid was about $0.79 per million. crediting $2.00 for a token that cost $0.79 accounts for most of the gap.

the rest of the gap is what the metric never does at all: it doesn’t subtract the cache reuse the compression invalidated. it counts the tokens it removed, not the tokens it knocked out of the cache somewhere else in the loop.

savings on the run, as claimed and as measured

-$10 $0 +$10 claimed measured clean set +$15.41 +$8.13 -$8.98 -$10 $0 +$10 claimed measured clean set +$15.41 +$8.13 -$8.98 the gap between the first two bars is the 1.9x overstatement. the third bar is what is left after dropping the trials that crashed early and booked a saving for work they never did. per task, the meter happily reported a saving on trials that came out more expensive than baseline. drop the trials that crashed early and booked a saving for work they never did, and the remaining tasks came out $8.98 more expensive than baseline while the meter still showed $15.41 saved. a savings number that can’t come out negative isn’t measuring savings.

solve rates were flat: 5 of 25 for token mode, 4 for baseline, 4 for cache mode. for every task where the arms disagreed i had subagents read the full session transcripts side by side, rather than trusting the aggregate.

the disagreements were model nondeterminism, not compression. where compression “won,” it won by spawning a research subagent, or by diagnosing the real root cause while the baseline patched the wrong layer. none of the baseline runs showed the symptoms compression is supposed to cure: nothing forgotten, no file read twice. the one hard regression turned out to be an infrastructure cutoff mid-run, with the transcript ending on a credit-balance error and zero edits made.

so compression matched the baseline’s reasoning quality on 39% fewer input tokens. that part is genuinely useful, it just has nothing to do with cost.

realistically it’s probably not saving money at all. the $0.37 saving i measured is inside the noise, and these tasks were picked to favor headroom. if i’ve learned anything: before you trust one of these tools, run some benchmarks yourself and see if they align with what it’s reporting.

where compression still earns its place is when you’re up against a context window rather than a budget. matching baseline reasoning on 39% fewer tokens is real, it’s just a memory result and not a cost one. so if you’re working on problems that lean less on caching i can see this working, but ultimately for multi-turn agents i’d skip it.

everything here is my personal opinion, based on my own testing, and that testing is small: 25 tasks on one benchmark. treat it as a data point, not a verdict.

the arms, same 25 tasks each: baseline is plain claude with no compression. token mode is headroom rewriting history and caching tokens mid transcript. cache mode is headroom rewriting only the newest part of the transcript, the closest thing to running baseline twice. ↩ ↩ 2 ↩ 3