Token-Compression Skills Do Not Survive Measurement

The skills are not lying about the tokens they remove. They are counting a number that has almost nothing to do with what gets billed, and nobody checks the difference because checking it is tedious and the dashboard looks so convincing.

How to Count What a Compression Skill Actually Costs#

Most compression skills report one number, which is the difference between the payload before and the payload after. That number is real and it is nearly useless on its own.

An honest count has to include the system prompt, every tool schema loaded on every turn, cache writes at 1.25x or 2x base input, cache reads at a tenth of base input, each compaction iteration, retries, subagent bootstraps, and the full rerun cost of any task the compressor caused to fail.

Most of that never appears in a before and after payload diff. Two of those components usually dominate the bill.

The one thing worth running before anything else is a sum across every billed iteration rather than the top-level usage fields, because those fields are not the total.

// top-level usage is NOT the bill when compaction is on
const billed = (res.usage?.iterations ?? []).reduce(
  (acc, it) => ({
    input: acc.input + it.input_tokens,
    cacheRead: acc.cacheRead + (it.cache_read_input_tokens ?? 0),
    cacheWrite: acc.cacheWrite + (it.cache_creation_input_tokens ?? 0),
    output: acc.output + it.output_tokens,
  }),
  { input: 0, cacheRead: 0, cacheWrite: 0, output: 0 }
);

The second rule is the denominator. Measure tokens per completed task, not tokens per request, because a compressor that saves 20 percent per call and causes one rerun in five has saved nothing at all.

A paired run is the whole protocol. Pick twenty tasks you actually do, run them with the skill enabled and disabled against the same repo state, and record four numbers per run alongside whether the task succeeded.

1. Cache read tokens and cache write tokens, kept separate, because they bill at wildly different rates
2. Uncached input tokens, which is the only column a compression skill can move directly
3. Output tokens, multiplied by five before comparing them to anything on the input side
4. Task outcome, because a failed run's rerun cost belongs to whichever arm caused it

Skip the outcome column and the measurement is worthless, since the cheapest possible agent is one that gives up early. That is not a hypothetical failure mode. It is what several compression arms in the published data were quietly doing.

Caveman prompting is the clean illustration. Stripping grammar and filler from prompts does cut per-request tokens, and when InfoWorld benchmarked it during JetBrains testing the cumulative cost across the benchmark came out higher for the caveman runs anyway. Every individual request looked cheaper. The job cost more.

Cache Reads Own the Bill, Not the Tokens You Removed#

The 2,848-run study reconstructed billing across four components and found that prompt-cache traffic accounted for about 87 percent of cost. Generated output was 10.4 percent. Everything a typical compression skill can reach lives in the remainder.

That is the whole mechanism. Compression works hard on the cheapest slice of the bill and, in doing so, disturbs the thing that made the rest of it cheap.

Cache reads bill at roughly a tenth of base input. Editing the prompt prefix is what a compressor does for a living, and an edited prefix is a cache miss that gets rewritten at 1.25x or 2x. Trading tokens priced at 0.1x for a rewrite priced at 2x is a bad trade even when the token count falls.

That table is a map of where the money actually is. A skill that only touches the 1x row is working on the least valuable real estate on the invoice, and it reaches that row by disturbing the 0.1x row and inflating the 5x row.

Only the simplest arm saved anything, and it saved 2.7 percent. The arm that removed the most tokens raised cost. The correlation between token reduction and cost change across all arms was r = 0.15, which is close enough to nothing that token count should not be used as a proxy for spend at all.

Compression Moves Tokens to the Expensive Side of the Invoice#

Output tokens bill at five times input on current Sonnet 4.6 and Opus 5 pricing. So the interesting question is not whether the prompt got smaller but what the smaller prompt did to the answer.

It makes the answer longer. Stripping a prompt removes the structural cues that tell a model how terse to be, and the model compensates by explaining itself. On MBPP at 0.3 retention, output ran 56.4 times longer than baseline, reaching 1,020 tokens.

HumanEval is the tell. It expands least because the function signature survives mild compression, so the model still knows what shape the answer takes. Where structure survives, output stays sane, and where structure is stripped, the model fills the vacuum at 5x the price.

One measured case put DeepSeek-Chat at 21 tokens of output uncompressed and 798 compressed, which the authors scored as a 2,140 percent increase in energy for the same task. Push retention to 0.7 and pass rate on code tasks fell from 26.0 percent to 1.5 percent.

The Compaction Call Bills You for the Context It Deletes#

Compaction has to read a conversation before it can shorten it, and that read is a full billed request. Anthropic documents the arithmetic in its own compaction docs, and the shape surprises people.

The documentation is blunt about the reporting gap. "The top-level input_tokens and output_tokens do not include compaction iteration usage", which means a run that compacts three times can bill several hundred thousand tokens the usage fields never mention.

Run the arithmetic forward and it gets worse. A session that compacts three times on a large context can bill more than half a million input tokens purely on the compaction reads, none of which the usage fields will show you.

Compaction still pays off in one specific case, which is when the compacted prefix is reused across many later turns. The saving lives in the reuse, not the summarizing.

So compact late and rarely, at a natural boundary where the work genuinely changes shape, rather than early and often to keep the context meter looking tidy. Compacting at 40 percent full is paying a large bill to solve a problem you do not have yet.

Accuracy Falls Where the Benchmarks Never Looked#

The number everyone repeats is LLMLingua's 20x compression at a cost of about 1.5 points of performance. That result is real. It was measured on GSM8K and BBH, which are reasoning benchmarks where the answer is short and the input is mostly scaffolding.

Agent work is the opposite of that. It is long, detail-dense, and the specific detail matters, which is exactly the shape compression handles worst.

The mismatch is structural rather than accidental. A compressor optimizes for a token budget, and nothing in that objective knows which tokens are the ones an action depends on.

On a reasoning benchmark, losing a bit of phrasing costs nothing because the model reconstructs the intent. On a patch, losing one file path, line number or exact error string costs the entire task. Same compression ratio, completely different blast radius, and only one of them was benchmarked.

An evaluation across 13 datasets found that every method tested increased hallucination to some degree, naming information loss as the primary cause. Fine-detail tasks lost 21 percentage points of accuracy at 33 percent retention.

The agent version of that number is the one worth remembering. Under compression, successful patch application fell from 27 out of 40 to 15 out of 40. Twelve tasks that used to work now need a rerun, and a rerun costs more than every token the compressor saved on them.

The speed argument does not survive either. The largest speedup observed under any tested condition was 18 percent, and on commercial APIs the compression step frequently made the round trip slower than sending the full prompt.

Fanning Work Out to Subagents Is Not a Context Diet#

Offloading work to subagents gets sold as compression because the parent context stays small. The parent context is not the bill.

Each subagent boots as a complete agent with its own system prompt and its own copy of every tool schema, reloaded on every turn it takes. Measured against doing the same work sequentially, two subagents consumed 2.6x to 5.9x the input tokens depending on the model.

Anthropic's cost documentation puts agent teams in plan mode at roughly 7x a standard session, for the same reason. Each teammate keeps its own context window and runs as a separate instance.

Fan-out buys wall-clock time and parallel attention, which are worth paying for on the right task. It does not buy tokens, and treating it as a context diet inverts what it actually does.

The distinction that matters is whether a tactic removes turns or trims their contents. Cutting MCP round trips removes turns, so the baseline overhead disappears with them. Compressing a payload leaves every turn in place and shaves the cheapest part of each one.

Where Compression Genuinely Survives#

None of this means context management is pointless. Four things hold up under honest counting, and pretending otherwise would be its own kind of dishonesty. What they share is that each one changes what gets sent, or how often, rather than compressing a payload that was going to be sent anyway.

• Stable cached prefixes. Not compression at all, technically. Freezing the prefix so cache reads bill at 0.1x is the single largest lever available, and it is the one most compression skills accidentally break.
• Moderate compression on genuinely long contexts. The same 13-dataset study that found universal hallucination increases also found performance gains from moderate compression on LongBench. Moderate, on long inputs, is a real sweet spot.
• Retrieval. Fetching the three relevant files instead of carrying forty works because it removes tokens that were never needed, rather than damaging tokens that were.
• Structured summarization of repetitive content, when the summary is reused many times downstream. The reuse is what pays, not the summarizing.

The tactics that survive are mostly about sending less in the first place rather than squeezing what gets sent. Several of them are covered in how to reduce AI coding tool token usage, and they hold up because they change the number of turns rather than the contents of one.

The 2,848-run result is one study on one agent product, the output-expansion work is recent and not yet widely replicated, and practitioner reports of higher bills after installing a token-saving skill are plentiful but rarely instrumented. The cost inversion is also specific to endpoints that bill prompt caching, because breaking a cached prefix is what makes the trade a loss.

Send the same prompts somewhere with no cache and token reduction becomes a plain saving again. What none of the evidence supports is the direction most guides assume for agents.

The Short Version#

Keep a compression skill only if a paired before and after run over the same task set moves total cost, counted across cache reads, cache writes, output, compaction iterations and reruns, while task success holds steady. That is a real afternoon of work and it is the only thing that settles the question.

Most skills will not clear it, and the failure is usually the same one. They optimize the number they can see, on the cheapest quarter of the bill, while the expensive three quarters move in the wrong direction underneath.

Anything that removes turns, stabilizes a prefix, or stops sending what was never needed will beat anything that squeezes what is already being sent. Start there before installing anything that promises a percentage.

Below that bar, the skill is trading a smaller number on a dashboard for a larger one on the invoice. The dashboard number is the one it was built to move.