Caveman Saves Tokens by Doing Less, Not Just Saying Less

Caveman promises an appealing optimization for coding agents:

Same answers, 65% fewer output tokens. Brain still big. Mouth small.

The idea is simple. Models often produce unnecessary explanations, narrate tool calls, repeat conclusions, and surround useful information with polite filler. Remove that verbosity and the agent should become faster and cheaper without becoming less capable.

Coding agents can consume a surprising number of tokens while inspecting repositories, running commands, reading test output, and correcting their own mistakes. A small skill that removes wasted language sounds like an easy efficiency win.

But a coding agent is not a chatbot producing one final answer. It operates through a trajectory: inspect, reason, call a tool, interpret its result, update the plan, and decide whether to continue. An instruction that changes how the agent communicates can also change how it acts.

To measure that effect, I ran 60 tasks across Claude Code and Codex CLI using SWE Bench Verified, Terminal Bench 2, and USACO tasks.

Caveman reduced Claude Code token usage by 18.4% and its reported cost by 24.5%. In Codex, however, token usage fell by only 3.9%.

The more important result was not how many tokens disappeared. It was where they disappeared from.

With Caveman enabled, Claude Code performed 17% fewer steps and made almost 20% fewer tool calls. In Codex, hidden reasoning tokens fell by 13.5%. Caveman did not merely make the agents speak less. It changed how much work they performed.

What Caveman Actually Promises

Caveman is a skill for Claude Code, Codex, Gemini, and other coding agents. It instructs the model to avoid narration, remove hedging, shorten explanations, and communicate using compact fragments.

Its README summarizes the intended behavior with a memorable distinction:

Brain still big. Mouth small.

The main benchmark compares ten standalone answers, including prompts such as explaining Git rebase, React rerenders, and Docker multistage builds. The reported average falls from 1,214 output tokens to 294, producing the headline reduction of approximately 65%.

That result is plausible for those prompts. They ask for prose, and Caveman aggressively compresses prose. The problem is the extrapolation.

A single conversational answer and a repository level coding task have very different token distributions. In a coding agent, useful output includes source code, patches, shell commands, tool arguments, logs, and exact error messages. And caveman leaves much of this content unchanged.

The project itself acknowledges this distinction. Its documentation warns that Caveman only targets output tokens, adds input tokens of its own, and may become net negative when the original model is already concise. It also describes the honest evaluation control as Caveman versus a simple terse instruction, rather than Caveman versus an unconstrained baseline.

Those caveats are good. Caveman is free, open source, and does not collect telemetry. The issue is not that the project provides no evidence or hides every limitation.

The issue is that the headline claims are much broader than the evidence supporting them.

The Experiment

I evaluated five agent configurations:

The benchmark contained 60 tasks:

Each task was executed three times under every configuration:

60 tasks × 5 configurations × 3 trials = 900 trials

The runs used Harbor as the evaluation harness and subscription authentication rather than API keys. All 900 trials completed without infrastructure failures.

The claude-medium configuration is particularly important. Comparing only baseline against Caveman mixes together two possible effects:

1. Caveman may compress communication.
2. Caveman may cause the model to spend less effort.

The medium effort arm provides a native way to reduce model effort without installing Caveman. It lets us ask whether the skill offers something beyond indirectly making the agent do less work.

The Top Level Results

Claude Code showed substantial savings:

Codex showed a much smaller reduction:

If we stop here, Caveman looks like a reasonable tradeoff for Claude and a poor one for Codex.

Claude saves approximately one quarter of its modeled cost while losing little Pass@1. Codex gives up considerably more performance for a 3% cost reduction.

But these aggregated numbers hide the most important part of the experiment.

The Wrong Denominator

Caveman promotes a reduction in output tokens. Coding agent cost, however, is not dominated by visible prose.

For Claude Code, the baseline token distribution looked like this:

The prose estimate comes from converting visible agent text into an approximate token count. Directly generated prose accounts for only about 0.046% of Claude’s reported session token volume. A 41% reduction therefore removes only about 0.02% of the total tokens directly.

Shorter replies also make subsequent contexts slightly smaller, so the full mechanical saving from prose compression is larger than 0.02%. But even a deliberately generous fixed trajectory estimate puts that effect well below 1% of total token volume. Most output tokens are not conversational filler. They are code, diffs, commands, tool arguments, or other content that must remain exact.

That is nowhere near the observed 18.4% reduction.

The missing savings have to come from somewhere else. The trajectory data provides the answer: with Caveman, Claude took 17% fewer steps and made almost 20% fewer tool calls. Instead of merely making each turn shorter, Caveman removed entire turns, and each removed turn avoided processing a large context again.

The expensive part of an agent is not necessarily what it says between tool calls. It is taking another turn.

Where the Tokens Actually Went

The total token count can be decomposed approximately into two factors:

total tokens = number of steps × tokens per step

Applying that decomposition gives the following:

For Claude, approximately 92% of the token reduction is explained by taking fewer steps. Tokens per step barely changed.

Caveman certainly changed the writing style. Visible prose fell by 41%, and prose per step also decreased. But this compressed prose was too small to explain the session level savings.

The savings appeared because the model ended its trajectory earlier.

That interpretation is also visible in the tool usage:

A style instruction propagated into the action policy of the agent.

This does not automatically mean that the removed steps were useful. Agents can overinspect repositories, repeat tests unnecessarily, and continue reasoning after they already have a valid solution. Some trajectories should be shorter.

But it does mean that Caveman is not the intervention described by “brain still big, mouth small.” It changes the brain as well as the mouth.

The Control That Changes the Conclusion

The strongest argument against using Caveman is not the small drop in quality. It is the claude-medium control.

Both configurations reached exactly the same Pass@3:

Claude Caveman: 86.7%
Claude medium:  86.7%

But Caveman used 29.8% more tokens and cost 23.0% more than medium.

The chart makes the relevant comparison visible. On this plane, a configuration is dominated when another configuration achieves at least the same quality at a lower cost. Claude medium sits directly to the left of Claude Caveman: same Pass@3, lower cost.

The difference remains after accounting for successful trials:

The quality comparison between Caveman and medium is inconclusive at Pass@1: Caveman was 2.8 points higher, but the difference was not statistically significant. The efficiency comparison is much clearer. Both configurations reached 86.7% Pass@3, yet Caveman consumed 25.3% more tokens per resolved trial and cost 16.6% more.

This comparison changes the practical recommendation. Caveman can reduce resource use, but it does so by indirectly altering the agent’s trajectory. When Claude already provides a direct control for effort, using a global style instruction to reach a similar operating point is difficult to justify.

Why Caveman Barely Saves Anything in Codex

Caveman reduced Claude token usage by 18.4%, but reduced Codex usage by only 3.9%.

One possible explanation is that Codex had much less waste available to remove.

Before applying Caveman, the Codex and Terra combination already used substantially shorter and cheaper trajectories:

Codex reported around 74% fewer tokens while obtaining slightly better resolution rates.

This difference appeared at two levels. Codex took around 30% fewer steps, and each step processed around 62% fewer tokens.

That does not prove that GPT 5.6 Terra is intrinsically more efficient than Sonnet 5. The comparison mixes the model, agent scaffold, tokenizer, context management, caching behavior, tool representation, and effort configuration.

The defensible conclusion is narrower:

In this harness, Codex with Terra started from a much more token efficient trajectory than Claude Code with Sonnet 5.

This provides a plausible explanation for the different Caveman results.

Claude offered a long trajectory that the skill could truncate. Codex already operated closer to a floor.

Caveman needs waste to remove

An exploratory analysis inside the Codex results supports this interpretation.

I divided the tasks into four groups according to their baseline token consumption. Caveman increased median consumption among the cheaper tasks and only produced savings among the more expensive ones:

This analysis should not be treated as causal. The groups mix datasets, contain relatively few tasks, and are defined using observed baseline consumption. Regression to the mean may explain part of the pattern.

Still, the result is consistent with a floor effect.

When the original trajectory is already compact, Caveman has little redundant work to remove. Its additional prompt and behavioral changes may cost more than they save. When a trajectory becomes unusually expensive, Caveman has more room to terminate it early.

The 3.9% aggregate reduction in Codex was therefore not a small uniform saving across all tasks. It was driven by the expensive tail.

For the median Codex task, Caveman did not save tokens.

This suggests a better mental model:

Caveman is not a general token compressor. It behaves more like an indirect effort limiter whose apparent effectiveness depends on how inefficient the baseline agent already is.

What Happened to Quality

Caveman did not improve aggregate Pass@1 or Pass@3 in any of the main comparisons.

For Claude:

Pass@1: 81.1% → 80.0%
Pass@3: 91.7% → 86.7%

For Codex:

Pass@1: 89.4% → 85.0%
Pass@3: 93.3% → 90.0%

The direction is consistent, but the experiment is not large enough to establish the magnitude precisely.

Bootstrap confidence intervals computed at the task level all crossed zero:

The correct conclusion is not that Caveman has been proven to degrade coding agents. The correct conclusion is:

We found no evidence that Caveman improves task resolution. Every aggregate quality comparison moved in the negative direction, but the experiment was not large enough to estimate effects of a few percentage points precisely.

There is one statistically detectable behavioral effect. Codex reasoning tokens fell by 13.5%, with a 95% confidence interval from 4.6% to 19.8% lower. This is difficult to reconcile with the claim that Caveman leaves reasoning untouched.

Repeating Trials Did Not Add as Much Evidence as Expected

Each task was executed three times. Most tasks were either solved in all three attempts or failed in all three attempts. Only around ten or eleven of the sixty tasks produced mixed results within a configuration.

This matters because three trials do not provide three fully independent observations. Much of the variance belongs to the task itself rather than random sampling within the task.

The practical lesson for future evaluations is simple:

When agent failures are task correlated, adding more tasks is more valuable than repeatedly running the same small set.

This also explains why small evaluations can produce dramatic and unstable numbers.

Terminal Bench contained only ten tasks. A difference of ten Pass@3 points represents a single task. One contrast in that subset produced a token ratio above 100% because of one unusual task.

JetBrains reported a similar experience. Its first ten task Caveman evaluation suggested savings near 30%. After expanding the benchmark, the result fell to 8.5%.

Independent Evaluations Point in the Same Direction

A JetBrains evaluation tested Caveman on Claude Code using Harbor and SkillsBench.

It measured an 8.5% reduction in output tokens, far below the advertised 65%, with no detectable quality difference. Its explanation was structural: coding agent output is dominated by code, diffs, tool invocations, and exact strings that Caveman cannot safely compress.

That result is compatible with this experiment, but the medium effort arm adds an important control. Even when Caveman reduces a trajectory without producing a detectable quality loss, it may still be an inefficient way to reach that operating point.

Other evaluations tested an even simpler question: does Caveman outperform asking the model to “be brief”?

Max Taylor evaluated 24 prompts with prompt specific quality rubrics and found Caveman in roughly the same token and quality range as a two word concision instruction.

Alex Rios described the problem as the Terse Control Fallacy: comparing a sophisticated compression technique against silence instead of comparing it against the trivial alternative.

Examining the Brevity Evidence

The Caveman README references the preprint Brevity Constraints Reverse Performance Hierarchies in Language Models as evidence that shorter answers can sometimes improve accuracy.

The paper evaluated 31 models on 1,485 questions and identified 115 cases where smaller models systematically outperformed larger ones. The authors associate part of this behavior with overthinking: larger models sometimes elaborate beyond what is useful and introduce additional opportunities for error. On this selected subset, constraining responses improved large model accuracy from 40.2% to 66.5%.

This is a meaningful result, but also a targeted one. The brevity intervention was evaluated on problems already exhibiting unusual scaling behavior, rather than across the benchmark as a whole.

There is also an important difference in settings. The study evaluates standalone questions without tools, repositories, iterative debugging, or decisions about whether further exploration is necessary. A coding agent must repeatedly decide not only what to say, but what to do next.

The paper does not validate Caveman as an agent optimization. At most, it shows that excessive elaboration can hurt on a selected group of standalone reasoning questions.

Dataset Results Were Not Consistent

The aggregate results also hide substantial variation across task types:

Each row compares that agent’s Caveman run against its own baseline, broken out by dataset.

SWE Bench with Claude was Caveman’s strongest case. It preserved Pass@1 while reducing token usage by almost 29%.

Terminal Bench with Claude was the opposite. Token usage increased while both success metrics fell.

USACO with Codex showed no quality degradation, but the baseline already reached 100% Pass@3. That subset had little room to distinguish the configurations.

The result depends on the agent, model, dataset, original trajectory, and probably the individual task.

Limitations

This experiment has several limitations.

The reported cost is modeled rather than billed because the runs used subscription authentication.

Timed out runs did not report token or cost metrics. The estimated savings should therefore be treated as approximate rather than exact.

Claude and Codex receive Caveman through different integration mechanisms. That reflects how the skill is distributed for those tools, but it means cross agent comparisons mix the skill with its delivery path.

The medium arm controls for cost and effort, but not for style. A stronger future design would include a plain instruction such as “communicate briefly, but do not change your exploration or verification behavior.”

The experiment also needs more tasks to estimate small quality effects. With sixty tasks and few discordant pairs, differences around five percentage points remain uncertain.

Should You Use Caveman?

I would use Caveman primarily as a style preference. If you find terse status updates and compact final answers easier to read, that alone may be a good reason to enable it.

What the experiment does not show is that Caveman is a reliable systems optimization. Its effect extends beyond wording and can alter the agent’s trajectory, including how much reasoning and tool use it performs. If you are concerned about possible quality loss, increasing reasoning effort may help, but that becomes another parameter to tune rather than a property guaranteed by the skill itself.

If the goal is reducing cost, I would prefer explicit controls such as reasoning effort, turn budgets, or context management. These mechanisms target resource usage directly and are easier to evaluate.

Caveman can make an agent more concise. The evidence is much weaker that it makes the agent more efficient.

The Broader Lesson

The most interesting result is not specific to Caveman.

A system prompt is part of an agent’s policy.

Instructions about tone, confidence, explanation, or verbosity may look cosmetic, but they can influence more than the text an agent produces. Because the agent repeatedly conditions its next action on those instructions, a request for concision can also affect how long it explores, how many tools it uses, and when it decides that the task is finished.

That is what happened here. Caveman did save tokens in Claude Code, but most of the reduction came from shorter trajectories rather than substantially fewer tokens per step. Codex, which already started from a shorter and cheaper trajectory, had much less room for that effect.

This is also why explicit controls matter. When the goal is to trade reasoning effort for cost, a native effort setting targets that tradeoff directly. In this experiment, medium reasoning reached the same Pass@3 as Caveman while using fewer tokens and costing less.

The broader lesson is simple: instructions that look like presentation choices can become behavioral interventions once they are placed inside an agent loop.

References

Caveman repository: https://github.com/JuliusBrussee/caveman

Caveman evaluation directory: https://github.com/JuliusBrussee/caveman/tree/main/evals

JetBrains evaluation: https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/

Max Taylor, I benchmarked caveman against two words: https://www.maxtaylor.me/articles/i-benchmarked-caveman-against-two-words

Alex Rios, There is a reason that caveman is a character from the past: https://alexriosme.substack.com/p/there-is-a-reason-that-caveman-is

Brevity Constraints Reverse Performance Hierarchies in Language Models: https://arxiv.org/abs/2604.00025

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks: https://arxiv.org/abs/2604.22750