Claude Code Costs Twice as Much as Pi, Berkeley Study Finds

Anthropic's Claude Code costs about twice as much as a rival open-source coding-agent harness for barely any gain in task success, according to a study published this week by researchers at the University of California, Berkeley and the AI benchmarking group Arena.

The study, called HarnessTax, tested 21 model-harness pairs spanning seven models and three harnesses, Claude Code, OpenAI's Codex CLI and the minimal open-source harness Pi, on the SWE-bench Lite and Terminal-Bench 2.0 coding benchmarks. Each pair ran three times across 30 sampled tasks per benchmark, with token costs priced against a fixed list dated Sept. 1.

A harness is the scaffolding wrapped around a model, the tool definitions, system prompts and retry logic that let it read files, run commands and edit code, as distinct from the model's own weights. Anthropic's own engineering team described the concept in a Nov. 26 post explaining why agents need a structured harness to make progress across sessions with no memory of what came before.

Same success rate, different price

The study's authors, Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia, found that swapping harnesses barely moved how often a model solved a task. Claude Fable 5 solved 97.8% of attempts in Claude Code, 96.7% in Codex and 96.7% in Pi. Harness choice moved the average success rate by only about 2 percentage points on SWE-bench Lite and about 5 points on Terminal-Bench 2.0 across the models tested.

Cost moved far more. Across shared models, the study found Claude Code cost about 2.0 times as much as Pi and 1.6 times as much as Codex on SWE-bench Lite, and about 1.5 times as much as Pi on Terminal-Bench 2.0, using geometric means of the cost ratios. For Claude Fable 5 specifically, Claude Code averaged $1.33 per SWE-bench Lite attempt against $0.67 for Pi, a nearly identical 15.4 versus 15.3 turns per attempt.

Model	Harness	Success rate	Cost per attempt
Claude Fable 5	Claude Code	97.8%	$1.33
Claude Fable 5	Pi	96.7%	$0.67
Claude Sonnet 4.6	Codex CLI	68.9%	not reported
Claude Sonnet 4.6	Claude Code	66.7%	not reported
GPT-5.6 Sol	Pi	83.3%	$0.42
GPT-5.6 Sol	Codex CLI	78.9%	$0.76

1. vs. Pi (SWE-bench Lite)2 ×
2. vs. Codex CLI (SWE-bench Lite)1.6 ×
3. vs. Pi (Terminal-Bench 2.0)1.5 ×

Source: HarnessTax, UC Berkeley and Arena, accessed Sept. 17, 2026

Why a four-tool harness keeps up

Pi ships with just four tools, read, write, edit and bash, and reached the Pareto frontier, the best success rate available at a given cost, on both benchmarks. The researchers traced part of the gap to context: across all seven models, Claude Code's mean initial context before the first model call ran more than 10 times longer than Pi's, driven by longer instructions and larger tool schemas.

The methodology behind those numbers was deliberately conservative. Each attempt was capped at 100 agent turns to control the cost of long runs, and the researchers estimated 95% confidence intervals from 10,000 bootstrap resamples of the 30-task averages. For SWE-bench Lite, they blocked external network access from every task container and disabled the default web tools in both Claude Code and Codex, so no harness could win by browsing for an answer the others could not reach. For Kimi K3, the one open-weight model in the set, they accessed it through Fireworks AI and used its single native thinking mode across all three harnesses, so differences could not be blamed on a weaker configuration on one platform than another.

The pattern held even for OpenAI's own model-harness pairing. On Terminal-Bench 2.0, GPT-5.6 Sol solved 83.3% of attempts in Pi against 78.9% in Codex, OpenAI's own harness, at about half the cost, $0.42 versus $0.76. Across six Anthropic and OpenAI models on both benchmarks, an alternative harness beat the model's own provider's harness on success rate in nine of 12 comparisons.

That runs against the marketing pitch each provider makes for its own tooling. The study's authors note that OpenAI has described GPT-5-Codex as optimized for agentic software engineering specifically inside Codex, its own coding harness. Providers do report optimizing some models for their own coding environments, the authors wrote, but a shared provider does not guarantee the best pairing on the tasks they tested.

What the study does not claim

The researchers flagged their own limitation directly: both benchmarks are open-source and the models tested may have encountered them during training, so the gap could differ on private or newer workloads. Matei Zaharia, Databricks' chief technology officer and a UC Berkeley computer science professor, is a co-author.

Anthropic's own harness writeup frames Claude Code's design around maintaining continuity across long-running, multi-session work, not around minimizing token cost, so the study's cost gap does not necessarily contradict what Anthropic says the extra scaffolding is for. The researchers said the next step is automating harness selection so developers do not have to make the tradeoff by hand.

The same trending-tools pattern the study points to, cheap open harnesses catching up to vendor defaults, shows up elsewhere on GitHub this week: alphaXiv's OpenResearch and Cloudflare's security-audit skill both gained more than 1,000 stars in a day built on the same premise, that the harness around a model, not just the model, decides what an agent can actually do.