Parsimony Ranks Coding Agents by How Much Code Their SWE-bench Fixes Add, and Its Own Page Warns Smaller Is Not Better

An open-source benchmark called Parsimony now ranks A.I. coding agents by how much code their bug fixes add, using patches the agents already published for SWE-bench Verified. It appeared on Hacker News on Oct. 14 as a Show HN post, and its own page says that writing less code does not mean writing better code.

The Parsimony site names no author. The GitHub repository, under the user La5u, is MIT-licensed and carries 147 commits on main. Its README describes it as a "benchmark that rewards AI coding agents for smaller, simpler fixes."

How it counts

Existing coding benchmarks track solve rate and cost, the site says, but not code size. Parsimony counts coding units: statements, calls, comparisons, names and literals. It parses both versions of every changed file and reports net units, meaning units added minus units removed.

Lines are not counted, so squeezing code onto fewer lines does not improve a score. Tests, docs and generated files are excluded. Nothing is executed, and pass or fail comes from SWE-bench's published results rather than a rerun.

SWE-bench Verified is a 500-instance subset of SWE-bench that human annotators reviewed, made in collaboration with OpenAI. Parsimony ranks each model by mean net units added on tasks it solved, lowest first. Only tasks solved by at least three other models count.

What the author concedes

The site lists its own limits. In its words: "Smaller isn't always better: units measure size, not readability or design." Only Python is measured, and attempts with missing files or parse errors are dropped.

The agent matters as well as the model. All boards use mini-SWE-agent, but the README says agent setup still differs across boards.

Check the site reports	Value
Spearman rank correlation, code volume vs solve rate	0.33 to 0.60
Boards with that correlation	26 or more models
Min. other solvers for a task to count	3
Language measured	Python only

That correlation is the awkward finding. Models that write more code also tend to solve more tasks, so a low Parsimony score can reflect a model that solves fewer problems rather than one that writes tidier fixes. The page says so directly.

Unverified runs and one example

The site also shows runs it labels unverified and unranked: OpenAI models with the pi agent, the GPT-6 family with Codex, and Anthropic models with Claude Code, covering the first five to 10 DeepSWE Python tasks. Those samples are too small to rank anything.

One example is on the page: "GPT-5.6 Sol ran both ways: my run added 0.88× the code of its published run." The author says a rerun of the same model produced 12 percent less code than its published run, and the site does not say why.

The rendered page did not show the leaderboard rows to The Terminal, so no per-model score is reported here. The same question of cost per finished task is behind the site's earlier look at Claude Haiku 5.5, which writes 440 million tokens to finish one benchmark, and tools that claim to trim agent output are covered in the context-mode piece.

The Artificial Analysis Intelligence Index appears on the site as an outside reference and does not affect rank.