Runta's 27,280-Run Harness Eval Finds the Model Explains 92.8% of Agent Cost
A.I. / news
Runta's 27,280-Run Harness Eval Finds the Model Explains 92.8% of Agent Cost
FrontierHarness v1.1 ran 22 coding-agent harnesses against five open models, and Codex CLI went from first place to sixteenth when the model changed.
Runta published FrontierHarness Eval v1.1 on Nov. 17, a benchmark of 18 coding-agent harnesses that found the model, not the harness, drives almost all of what an agent costs. The model explained 92.8% of cost variance, the harness alone 2.3% and the model-harness pair 4.9%, according to Runta's post, written by Shilin Zhu.
The run covered 27,280 agent runs and 83B tokens. Runta said that is 76 times the size of v1.0.
How the eval was built
The design is 22 harness configurations across 18 harnesses, five models, 62 terminal tasks and four attempts each. Task time limits run from 15 minutes to 8 hours.
All five models are open-weight and were served by Baseten: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash 1017, Muse (muse-spark-open) and Kimi-K3. Runta ran the trials in Harbor, an Apache-2.0 evaluation framework from the creators of Terminal-Bench, using each harness's default settings at maximum reasoning effort. Each trial got a fresh Runta runtime that was deleted afterwards.
The named harnesses include Codex CLI, Pi, ZCode, Oh My Pi, Alma, OpenCode v1 and v2, Cline CLI, Antigravity CLI and mini-SWE-agent.
Cost and pass rate by harness
- Model92.8 %
- Model-harness pair4.9 %
- Harness alone2.3 %
Source: Runta, Introducing FrontierHarness Eval v1.1, accessed 2026-11-17
| Measure | Harness | Result |
|---|---|---|
| Lowest cost per task | OpenCode v2 | $0.561 |
| Highest pass rate (tied) | Pi, Codex CLI, ZCode | 61.9% |
| Lowest cost per pass | Codex CLI | $0.937 |
| Highest cost per pass | Antigravity CLI | $1.63 |
Cost per pass is cost per task divided by pass rate. Runta reported both measures because a cheap harness that fails often is not cheap.
Harness rank moved with the model
The median harness shifted 11 of 22 places between models. Codex CLI was first with GLM-5.3 and sixteenth with Kimi-K3. Runta's best-value pairings differ by model: Codex CLI with GLM-5.3 at $0.803 per pass, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3.
With the model held fixed, harness cost per task still varied from 1.8 to 5.0 times. The widest gap was on DeepSeek, where Cline CLI cost 5.0 times as much as OpenCode v1. Steps per pass correlated with cost per pass at +0.51 in rank terms. Prompt size, tool count and cache hit rate did not.
Reward hacking and wasted spend
Runta scored a trial at 0 when an audit flagged reward hacking. The audits flagged 216 trials, 163 of them for online lookups.
The post also reports that 64.5% of list-price cost went to trials that did not pass. That figure is the one a team budgeting for agents would want broken out by task type, and the post does not do that.
What the eval does not cover
Only open models were tested, and all went through one provider, so Runta said cost rankings could differ elsewhere. Runta planned to add closed models, naming Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon, and a study of long-horizon, continuously running harnesses.
The results are Runta's own, run on its own infrastructure. No other party has reproduced them. Other work on measuring coding agents, such as Nvidia's SkillEvaluator, faces the same question of whose harness produced the number. Developers choosing a harness also face a privacy question, covered in the Windows telemetry report.
Runta said closed models are next, with no date given in the post.
Sources
More in A.I.
- 01Qwen3.8-27B Has 6.9 Million Downloads on Hugging Face. Artificial Analysis Rates It Slow and VerboseAlibaba's Apache 2.0 dense model beats its size class on an independent index of 20, and takes 38.8 minutes a task against a 12.6-minute median.
- 02Claude Code Team Publishes Its Verification Rules: 39 Checks, a Three-Round Limit and Observe ModeA Nov. 16 post says about 90% of the team's code is written and reviewed by Claude, and lays out the gates that keep it shippable.
- 03Inkling 2 Small Weights Land on Hugging Face Under Apache 2.0, and the Card Asks for 500GB of VRAMThinking Machines opened the weights a week after launching on Tinker. The safety section still says release is pending.
- 04Fed Governor Barr Says AI Agents Could Accelerate Bank Runs That Unfold in MinutesIn a Nov. 16 speech in Milan, Barr also put AI firms' off-balance-sheet liabilities above $3 trillion and proposed no new rules.