Google Research's Harness2 Has 10,700 Stars and Three Commits, and Publishes No Absolute Scores
Software / analysis
Google Research's Harness2 Has 10,700 Stars and Three Commits, and Publishes No Absolute Scores
The recursive agent-harness framework reports gains of up to 11.8 percentage points, but its README and arXiv page give no baselines, no cost per run and 16 open bugs.
Harness2, a Google Research repository for improving the prompts, tools and memory around a language model without touching its weights, has 10,700 GitHub stars, 956 forks and three commits on its main branch. Its README reports gains of up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB, and gives no absolute score for any of them.
The repository describes itself as "not an officially supported Google product," and says it is not eligible for Google's Open Source Software Vulnerability Rewards Program. The code is Apache 2.0.
What the framework does
A harness is the scaffolding an agent runs inside: its prompts, tools and memory. Harness2 works on that layer in two loops. Task-level recursion refines the harness for the task at hand through a Propose, Probe and Compose step. Harnessing-level recursion keeps the harnesses that were adopted, and the lessons from each task, in a persistent improvement harness for later tasks.
The --k flag sets the budget. --k 1 runs one step. --k K --mode parallel probes K general and K domain-specific candidates and produces K compositions, while --k K --mode sequential refines the harness through K successive steps. It supports OpenCode and Codex as executors.
The companion paper is arXiv:2610.14224, "Harness^2: Recursive Agent Harnessing for an Open World," submitted Oct. 9, 2026 under cs.AI by Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee and Tomas Pfister. Its summary says the approach was tested on three professional benchmarks and four model-harness configurations, and that a single step improved GLM-5 and Gemini 3.5 Flash over their base harnesses.
What the numbers leave out
The gains are the vendor's own, from the authors' paper and README, and no outside party is named as having reproduced them. More important, they are differences. The README says larger task-level recursion budgets bring further gains without quantifying them, and the arXiv page gives no baseline scores to subtract from.
- LAB11.8 points (LAB is percentage points of partial pass rate)
- WorkBuddy-Bench11.5 points (LAB is percentage points of partial pass rate)
- JobBench4.5 points (LAB is percentage points of partial pass rate)
Source: Harness2 README, google-research/harness2, accessed 2026-10-19
A gain of 11.5 points from a base of 20 is a different result from 11.5 points from a base of 70. The README also does not say what a run costs. It says only that "Model-backed runs incur API costs" and suggests starting with a small task selection.
One of the repository's open proposals is issue #4, a --max-usd spend limit for harness2 runs, opened Oct. 16. The README documents no spend cap, so a --k value set high on a model-backed benchmark is paid for in API charges with nothing in the tool to stop it.
The arXiv page lists nine authors but no affiliations in the text available for this report. The repository sits under Google Research's GitHub organisation, and the README's disclaimer is the only statement of how Google relates to the code.
The sandbox warnings in the README
The README lists limits that matter before running it. Executor network access is unrestricted, and Codex runs with its sandbox disabled. Public benchmark rubrics may be accessible online, so the README says to configure network restrictions before evaluation. An agent that can reach the internet while being scored on a public benchmark can look up answers.
Setup needs Linux, Python 3.12 and uv, plus Node.js 20 or later and the Google Cloud CLI for model-backed benchmarks, Pandoc and Poppler for LAB, and Docker for WorkBuddy-Bench. It also needs a Vertex AI project with model IDs set in a .env file. A local demo needs no credentials.
Sixteen open issues, most of them about bookkeeping
The issues page lists 16 open issues, with the earliest dated Oct. 16 and the latest Oct. 19, plus 64 open pull requests. Several concern whether results can be trusted.
| Issue | Opened | Problem as titled |
|---|---|---|
| #80 | Oct. 19 | judge_one trusts all_pass and n_passed from the payload, so a passing task reads as a failure |
| #77 | Oct. 19 | LAB runner silently fails when the timeout command is unavailable |
| #65 | Oct. 18 | Resume and rollout caches silently reuse outputs after task inputs change |
| #50 | Oct. 18 | harness2-report undercounts improver spend for tasks re-run after an interrupted invocation |
| #44 | Oct. 18 | harness2-judge keeps an old grade after a task is re-run if the re-judge fails |
Issue #39, also from Oct. 18, is a related-work citation request pointing to Project Peritus's harness evolution. Issue #71 is titled "Harness^2 analyzed by Claude and compared with similar systems."
The two grading bugs, #80 and #44, bear directly on the headline numbers, because both concern how a task's score is recorded. The page does not say whether the paper's figures were produced before or after either bug appeared.
What to check before trusting it
Harness2 is a research artifact with a star count that outpaces its history: 10,700 stars against three commits. Stars measure attention rather than use, and the site's report on CLI-Anything covers another agent repository with a large star count. For a different way agents get tested, the Vals refund study pitted six of them against a task they could not complete.
The next check is the paper's full text, which would hold the baseline scores that neither the README nor the arXiv summary gives.
Sources
More in Software
- 01Rivet, a YC W23 Company, Runs Agent Actors on S3 and Claims 72 KB EachThe Apache 2.0 orchestrator says an actor cold-starts in 4.99 ms at p99. Every figure is Rivet's own, and the Launch HN thread questioned both the premise and the AWS bill.
- 02GitHub Actions Lockfiles: What gh actions-lock Pins and What It Leaves LooseWorkflow lockfiles went generally available on October 28. The extension's own README lists four gaps, including Docker images and reusable workflows in other repositories.
- 03C++26 Reportedly Fails Its ISO Ballot, With Contracts the Sticking PointSecondary reports put the October 23 vote at 12 yes of 20 and nine no votes of 22 bodies. The numbers do not reconcile, and no ISO or WG21 record has been published to settle them.
- 04A Postgres Benchmark Found jsonb Read One Key 2.7 to 10.6 Times Faster, and Larger on Disk in 6 of 11 CasesChris van Eijk of Now-Next loaded 1 million small documents and 5,000 large ones into PostgreSQL 17.11 and 18.6. The speed result is consistent. The storage result depends on whitespace and the 2 kB compression threshold.