Google Research's Harness2 Has 10,700 Stars and Three Commits, and Publishes No Absolute Scores

Harness2, a Google Research repository for improving the prompts, tools and memory around a language model without touching its weights, has 10,700 GitHub stars, 956 forks and three commits on its main branch. Its README reports gains of up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB, and gives no absolute score for any of them.

The repository describes itself as "not an officially supported Google product," and says it is not eligible for Google's Open Source Software Vulnerability Rewards Program. The code is Apache 2.0.

What the framework does

A harness is the scaffolding an agent runs inside: its prompts, tools and memory. Harness2 works on that layer in two loops. Task-level recursion refines the harness for the task at hand through a Propose, Probe and Compose step. Harnessing-level recursion keeps the harnesses that were adopted, and the lessons from each task, in a persistent improvement harness for later tasks.

The --k flag sets the budget. --k 1 runs one step. --k K --mode parallel probes K general and K domain-specific candidates and produces K compositions, while --k K --mode sequential refines the harness through K successive steps. It supports OpenCode and Codex as executors.

The companion paper is arXiv:2610.14224, "Harness^2: Recursive Agent Harnessing for an Open World," submitted Oct. 9, 2026 under cs.AI by Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee and Tomas Pfister. Its summary says the approach was tested on three professional benchmarks and four model-harness configurations, and that a single step improved GLM-5 and Gemini 3.5 Flash over their base harnesses.

What the numbers leave out

The gains are the vendor's own, from the authors' paper and README, and no outside party is named as having reproduced them. More important, they are differences. The README says larger task-level recursion budgets bring further gains without quantifying them, and the arXiv page gives no baseline scores to subtract from.

1. LAB11.8 points (LAB is percentage points of partial pass rate)
2. WorkBuddy-Bench11.5 points (LAB is percentage points of partial pass rate)
3. JobBench4.5 points (LAB is percentage points of partial pass rate)

Source: Harness2 README, google-research/harness2, accessed 2026-10-19

A gain of 11.5 points from a base of 20 is a different result from 11.5 points from a base of 70. The README also does not say what a run costs. It says only that "Model-backed runs incur API costs" and suggests starting with a small task selection.

One of the repository's open proposals is issue #4, a --max-usd spend limit for harness2 runs, opened Oct. 16. The README documents no spend cap, so a --k value set high on a model-backed benchmark is paid for in API charges with nothing in the tool to stop it.

The arXiv page lists nine authors but no affiliations in the text available for this report. The repository sits under Google Research's GitHub organisation, and the README's disclaimer is the only statement of how Google relates to the code.

The sandbox warnings in the README

The README lists limits that matter before running it. Executor network access is unrestricted, and Codex runs with its sandbox disabled. Public benchmark rubrics may be accessible online, so the README says to configure network restrictions before evaluation. An agent that can reach the internet while being scored on a public benchmark can look up answers.

Setup needs Linux, Python 3.12 and uv, plus Node.js 20 or later and the Google Cloud CLI for model-backed benchmarks, Pandoc and Poppler for LAB, and Docker for WorkBuddy-Bench. It also needs a Vertex AI project with model IDs set in a .env file. A local demo needs no credentials.

Sixteen open issues, most of them about bookkeeping

The issues page lists 16 open issues, with the earliest dated Oct. 16 and the latest Oct. 19, plus 64 open pull requests. Several concern whether results can be trusted.

Issue	Opened	Problem as titled
#80	Oct. 19	judge_one trusts all_pass and n_passed from the payload, so a passing task reads as a failure
#77	Oct. 19	LAB runner silently fails when the timeout command is unavailable
#65	Oct. 18	Resume and rollout caches silently reuse outputs after task inputs change
#50	Oct. 18	harness2-report undercounts improver spend for tasks re-run after an interrupted invocation
#44	Oct. 18	harness2-judge keeps an old grade after a task is re-run if the re-judge fails

Issue #39, also from Oct. 18, is a related-work citation request pointing to Project Peritus's harness evolution. Issue #71 is titled "Harness^2 analyzed by Claude and compared with similar systems."

The two grading bugs, #80 and #44, bear directly on the headline numbers, because both concern how a task's score is recorded. The page does not say whether the paper's figures were produced before or after either bug appeared.

What to check before trusting it

Harness2 is a research artifact with a star count that outpaces its history: 10,700 stars against three commits. Stars measure attention rather than use, and the site's report on CLI-Anything covers another agent repository with a large star count. For a different way agents get tested, the Vals refund study pitted six of them against a task they could not complete.

The next check is the paper's full text, which would hold the baseline scores that neither the README nor the arXiv summary gives.