Trail of Bits Finds 21 Regressions in 130-Plus Merged Patches With a Codex Plugin, and None Touched Security
A.I. / news
Trail of Bits Finds 21 Regressions in 130-Plus Merged Patches With a Codex Plugin, and None Touched Security
The security firm's regression-hunt plugin cut false positives 25% against a baseline Codex prompt, while the Patch the Planet dashboard shows 466 of 1,995 reported issues patched upstream.
Trail of Bits published two open-source Codex plugins on Oct. 21 and reported that one of them, regression-hunt, confirmed 21 regressions across more than 130 pull requests already merged upstream. None of the 21 broke a security protection.
The post is by Anish Naik and Marcelo Morales and does not list their job titles. Both plugins came out of Patch the Planet, the vulnerability-patching programme Trail of Bits runs with OpenAI.
What the 21 regressions were
Trail of Bits defines a regression as a new defect that "breaks previously correct and supported behavior along a path a real user or attacker can reach". Of the 21, 16 were functional, 1 was a performance regression and 4 were build or test failures.
A single run of the plugin produced 58 candidates at about 36% precision, the post says. Against a baseline Codex agent prompted to "review the PR for regressions", regression-hunt produced 25% fewer false-positive findings, with a 95% confidence interval of 14% to 38%.
The authors concede the cost. "Fewer false positives therefore came with a tradeoff in how many known regressions it could confidently report," they wrote.
How the plugins work
The regression-hunt README lists seven skills. They pin the sources, hunt for reachable candidates, deploy the before and after states, run a proof of concept, replay it with a fresh agent, and adjudicate against the written definition.
The post's rule is "a live proof of concept for every claim". A step that cannot produce one returns an inconclusive result.
The sibling plugin, patching-bugs, uses what the post calls a three-arm proof. The original code must fail, the patched code must pass, and a full revert must fail again.
Requirements are Python 3.13 or later, uv, and Git with check-attr --source support. The README page did not state a licence, and it says candidates stay unresolved when the host lacks an agent dispatcher or execution tools.
The post asks maintainers to write a THREAT_MODEL.md, plus a PATCHING_GUIDELINES.md or AGENTS.md covering build steps, fix size and regression test requirements. The plugins read those files.
The Patch the Planet numbers
Patch the Planet was announced on June 22 with OpenAI and Trail of Bits. CSO Online reported "hundreds of security issues and merged dozens of patches" at launch. The programme's dashboard now shows 68 codebases and 1,995 issues found and reported.
- Awaiting a patch1316 issues
- Patch written679 issues
- Merged upstream466 issues
- Open upstream213 issues
Source: Trail of Bits Patch the Planet dashboard, accessed 2026-10-21
The dashboard counts 1,316 issues, 66%, as awaiting a patch, and 679 with a fix written. Of those, 466 (23% of all reported issues) are merged and 213 are open upstream.
Participants include Python, cURL, Sigstore, aiohttp, SciPy and Ladybird. A SciPy maintainer is quoted on the page: "The findings were all correct, realistically assessed for threat level, and came with working patches."
The offer is one Trail of Bits engineer for a full week on each selected project, patches submitted rather than only findings, and six months of ChatGPT Pro for selected maintainers.
What the numbers do not show
The post's 21 regressions come from a review of merged pull requests, and the 36% precision figure means roughly two of every three candidates in a run were not confirmed.
Devashri Datta, an open-source cybersecurity architect, said in CSO's June coverage that AI-driven findings need verifiable auditability, meaning an organisation must be able to trace how the AI flagged code, validated the exploit and assessed patch impact.
For how another lab's model has been pointed at a large open-source codebase, see the OpenSSH attack-surface report. cURL, a Patch the Planet participant, is the subject of this vulnerability report roundup.
Trail of Bits said it plans an evaluation of whether patches fix a bug and its variants without regressions, and an analysis of Codex traces to see which steps the agent skips.
Sources
More in A.I.
- 01Eka Robotics Raises $150 Million at $1.45 Billion for a Force-Sensing Robot Model With Unnamed CustomersThe vision-force-action model claims 1,500 picks an hour against a 600 human baseline, with every figure supplied by the company.
- 02DeepSeek Claims 42.3 on Terminal-Bench 4.0 for V4.1-Flash-1017; Vals AI Measured 31.8The MIT-licensed update shipped Oct. 17 with 1M context and API prices from $0.15 per million input tokens, but the card never states a parameter count.
- 03RuneBench Has AI Agents Grind RuneScape Skills, and Its Leaderboard Puts GPT-6 Astra on Top at 7.3%The benchmark's own page publishes no scores, and the aggregator that does says the numbers mix model quality with harness choices.
- 04OpenAI's Rosalind Academic Plan Gives US Labs 30 Free Seats for 12 Months, Without API AccessApplications opened Oct. 15, and the model itself still sits behind a separate trusted-access review.