Cloudflare's Security-Audit Skill Started as 450 Lines, and Its Own Numbers Show One Run Finds About Half the Bugs

Cloudflare's security-audit skill, an MIT-licensed package that turns a coding agent into a multi-phase security auditor, was the fastest-rising repository on GitHub's daily trending list on Oct. 23. The README is short. The blog post it grew out of is where the useful numbers are, and they say more about the limits of agent-driven auditing than about its promise.

Cloudflare published the skill at github.com/cloudflare/security-audit-skill and described its origins in a June 18 post, Build your own vulnerability harness, by Grant Bourzikas. The post says the work began as a roughly 450-line skill run against one repository, with prompts tuned until it surfaced real bugs.

What the skill does, and where the documents disagree

The README describes "a coding-agent skill for multi-phase security audits with independently verified, machine-readable findings." It lists six phases: reconnaissance, coverage-led hunting, candidate validation, structured output, independent record verification and target-neutral reporting.

The blog post calls the original skill a seven-phase audit. Neither document says which phase was merged or dropped between the June post, which was last modified July 15, and the README as it stands now. The mismatch is small, but it means the public skill is a reworked version of the one the post measured, and the post's numbers describe the fleet harness, not this package.

Installation is one command, npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit. The README asks for a coding agent whose model supports tool use and parallel sub-agents, and Node.js for two zero-dependency validators. It does not name a supported agent.

The design rule worth reading closely is this one: "The agent that checks a finding is never the agent that found it." Each candidate goes to a fresh verifier whose job is to disprove it, and results land in findings.json as confirmed, needs_validation or rejected.

The sandbox requirement is the catch

To confirm a bug, the workflow may build and run the target's own code. The README says that needs an OS-enforced sandbox with external networking disabled, a sanitised environment, resource limits and restricted writes. "Without these controls, the workflow keeps the lead as needs_validation instead of executing target code."

That is the honest default, and it is also friction. A developer who installs the skill on a laptop without a sandbox gets a report full of unconfirmed leads. Agent sandboxes are a live problem on their own; the NVIDIA OpenShell release coverage covers one runtime built for exactly this, and its breaking changes show how young the category is.

What one run misses

The README states its own coverage problem: "a single run found roughly half of the vulnerabilities that repeated runs found in total." Cloudflare's post adds that the early findings skewed toward simpler bugs. Multiple runs are additive in the README's design, and stale or unresolved work is not counted as covered.

So a clean single audit is weak evidence. If half the bugs a repeated process would find are missing from run one, a report with zero confirmed findings says the first pass found nothing, not that the code is safe.

The post is equally direct on the other side of the ledger: "We don't claim a false-negative rate for our system." There is no labelled ground-truth set behind the fleet numbers.

The numbers from 128 repositories

Codifying the skill into a fleet scanner took about six weeks, according to the post, and covered 128 repositories. Cloudflare reports its lifetime figures at the time of publication in a validation pool of 13,841 findings across 145 repositories.

Stage in the validation pool	Findings
Pool at publication	13,841
Folded away as duplicates	5,442
Routed as wrong-repo or low-risk	1,154
Left for engineering teams	7,245

The arithmetic holds: 13,841 minus 5,442 minus 1,154 is 7,245. Before the validation pool, the discovery harness produced 20,799 raw candidates, of which about 12,057 survived validation, the post says.

Two other figures describe the cost of trusting agents. Initial validation rejections fell from 40% to 11% as the prompts improved, and findings Cloudflare rates high-integrity rose from 35% to 58%. Agents also tend to edit source code so their exploits work, or to write tautological tests, which is why each confirmed finding ships with a proof-of-concept test that runs against the original, untouched codebase.

What Cloudflare has not released

The full harness is promised later; the repository holds only the skill. The post lists the pieces missing from it: a SQLite database keyed by run, repo and stage that lets work resume after a crash, a tracer that walks dependency graphs to spawn hunts in consumer repositories, and a fixer.

The fixer averages about five minutes per bug and never merges code, so a human reviews every branch. Cloudflare calls it the "youngest and slowest" part of the system. On a roughly 30,000-line repository, the post says, a run produced about 100 initial findings that compressed to about 80 distinct bugs, and roughly 10 critical or high-severity, exploitable ones were fully patched in about five days.

Cloudflare also sells a related product. On Sept. 7 it announced Vulnerability Discovery and Remediation, in early access for select Enterprise customers through Cloudflare Managed Defense, built on OpenAI's Daybreak models including GPT-5.6 Cyber, eSecurity Planet reported. That article does not mention the open-source skill, and Cloudflare's post does not say how the two relate.

Stars are not adoption, and none of the figures above come from anyone outside Cloudflare. The post says the experiment was ring-fenced and that its numbers do not represent active, unpatched production vulnerabilities. The next public checkpoint is the harness release the post promises, which has no date. Another agent project that spread fast, Omnara, drew its own scrutiny over defaults, a reminder to read any agent tool's configuration before pointing it at code you care about.