Cloudflare's Viral Audit Skill Meets a Rival's Blind Test

Cloudflare's security-audit-skill, a coding-agent skill that turns an AI agent into a vulnerability hunter, added roughly 3,006 stars on GitHub on Friday, per GitHub's trending page. The repository has 13,673 stars in total, meaning close to a quarter of everything it has ever earned arrived in a single day, three months after Cloudflare published it.

The spike coincides with a public benchmark that Cloudflare has not yet answered. On Wednesday, a GitHub user named HetCreep filed an issue against the repository laying out a blind, three-round comparison between the skill, a competing "multi-lens" review pipeline the poster built, and a plain single-agent control. The poster disclosed the conflict upfront: "the multi-lens pipeline is our own tool, so we have an interest here." The issue, and Cloudflare's silence on it, is still open.

The skill is one of several AI coding tools this desk has covered. Alibaba added IntelliJ support to its own Open Code Review plugin, and researchers separately traced a chain of bugs into OpenAI's internal repositories that started with a single sign-on misconfiguration.

What the tool claims, in Cloudflare's own numbers

Cloudflare's engineering post describing the harness behind the skill was written by Grant Bourzikas, the company's chief security officer, and published June 18. It says the system generated 20,799 raw findings across 145 internal repositories, of which 12,057 survived automated validation and 7,245 were sent to engineering teams as actionable. Cloudflare said the validator's rejection rate for weak findings improved from 40 percent to 11 percent as the pipeline matured, and that the share of findings backed by an executed check, rather than a plausible-sounding read, climbed from 35 percent to 58 percent. On a roughly 30,000-line repository, Cloudflare said a standard scan compressed 100 initial findings into 80 distinct bugs in about 14 hours, with automated patches averaging five minutes each.

What the blind test found

HetCreep's comparison ran three rounds per arm against one seeded target, scored blind by someone who had not watched any run, using a pre-registered answer key. On the metric the skill is built for, it held up: median precision of 90 percent across rounds, with zero hits on either planted decoy, and every confirmed claim carrying a source trace back to the code. The post names six class-level gaps anyway.

Two were coverage blind spots: a real, disclosed vulnerability in a pinned dependency went unfound in all three rounds, because the skill's sandbox has no network access to check an advisory against a cutoff it cannot see past, and a locale-specific formatting bug never got logged as a finding at all, not even as one the skill declined to pursue. A third was fragility: the skill's fastest "quick" profile runs exactly one wave of hunting agents, so a single malformed response from one of them left a whole category of easy, common bugs uncovered in one of the three rounds, with no second wave to recover. A fourth concerned judgment calls the skill made on purpose: it downgraded a real, exploitable credential-handling flaw to a hardening note because the access path was not active in the test deployment's current configuration, a rule HetCreep called "deliberate and well-documented" but one that can read to a user as "not a real defect" rather than "not reachable today."

The fifth gap is the one with a dollar figure attached. HetCreep priced a median run of the skill's quick profile at $29.95, against $7.66 for their own multi-lens pipeline and $2.06 for a plain single-agent review, all on the same model tier and the same target. Recall, the share of the pre-registered answer key each method actually found, came out identically for the skill and the free-standing single-agent review: 0.467 each, across 15 seeded facts. The pipeline scored 0.40 recall points higher than both. On this one target, the skill's extra spend did not buy any extra recall over the cheapest option that was run.

1. security-audit-skill (quick)29.95 $
2. Multi-lens pipeline (HetCreep's own tool)7.66 $
3. Single-agent review2.06 $

Source: GitHub issue cloudflare/security-audit-skill#20, accessed 2026-09-19

The sixth finding tested the skill unattended, at the weakest available model tier. Precision and the zero-decoy record both held, but recall fell to 27 percent from 47 percent at the strong tier, and in two of the three rounds the finished report was written to a scratch directory instead of the one requested, because the run's permissions did not extend to the target folder. HetCreep called that a limit of the run's permission grant rather than the model, and said the same design would reproduce independently on any target.

What would change this read

The sample is one seeded target, three rounds per arm, with the strongest counter-evidence coming from a rival tool's own maker. HetCreep called it "a measured signal, not a verdict" and offered the underlying method for anyone to rerun. Cloudflare had not commented on the issue as of Friday, and an open pull request against the repository, numbered 13, already proposes a retry path for a different hunter-failure mode, a safety-filter refusal, without touching the malformed-return gap HetCreep flagged in the quick profile. Whether Cloudflare responds to the benchmark, or whether the skill's next release closes the gap, is the next thing to watch on a repository whose growth chart just went vertical.