NVIDIA SkillEvaluator Measures Skill Lift, Where Static Scans Fall Short
Software / news
NVIDIA SkillEvaluator Measures Skill Lift, Where Static Scans Fall Short
The Apache-2.0 tool pairs three tiers of checks with a paper reporting a 0.14 correlation between scans and LLM judges across 145 real skills.

NVIDIA's open-source SkillEvaluator checks agent skills in three tiers, and the August paper behind it reports that structural scans and LLM-judge scores for 145 real skills correlate at only 0.14.
The tool lives in a GitHub repository under the Apache-2.0 licence, with about 2,900 stars, 387 forks and 13 open issues on Nov. 17. The README labels support "Experimental": community-supported, best-effort, with no SLA. It is part of what NVIDIA calls its Verified Skills pipeline, a term the README does not define.
What the three tiers do
A skill, in the Agent Skills format, is a folder of instructions that a coding agent loads when a task matches. SkillEvaluator checks one in three steps, and only the last asks whether the skill helps.

| Tier | What it checks | What it needs |
|---|---|---|
| 1, Validation | Safety, format, harness compatibility | SkillSpector and Gitleaks, Semgrep optional |
| 2, Deduplication | Overlapping guidance across skills | Embeddings and chat models |
| 3, Live evaluation | Whether the skill changes agent results | Provider access, an agent, a Docker sandbox |
If a Tier 1 scanner cannot run, the README says the result is INCOMPLETE and the command exits nonzero, so a CI job fails rather than passing on a missing check. The bundled text-analyzer skill scores 88.8 out of 100, a B, on the static quality check against a default passing threshold of 70.
A separate harness check uses no LLM. It reports whether claude-code, codex, opencode, hermes, openclaw and copilot-cli load a given skill. The rubric evaluation does send skill documentation to a model, and the README says NVIDIA Build's free hosted endpoints are rate-limited and that model and sandbox use can incur charges.
What the paper measured
The paper, "Evaluating Skills, Not Just Agents," was submitted Aug. 20 by 12 listed authors led by Christopher Kevin. Its abstract page does not show affiliations, and says an open-source implementation is available as NVIDIA SkillEvaluator. The method runs paired trials with and without a skill, using the same model, sandbox and grading policy, and reports the difference as Skill Lift.
Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift was 0.2134, with a 95 percent confidence interval of 0.1967 to 0.2301. Composite lift was positive in 72.8 percent of cases, which means it was zero or negative in the other 27.2 percent.
- Composite lift0.21 lift
- Outcome-only lift0.18 lift
Source: arXiv:2608.20614, accessed 2026-11-17
Outcome-only lift, the average of accuracy and goal accuracy, was 0.1799. The paper says process metrics improved most in skill execution, behavior check and skill efficiency. These are the authors' figures for their own task set, and the abstract does not say who built the 64 skills or the tasks.
Where the caveats sit
The 0.14 correlation is the finding that explains the product. If static checks and judges tracked real usefulness, Tier 3 would be redundant. The authors argue they do not, so a skill can lint cleanly and still do nothing.
Tier 3 costs more and covers less than the headline suggests. The README says its autopilot mode may create only one skill-grounded case with limited coverage, and that sandbox setup adds overhead. Quality checks alone leave publication evidence INCOMPLETE.
Skill-adjacent tooling is multiplying on this beat, from Plannotator, which reviews agent plans locally, to Chrome DevTools MCP and its 63 tools. SkillEvaluator aims at the instructions an agent loads, not the tools it calls.
For a team that already ships skills to Claude Code or Codex, the practical entry point is skillevaluator validate ./my-skill --tiers 1, which needs no model. The README links a performance dashboard in its documentation, and that is where per-skill lift numbers would show whether 0.2134 holds outside the paper's own set.
Sources
More in Software
- 01Zigzag, a 35 MB Mac Browser, Courts Arc Users 35 Days Before ShutdownThe WebKit-based browser imports Arc spaces and pinned tabs, but its README lists no sync, no Boosts and a partial Chrome extension API.
- 02Nimony 0.6.2 Sets a Year-End Goal for Nim 3 With 44 Days LeftThe compiler that will become Nim 3 is MIT-licensed and builds natively without LLVM, but its FAQ says it cannot yet compile Nim 2 code unchanged.
- 03Alloy CSS Runs Menus and Tabs Without JavaScript, but Dialogs Need Safari 26.2The 24 KB MIT stylesheet leans on popover and invoker commands, so its no-script promise has browser floors that differ by component.
- 04Windows 11 Telemetry Flags Which Coding Agents You Run, Researcher FindsXusheng Li traced a service-experiment feature in build 26200.9974 that matches executable names such as claude.exe and queues the result as optional diagnostic data.