CLI-Anything's "100%" Test Claim Has Three Totals

CLI-Anything, an open-source framework built by the University of Hong Kong's Data Intelligence Lab, gained 415 stars on GitHub's trending page on Sept. 24. The project's pitch is in its own tagline: "Making ALL Software Agent-Native," turning ordinary desktop programs into command-line tools an AI agent can drive directly. Its README backs that up with a specific claim, "100% pass rate" across its test suite. The README's own numbers do not agree on what that total is.

A summary table near the bottom of the file lists a grand total of 2,461 tests. One line down, the same document states the total differently: "1,732 unit tests + 579 end-to-end tests + 19 Node.js tests," which adds to 2,330. A separate sample terminal output further up the page, meant to show what a real test run looks like, prints a third figure: "TOTAL 2,464 passed." None of the three numbers match any of the others.

What the passing tests miss

Where the number appears	Total claimed
Summary table	2,461
Prose breakdown (1,732 + 579 + 19)	2,330
Sample terminal output	2,464

The project's own issue tracker shows what a passing test can miss. In issue #451, filed Aug. 25, a user reported that CLI-Anything's Blender harness prints a successful render report and the shell command that would run it, but "never actually calls subprocess.run() (or equivalent)": no Blender process starts, no file is produced, and the command still exits 0. In issue #406, filed July 22, the Inkscape harness's union and difference operations were shown to consume both shapes a user combined and emit an empty path, while reporting success, which the filer described as leaving "an agent following the documented workflow" with "fewer objects than it started with, a success message in its context, and no signal that anything went wrong." Neither issue shows a maintainer response as of publication.

A third issue, #465, filed Sept. 12, found a wider pattern: only 13 of the project's 68 generated harnesses are actually published on PyPI as the documentation instructs, with the other 55 returning a 404 when installed as described, and 10 harnesses contain no code that calls a subprocess, an HTTP client or any other mechanism that could reach the software they claim to control.

A license nobody can find

A separate issue, #475, filed Sept. 20, found that while the repository's root license is Apache-2.0, 52 of its 67 individual harness packages declare MIT in their own setup.py files, two declare Apache, and the rest set no license field at all. None of the harnesses ship a license file inside the package itself, so a copy installed from PyPI shows license: null. A pull request to align the metadata, filed the same week, was still open as of Sept. 24.

The lab behind the project, run by Chao Huang, an assistant professor in HKU's Department of Computer Science, has integrated more than 40 applications into the framework, including Blender, GIMP, LibreOffice and FreeCAD, according to the README. The gap between that list and a codebase where a documented command can silently destroy a user's work is the kind of detail GitHub's star count does not capture, and it sits alongside a broader pattern this site has tracked in agent tooling that grows faster than its own security review or its maintainers' ability to keep up with it.