CLI-Anything's "100%" Test Claim Has Three Totals
Software / news
CLI-Anything's "100%" Test Claim Has Three Totals
The open-source project's README cites 2,461, 2,330 and 2,464 as its total test count, while its own issue tracker describes commands that report success without doing anything.
CLI-Anything, an open-source framework built by the University of Hong Kong's Data Intelligence Lab, gained 415 stars on GitHub's trending page on Sept. 24. The project's pitch is in its own tagline: "Making ALL Software Agent-Native," turning ordinary desktop programs into command-line tools an AI agent can drive directly. Its README backs that up with a specific claim, "100% pass rate" across its test suite. The README's own numbers do not agree on what that total is.
A summary table near the bottom of the file lists a grand total of 2,461 tests. One line down, the same document states the total differently: "1,732 unit tests + 579 end-to-end tests + 19 Node.js tests," which adds to 2,330. A separate sample terminal output further up the page, meant to show what a real test run looks like, prints a third figure: "TOTAL 2,464 passed." None of the three numbers match any of the others.
What the passing tests miss
| Where the number appears | Total claimed |
|---|---|
| Summary table | 2,461 |
| Prose breakdown (1,732 + 579 + 19) | 2,330 |
| Sample terminal output | 2,464 |
The project's own issue tracker shows what a passing test can miss. In issue #451, filed Aug. 25, a user reported that CLI-Anything's Blender harness prints a successful render report and the shell command that would run it, but "never actually calls subprocess.run() (or equivalent)": no Blender process starts, no file is produced, and the command still exits 0. In issue #406, filed July 22, the Inkscape harness's union and difference operations were shown to consume both shapes a user combined and emit an empty path, while reporting success, which the filer described as leaving "an agent following the documented workflow" with "fewer objects than it started with, a success message in its context, and no signal that anything went wrong." Neither issue shows a maintainer response as of publication.
A third issue, #465, filed Sept. 12, found a wider pattern: only 13 of the project's 68 generated harnesses are actually published on PyPI as the documentation instructs, with the other 55 returning a 404 when installed as described, and 10 harnesses contain no code that calls a subprocess, an HTTP client or any other mechanism that could reach the software they claim to control.
A license nobody can find
A separate issue, #475, filed Sept. 20, found that while the repository's root license is Apache-2.0, 52 of its 67 individual harness packages declare MIT in their own setup.py files, two declare Apache, and the rest set no license field at all. None of the harnesses ship a license file inside the package itself, so a copy installed from PyPI shows license: null. A pull request to align the metadata, filed the same week, was still open as of Sept. 24.
The lab behind the project, run by Chao Huang, an assistant professor in HKU's Department of Computer Science, has integrated more than 40 applications into the framework, including Blender, GIMP, LibreOffice and FreeCAD, according to the README. The gap between that list and a codebase where a documented command can silently destroy a user's work is the kind of detail GitHub's star count does not capture, and it sits alongside a broader pattern this site has tracked in agent tooling that grows faster than its own security review or its maintainers' ability to keep up with it.
Sources
More in Software
- 01Zigzag, a 35 MB Mac Browser, Courts Arc Users 35 Days Before ShutdownThe WebKit-based browser imports Arc spaces and pinned tabs, but its README lists no sync, no Boosts and a partial Chrome extension API.
- 02Nimony 0.6.2 Sets a Year-End Goal for Nim 3 With 44 Days LeftThe compiler that will become Nim 3 is MIT-licensed and builds natively without LLVM, but its FAQ says it cannot yet compile Nim 2 code unchanged.
- 03Alloy CSS Runs Menus and Tabs Without JavaScript, but Dialogs Need Safari 26.2The 24 KB MIT stylesheet leans on popover and invoker commands, so its no-script promise has browser floors that differ by component.
- 04Windows 11 Telemetry Flags Which Coding Agents You Run, Researcher FindsXusheng Li traced a service-experiment feature in build 26200.9974 that matches executable names such as claude.exe and queues the result as optional diagnostic data.