Billion Context Calls Its 1,500-Item Benchmark Third-Party, but the Results Sit in the Author's Own Repo

Billion Context, a proxy that compresses long coding-agent sessions, gained 460 GitHub stars on Nov. 8 while describing its benchmark as an "independent third-party" evaluation that names no evaluator and sits in the author's own repository.

The project belongs to a user named ranxianglei. GitHub's API shows a repository created on Aug. 6, with 5,022 stars, 401 forks and 927 open issues and pull requests combined as of Nov. 8. The licence field reads NOASSERTION: the README confirms a LICENSE file exists but does not name its type, and says only that the accompanying paper is MIT-licensed.

What Billion Context claims

The README describes a proxy that "sits between any agent and its model API" and rewrites Anthropic and OpenAI streams. The model decides what to fold into summaries, using injected tools named compress, decompress, search_context and acp_status. It lists native plugins for pi, opencode, kimi, hermes and claude, among others.

Its headline claims are about 5 times fewer tokens, a window of about 100,000 tokens for sessions that run to billions, a prefix-cache hit rate of 95 to 97% in a healthy session, and compression overhead of 2% or less. A preprint cited in the README reports a 4.5-month study covering 174,327 model calls and 18.76 billion cumulative input tokens.

For readers who followed one developer's move from Claude Code to Pi, Pi is among the supported hosts. It is a project-specific plugin, not a generic hook.

The benchmark that is called independent

The README's results come from a folder named benchmarks/third-party-1500. Its own README calls the run an "independent, strictly controlled third-party evaluation." It names no evaluator. The folder sits under the same ranxianglei account as the project, so the page does not establish that anyone other than the developer ran it. It carries no date, only the tested version, v0.1.202.

The setup is 1,500 items, 500 each from LongMemEval-S, ContextBench and MemoryAgentBench, with five arms: Billion Context, a full-context baseline and three retrieval pipelines (BM25, embedding, embedding plus rerank). All arms used DeepSeek V4.1 Flash as the base model. The ContextBench runs used Pi CLI v0.73.1.

Benchmark	Billion Context	Comparison arm
LongMemEval-S	79.20%	62.60% (best retrieval arm)
ContextBench, Utilized F1	0.3097, first place	Not given on the page
MemoryAgentBench, 1.48M-token test	100% survival	Full context failed 199 of 199

Cost is where the README is most specific: 58.3% lower than full context on ContextBench and 71.1% lower on LongMemEval-S, using the author's own per-million-token prices.

1. LongMemEval-S71.1 %
2. ContextBench58.3 %

Source: Billion Context benchmarks README, accessed 2026-11-08

The 199-of-199 failure is the full-context arm overflowing its window. The page gives no context size for the benchmark model, so a reader cannot tell whether any long-context setup would have hit the same limit.

A paper on a different gateway warns against this reading

The README says its accounting follows "Full Cost Attribution" from arXiv:2609.22114. That paper, by Luzhuo Chen and Jiayu Shi, was submitted on Aug. 19 and studies a different product, a gateway called Paritok.

Its finding is that compressing file reads and tool output saved only about 2% of the cached prefix per turn, and that a strong single-shot compression result should not be cited as evidence of multi-turn savings. Billion Context's numbers are not covered by that paper. It is the only outside measurement method the README points to.

Install warnings the README itself carries

The README says the proxy has no authentication and should only face trusted networks. It performs certificate-based interception of traffic to whitelisted model hosts. Rewriting the base URL disables some account-bound Claude Code features, including Remote Control, and Claude Desktop's Code tab bypasses the proxy. The first compaction waits until about 200,000 tokens by default, and the master branch "can be broken."

Related reading: Anthropic's agent memory study reports a score with and without memory, run by the lab that built the model.

The raw data and evaluation code were not visible on the pages read for this report. Until the folder's contents or a named evaluator appear, the percentages above are the developer's.