Colibri Runs Giant Models From Disk at 0.1 Tokens a Second on a Laptop; Its Own Issue Tracker Says Why

Colibri, a pure-C inference engine from a single developer, has reached about 46,700 GitHub stars by promising to run very large models on ordinary hardware. Its own README puts GLM-5.2 on a 25 GB laptop at 0.05 to 0.1 tokens a second, which is one token every 10 to 20 seconds.

The project is by Vincenzo Fornaro (GitHub handle JustVugg) and is licensed Apache 2.0, with a copyright line reading "Copyright 2026 Vincenzo Fornaro." The README describes it as "Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk." Stars measure interest, not adoption, and the page shows 5,150 commits and 65 open issues.

How colibri streams experts from disk

A mixture-of-experts model activates only a few of its experts for each token. Colibri keeps the dense part, meaning attention, shared experts and embeddings, in RAM. The routed experts stay on disk and are read when the router asks for them, through a cache that learns which experts a user's prompts keep hitting.

A GPU, when present, holds the most-used experts. An expert's three matrices are read in one pread call, and a pool of loader threads fetches missing experts while resident ones compute. The README says a second drive holding a copy of the model, set with COLI_MODEL_MIRROR, measured 37.5% faster decode across two independent NVMe drives.

The disk is the engine's real hardware requirement. The README says speed depends mainly on the drive for large models, and the downloads are large: GLM-5.2 is 429 GB, DeepSeek V4 Flash is 167 GB and Kimi K3 is 1.56 TB. The smallest model needs 8 GB of RAM and 22 GB of free disk.

What the speed table says

The README's tables are the author's own measurements unless a number is marked otherwise. Machine D, an i5-13500 with 62 GB of RAM and a 24 GB RTX PRO 4000 Blackwell SFF, ran each model under coli serve with a 128-token streamed answer, best of two runs.

Model on machine D	CUDA tok/s	Vulkan tok/s	CPU tok/s
Qwen3.6-35B-A3B UD-Q4_K_M	104.8	92.9	13.9
LFM3-VL-2B Q4_K_M	235.1	214.1	34.5
OLMoE-1B-7B Q4_K_M	354.6	299.4	47.4
Qwen3.8-Flash-Next IQ2_XS (68 GB)	51.4	50.6	9.3

Those are models that fit in memory. The headline models are a different story, and the README lists them separately.

1. GLM-5.2, six RTX 5090s9 tok/s
2. GLM-5.2, Ryzen AI Max+ 395, 128 GB, CPU1.83 tok/s
3. DeepSeek V4 Flash, Ryzen 7 5800X, 32 GB, CPU0.93 tok/s
4. GLM-5.2, 25 GB laptop, cold0.05 tok/s

Source: colibri README, github.com/JustVugg/colibri, accessed 2026-11-03; the six-GPU and laptop rows are the low end of the README's ranges

The README gives the six-GPU figure as 9.0 to 9.2 and the laptop figure as 0.05 to 0.1; the chart shows the low end of each. Kimi K3 is listed at about 9.4 seconds per token. The README also says a blank cell in its tables means nobody has measured that model yet, and one Qwen3.6 result on a Tesla V100 16 GB, 17 to 19 tok/s, came from a user in issue #1852 rather than from the maintainer.

The DRAM wall is documented and unsolved

Issue #537, opened on July 22, 2026 by a contributor named ZacharyZcR, analyses the ceiling on a 6x RTX 5090 rig with dual Xeon Silver 4510 CPUs and 251 GB of DDR5. The author wrote that "DRAM bandwidth is the wall," 15 to 28 times below GPU memory bandwidth. About 12.2 GB is read per token, and measured throughput of 7.9 tok/s on a tuned placement was about 96% of the author's estimated ceiling.

A prototype that streamed RAM experts to the GPU ran 4 times slower, 4.24 tok/s down to 1.16. On an Apple M5 Max with 128 GB the run was disk-bound at about 2.1 tok/s. The maintainer closed the issue on September 5 and wrote: "The DRAM bandwidth wall itself is not solved and this closure does not pretend it is."

That sentence is the project's most useful disclosure. A reader who sees 9 tok/s on six RTX 5090s should know the card count is not what limits it.

Open issues a downloader should read first

The issue list shows correctness problems on specific platforms. Issue #2355, opened October 24, reports NaN activations from the first MoE layer on Windows with Vulkan. Issue #2287 reports a Qwen3-Coder int4 checkpoint with 43 of 48 layers. Issue #2211 reports that Qwen3.6 output on a GTX 1650 is not reproducible at temperature 0, and the README itself warns that Vulkan and CUDA output can drift from the CPU's by a word on long answers.

Issue #2189 reports Windows decode about 5 times slower than llama.cpp. Issue #2063 asks for secure browser authentication on the dashboard of API-key-protected servers. The README gives no dedicated guidance on binding the server beyond showing 127.0.0.1:8000, and says a server started without COLI_API_KEY accepts any key a client sends.

Licences, and what GLM-5.2 is

Apache 2.0 covers the engine only. Each model keeps its authors' licence: Z.ai's GLM-5.2 card lists MIT, BF16 and F32 tensors, a 1M-token context and 753B parameters on the Safetensors card. Issue #537 calls the same model 744B, and the card does not state active parameters. The README says Qwen-Image-2.1 is limited to non-commercial use.

Colibri suits someone with a fast NVMe drive, patience and a model that mostly fits in memory, where its table shows 50 to 350 tok/s. It does not suit anyone who wants a 753B model to feel interactive. Readers following the recent run of fast-rising agent repositories can compare how little most of them say about measurement, and the Claude-Mem installer episode shows why reading an install path matters. The next signal is the issue count: 65 are open today.