Edge0's 35B Model Runs in Under 3GB of Memory

Edge0-AI released Edge0-35B-A3B-preview, a 35 billion parameter mixture-of-experts model built on Alibaba's Qwen3.6-35B-A3B that the project says runs in under 3 gibibytes of active memory by streaming its experts from disk rather than holding all of them in RAM.

The model and its inference framework are described in Edge0-AI's GitHub repository and on its Hugging Face model card, both released under the Apache 2.0 license. Of the model's 256 total experts, only 4 are active for any given token, and the framework keeps the full 4-bit weight set on storage, loading only the experts a token actually needs.

How it fits in 3GB

Edge0 calls the disk-streaming technique SSD expert offload, which bounds peak memory to the active expert set rather than the full model. A second component, called prerouter, is a trained module that predicts which experts a token will need one step ahead of time, so they can be prefetched during computation; the GitHub README credits this with up to a 59% improvement in decode throughput over fetching experts on demand. A third piece, Recover-LoRA, freezes the 4-bit base weights and trains LoRA adapters by distillation from a full-precision teacher model to claw back some of the accuracy quantization costs.

What it costs in quality

MindStudio's write-up, published Sept. 15 and edited by Luis Chavez-Mattos, MindStudio's director of product, reports that 4-bit quantization lowers the model's average score across five OpenCompass benchmarks compared with the full-precision original:

1. fp16 original83.2 points
2. 4-bit + Recover-LoRA79.2 points

Source: MindStudio, Edge0-35B-A3B-preview write-up, accessed 2026-09-17

Speed on a Mac mini M4 Pro

MindStudio tested the model on a Mac mini M4 Pro and recorded decode speeds of 14.9 to 17.7 tokens a second, with prefill throughput of 113 tokens a second cold and 140 tokens a second warm. The GitHub README lists roughly 23 gigabytes of disk space needed for the 35B model's full weight set, against about 4.2 gigabytes for a smaller 8B sibling, Edge0-8B-A1B-preview, that ships alongside it.

What's not ready yet

Edge0-AI labels both models a preview and says in its own documentation that the release is weak on tool use, multi-step planning and other agentic tasks. The MLX backend that makes the memory trick work targets Apple Silicon only for now; a CUDA backend is on the project's roadmap with no timeline given, so the technique does not yet reach Windows, Linux or non-Apple hardware. The README also flags a known issue in which older MLX versions produce garbled output on Apple's A18 and A18 Pro chips.

A different project, Colibrì, streams a 744 billion parameter model from disk into 25 gigabytes of RAM using a similar disk-offload idea at a much larger scale; Edge0's approach targets a smaller model and adds the prerouter and distillation steps Colibrì does not use. For the broader question of whether local inference can match cloud accelerators at all, see Stanford's intelligence-per-watt study.