Edge0 Streams a 35-Billion-Parameter Model Into 3GB of RAM

Edge0 released an inference framework on Sept. 10, 2026 that runs a 35-billion-parameter mixture-of-experts model inside under 3 gigabytes of active memory, by streaming expert weights from SSD storage instead of loading the full model into RAM.

The open-source project published its runtime and two preview model checkpoints to GitHub and Hugging Face under an Apache 2.0 license, according to the project's own documentation.

How the memory trick actually works

Edge0 keeps a 4-bit quantized copy of the full model on disk and loads only the experts a mixture-of-experts layer selects for the current token, rather than the entire 256-expert set a model like this normally holds in memory at once. A trained "prerouter" head predicts which experts the next step will need one layer ahead of time, so loading overlaps computation instead of stalling it, a change Edge0 says adds up to 59 percent more decode throughput. A separate adapter, called Recover-LoRA and trained by distillation from a full-precision teacher model, claws back some of the accuracy that 4-bit quantization would otherwise cost.

What it actually measured

On a Mac mini M4 Pro, the 35-billion-parameter model, which Edge0 calls edge0-35b, held peak memory at 2.9 gigabytes while decoding at 14.9 to 17.7 tokens per second, with prompts processing at 113 to 140 tokens per second, according to Edge0's published benchmarks. A smaller companion model, edge0-8b, ran in 1.0 gigabyte of memory at 23.9 to 25.3 tokens per second. Against the original full-precision models, Edge0 measured an average benchmark loss of 3.9 points for the 35-billion-parameter version and 2.8 points for the 8-billion-parameter version, across AIME, HumanEval, GPQA-Diamond, MMLU-Pro and IFBench.

1. edge0-8b1 GB
2. edge0-35b2.9 GB

Source: Edge0 GitHub repository, accessed 2026-09-12

What is still missing from this release

The preview is not tuned for agentic tasks: tool use, multi-step planning and long-horizon autonomy "are currently weak," Edge0's own model card says. The MLX backend that makes the memory trick work runs only on Apple silicon; a CUDA backend is planned but not available, so the technique does not yet reach Windows or Linux machines with Nvidia GPUs. Long conversations also add a key-value cache that can push memory past the 3-gigabyte target, according to the model card.

Edge0-35B is a fine-tuned version of Alibaba's Qwen3.5-MoE 35B-A3B, according to Edge0's own documentation, a different release from the 27-billion-parameter Qwen3.8 model that topped Hugging Face's download charts this month. It joins a run of on-device releases that includes OpenBMB's 2.5-billion-parameter MiniCPM5, though Edge0 trades a smaller model for a larger one squeezed to fit in less memory than its size would suggest.

Edge0 has not said when a CUDA backend will ship, which is the change that would let the same SSD-streaming trick reach Windows and Linux machines with Nvidia GPUs rather than only Apple silicon.