Stanford Study Finds Local AI Loses on Watts, Not Coverage

A study from Stanford University and Together AI, posted to arXiv and last revised Sept. 6, 2026, finds that small language models running on local hardware can now correctly answer 88.7 percent of a set of 1 million real-world single-turn chat and reasoning queries, drawn from actual usage rather than a benchmark suite. The paper, "Intelligence per Watt: Measuring Intelligence Efficiency of Local AI", tested more than 20 local models of 20 billion active parameters or fewer against eight accelerators, and its list of authors runs from Stanford professor emeritus John Hennessy, who shared the 2017 Turing Award for his work on RISC computer architecture, to Christopher Ré, the Stanford computer science professor who separately co-founded Together AI, the inference company that is the paper's other listed affiliation.

The number that matters is not coverage, it's the 1.4x gap

The 88.7 percent figure is the one built for headlines, but it answers the wrong question. The paper's actual metric, intelligence per watt, or IPW, divides task accuracy by power draw, and on that measure the study found that cloud accelerators deliver 1.40 times higher IPW, and 1.6 to 2.3 times higher intelligence per joule, than an Apple Mac Studio with an M4 Max chip running the identical model. The paper attributes the gap to unified-memory architectures in local chips, which have to balance mixed workloads under thermal and power limits that a purpose-built data-center part does not face, against enterprise accelerators with dedicated tensor units and memory systems like HBM3e built for one job. Running a model locally answers more queries without a network round trip; it does not mean the chip answering them is the more efficient one. That is the denominator the coverage number leaves out, and it is the distinction a headline like "AI now runs on your laptop" tends to erase.

Accelerator	Architecture	Class
Apple M4 Max (Mac Studio)	Apple Silicon	Local
Nvidia A100 40GB SXM4	Ampere	Cloud
Nvidia H200 SXM	Hopper	Cloud
Nvidia GH200 Grace Hopper Superchip	Hopper	Cloud
Nvidia B200	Blackwell	Cloud
Nvidia Quadro RTX 6000	Turing	Cloud
Nvidia RTX 6000 Ada	Ada Lovelace	Cloud
AMD Instinct MI300X	CDNA 3	Cloud

Model progress did 3.1x of the work, hardware did 1.7x

The paper's headline efficiency claim is that IPW improved 5.3 times between 2023 and 2025, and it splits that gain into two multipliers rather than leaving it as one number: model progress contributed a 3.1x improvement in accuracy per watt, and accelerator improvements contributed another 1.7x, which compound to roughly the 5.3x total. The same two years pushed locally-serviceable query coverage, the share of the 1-million-query set a local model could handle without cloud help, from 23.2 percent to 71.3 percent. Stanford PhD researcher Avanika Narayan, one of the paper's authors, put the practical version of that trend in blunter terms in a Foundation Capital interview: "GPT-4 quality is now accessible in an 8-billion-parameter model." Co-author Jon Saad-Falcon framed the motivation the same way: "inference demand is going parabolic, and it's becoming an issue to satiate that demand," he said in the same interview.

1. 202323.2 %
2. 202571.3 %

Source: Intelligence per Watt, arXiv:2511.07885, accessed 2026-09-16

Hybrid routing splits the difference

The paper's own answer to the local-versus-cloud efficiency gap is not to pick a side. It reports that hybrid local-cloud routing, sending a query to a local model first and escalating to the cloud only when needed, cuts energy, compute and cost by 60 percent to 80 percent compared with a cloud-only baseline that batches every request. That range, not the 88.7 percent coverage figure, is the number a data-center operator or a device maker should actually be budgeting against, since it already nets out the 1.4x per-watt disadvantage local chips carry. The paper does not give a single point estimate inside that 60-to-80 range, and it does not say how sensitive the figure is to query mix, so the next test of the claim is whether a shipping product, rather than a research routing simulation, lands near the low end or the high end. Neither Stanford nor Together AI has named a commercial deployment of the routing scheme the paper describes.

The finding lands on both sides of the local-hardware argument at once, which is why it is worth reading past the headline number. It hands local inference projects like Colibri, which runs a 744-billion-parameter model on 16 GB of consumer memory, a real efficiency case, while also undercutting the marketing instinct to treat every local-inference chip announcement as a power-efficiency win on its own. Intel ran into the same asymmetry from the vendor side on Sept. 10, 2026, when it built a GPU explicitly around a tokens-per-watt pitch and then declined to publish a tokens-per-watt number for it. The Stanford paper is, in effect, the number Intel did not supply, measured independently and applied to a chip Intel did not make.