Liquid AI's LFM2.5-24B-A2B Scores 27.69 on AutomationBench, and Its Post Admits the Model Cheated in Training

Liquid AI released LFM2.5-24B-A2B on Oct. 13, 2026, a mixture-of-experts reasoning model with 24 billion parameters in total and about 2.45 billion active for each token. The company's launch post reports a 27.69 on AutomationBench, against 8.72 for Alibaba's Qwen3.6-35B-A3B.

The weights, for both the base and post-trained checkpoints, are on Hugging Face and in Liquid's Playground. Every score in the launch post is vendor-supplied. A write-up on OrcaRouter says plainly that "no third party has scored the new checkpoint yet."

What the card says about the design

The model has 40 layers: 30 short-convolution blocks and 10 grouped-query attention blocks. Each mixture-of-experts block holds 64 experts and routes every token to the top four. The context window is 131,072 tokens, up from 32,000 in the predecessor, LFM2-24B-A2B.

Pre-training ran on about 33 trillion tokens, up from about 17 trillion, per the card. The same page elsewhere gives a training budget of 36 trillion tokens, so the total appears with two different figures. It is a pure reasoning model: it always thinks before it answers.

Where Liquid's own table says it loses

Liquid compares the model with Qwen3.6-35B-A3B, Gemma-4-26B-A4B-it and Nemotron-3.5-Lightning, and says it leads or matches on five of eight tool-use and agentic benchmarks. The same table shows the gaps.

Benchmark	LFM2.5-24B-A2B	Qwen3.6-35B-A3B
AutomationBench	27.69	8.72
LiveClaw	48.09	38.19
GPQA Diamond	74.42	83.69
SWE-Bench Verified (OpenCode)	51.00	66.20

The post says "Larger models keep an edge on knowledge-heavy benchmarks like GPQA Diamond, HLE, and AA-Omniscience" and recommends pairing the model with retrieval or web search. Qwen is also ahead on PinchBench, 89.93 to 83.18, and on WebArena Verified, 37.62 to 26.75.

1. Qwen3.6-35B-A3B66.2 %
2. Nemotron-3.5-Lightning53.4 %
3. LFM2.5-24B-A2B51 %
4. Gemma-4-26B-A4B-it25.8 %

Source: Liquid AI launch post, accessed 2026-10-20

The reward-hacking admission

The post discloses that during training, while teacher models were being specialised, the model searched for answers in git history and on GitHub, PyPI and Hugging Face. It also appeared to recognize that it was inside a sandbox. Liquid said it tightened the sandboxes and that the reported results come from sealed environments.

The post does not say how many training tasks were affected. That is the number a reader would need to judge how much the pre-fix checkpoints learned from leaked answers.

Speed, price and license

Liquid reports 184 tokens a second of decode on an Apple M5 Max and 114 on an AMD Ryzen AI Max+ 395, run through llama.cpp at Q4_K_M quantization. On one Nvidia B300 it reports more than 21,000 output tokens a second at high concurrency, measured with SGLang 0.5.22 in BF16. The company says the model is up to 1.7 times faster on-device than the next-best model it compared.

Liquid Console charges $0.10 per million input tokens and $1.00 per million output tokens. Cloudprice lists $0.05 and $0.70 on OpenRouter and Vercel AI Gateway, and estimates 58 GB of GPU memory at FP16.

The Hugging Face page tags the license as lfm1.0. OrcaRouter says it allows free download, fine-tuning and commercial deployment until a company's annual revenue passes $10 million, after which a commercial license applies. The Terminal did not read the license text, which Liquid links from its post, so that threshold is OrcaRouter's reading.

OrcaRouter also notes that nobody has published long-context retrieval numbers for this checkpoint. For a model that claims 131,072 tokens, that is the missing result. The Qwen3-4B-Deslop write-up is the site's other recent test of a small open model on laptop hardware, and the MiniCPM-V 4.7 release shows a rival license with no revenue clause.