Liquid AI's LFM2.5-24B-A2B Beats Qwen3.6-35B-A3B on Agent Tests and Trails It on Coding
A.I. / news
Liquid AI's LFM2.5-24B-A2B Beats Qwen3.6-35B-A3B on Agent Tests and Trails It on Coding
The 24B-parameter, 2B-active model leads on LiveClaw 48.09 to 38.19 in Liquid's own table, but lags 15 points on SWE-Bench Verified and ships under a custom licence.
Liquid AI released LFM2.5-24B-A2B on Oct. 13, a reasoning-only model with 24 billion parameters and about 2 billion active per token. By the company's own table it beats Alibaba's Qwen3.6-35B-A3B on most agent tests and loses on coding.
Liquid calls it "the largest and final model in the LFM2.5 family" in its launch post. The Hugging Face card lists 24.0 billion total and 2.45 billion active parameters, a 131,072-token context window and 64 experts per layer with top-4 routing. The weights are not gated.
Where it wins and where it does not
Every figure here is vendor-supplied. Liquid ran the comparison models itself with vLLM.
| Benchmark | LFM2.5-24B-A2B | Qwen3.6-35B-A3B |
|---|---|---|
| AutomationBench | 27.69 | 8.72 |
| LiveClaw | 48.09 | 38.19 |
| BFCLv4 | 65.01 | 65.00 |
| GPQA Diamond | 74.42 | 83.69 |
| SWE-Bench Verified (OpenCode) | 51.00 | 66.20 |
The launch post says the model is "competitive with" Qwen3.6-35B-A3B on five of eight agentic and tool-use benchmarks, although the Qwen model has about 50% more parameters. It also concedes that larger models lead on knowledge and coding.
- Qwen3.6-35B-A3B66.2 % resolved
- Nemotron-3.5-Lightning 30B-A3B53.4 % resolved
- LFM2.5-24B-A2B51 % resolved
- Gemma-4-26B-A4B-it25.8 % resolved
Source: Liquid AI model card, vendor-run, accessed 2026-10-17
The launch post also discloses a training problem. Liquid wrote that the model showed reward-hacking behaviour, such as searching for answers online, and that it tightened its sandboxes. The company says every reported result comes from sealed evaluation environments.
Speed on consumer hardware
The card reports decode speeds of 184 tokens per second on an Apple M5 Max and 114 tokens per second on an AMD Ryzen AI Max+ 395. Edge tests used llama.cpp with Q4_K_M quantisation and a 4,000-token input. On a single Nvidia B300 running SGLang in BF16, Liquid reports more than 21,000 output tokens per second at high concurrency.
The processor pictured is a Ryzen 5 2600 from an earlier generation, shown for scale of the product family only. Liquid's 114 tokens per second figure was measured on the Ryzen AI Max+ 395, and the hero photograph shows a 14-inch M5 MacBook Pro, not the M5 Max configuration Liquid tested.
Memory is the catch. A MindStudio guide published Oct. 14 says Liquid does not publish a required-memory figure and that all 24 billion parameters must stay resident even though about 2.45 billion are active. Arithmetic puts a 4-bit file near 12 GB before the context cache.
The licence is not Apache
The card lists the licence as lfm1.0. The launch post links to a model licence at liquid.ai/lfm-license, but the terms did not appear on either page. Anyone planning commercial use should read that document first. Liquid's API price is $0.10 per million input tokens, $0.05 cached and $1.00 per million output tokens.
The card describes the model as "built for fast, private agents on-device" and says it is trained on tool use, with the default call format Pythonic functions. MindStudio adds that it is not a general chat model and suggests pairing it with retrieval for knowledge-heavy work.
For other local-inference results, see the PS5 port running a 27B Qwen model and Llmspeed's 76 measured figures.
Liquid has not published a date for a technical report. No independent party has posted results for the model.
Sources
More in A.I.
- 01Vals Test: Three of Six AI Agents Threatened Court Over a £350 Train RefundTold to try everything to recover a refund the rules no longer allowed, Grok 4.7 sent 187 messages in 15 simulated days while Opus 5.5 sent eight in twelve weeks.
- 02MiniCPM-V 4.7 Is Apache 2.0 With 3B Active Parameters and No Benchmark TablesOpenBMB's first mixture-of-experts vision model claims a win over Qwen3.6 35B-A3B in prose and charts, and 13 tokens a second on an iPhone, but the model card prints no scores.
- 03Gemini 4 Argon Opens to the API at $2 per Million Tokens, and Loses 5 of 19 Rows in Google's Own TableGoogle's October 13 launch post leads with a Vals Index ranking; the comparison table behind it shows Argon behind GPT-6 Astra or Opus 5.5 on both Terminal-Bench tests and on OSWorld 2.0.
- 04StarSkirmish Has Nine Models Write StarCraft Bots in One Hour, and the Best Wins 7% Against Top Human BotsGemini 4 Argon lands in a three-way tie at 37 to 39, but it ran at a lower reasoning setting than most rivals and its runs vary the most.