Contrastive-LM Releases CLM-8B, an Apache 2.0 Verifier Claiming 87.6% on Terminal-Bench

Contrastive-LM posted the weights for CLM-8B on Hugging Face under the Apache 2.0 licence, and its repository claims the best published results as a verifier on two agent benchmarks: 81.6 percent on DeepSWE and 87.6 percent on Terminal-Bench 2.1. Both figures are the developers' own, and the model card says they come from fine-tuned heads, not the checkpoint on offer.

The paper is credited to seven authors, including Jacky Kwok, Marco Pavone, Christopher Ré and Azalia Mirhoseini, according to the card's citation block. The card lists no release date. It says a multimodal CLM-35B is "coming in early October."

What a contrastive decision model does

CLM stands for Contrastive Language Model. The card calls it a System One model, a class that scores options instead of writing text. Cloudflare put out a rival in the same class on Oct. 1, Clef, and both are open-weight alternatives to the closed Jev, which TypeSafe AI released on Sept. 15 with weights available only to early-access users.

CLM-8B is two small projection heads, one for states and one for actions, on top of a frozen Qwen3-8B encoder. They are trained with a bidirectional InfoNCE loss, a contrastive objective that pulls a state toward the action that fits it. The card gives the training mix as roughly 60 million Nemotron question-and-answer pairs, 30 million synthetic hard negatives and 1 million agentic trajectories.

Because states and actions are encoded separately, the action embeddings can be cached. With about 1,000 candidates, the card says CLM is 13 times faster than Jev.

1. Verifier tasks, low end4.1 x
2. Verifier tasks, high end5.7 x
3. Zero-shot, up to9 x
4. 1,000 candidates, cached13 x

Source: CLM-v0.1-8B model card and Contrastive-LM GitHub repository, accessed 2026-10-03

The 87.6 percent figure and its test set

The Contrastive-LM repository describes the verifier setup. For each task the team sampled several candidate solutions, with Opus 5 producing them for DeepSWE and Fable 5 for Terminal-Bench 2.1. CLM or Jev then picks the best one.

Benchmark	Reported score	Held-out tasks	Candidate generator
DeepSWE	81.6%	38 (31 solved)	Opus 5
Terminal-Bench 2.1	87.6%	30	Fable 5

The DeepSWE number is 31 of 38, so one more or fewer solved task moves it by 2.6 points. The repository does not say how 87.6 percent is derived from 30 tasks, since no whole number of solved tasks out of 30 produces it. Nothing in the card or repository reports a run by a party outside the project.

The repository also says Jev "fails to serve as a verifier for these long-horizon tasks, scoring below pass@1," meaning worse than taking the first sample. That is a competitor's model measured by its rival.

Limits the card lists itself

The card is direct about constraints. The heads require Qwen3-8B last-token embeddings, so the encoder is fixed. CLM generates nothing, and its probabilities are relative to the candidates supplied. States longer than 2,048 tokens are truncated, which is short for an agent trace.

Zero-shot, the repository says CLM-8B is on par with Jev on four tests: T-Rex, BFCL v4, WikiRacing and Super Mario.

The repository had 2,700 stars, 238 forks and 12 open issues on Oct. 3. Stars measure attention, not use. For how vendor-run scoreboards have held up before, see the Gemini 4 Argon benchmark report, and for the base-model family, the Qwen3.8-27B release. The next date to watch is the CLM-35B release the developers put in early October.