A $998 LLM Training Run Beats Karpathy's $2,500 One

Independent developer Hugo Vergnes published a writeup on Sept. 4 describing a 3.848-billion-parameter language model, which he calls little-lm, trained from scratch for $998 on eight rented Nvidia B200 GPUs in 43 hours. It scores 0.384 on CORE, a benchmark that averages accuracy across 22 tasks against a random-guess baseline. The claim worth checking is not the score itself but what it is being compared to.

What $998 bought, and what didn't work first

The final model is a 28-layer decoder with a 3,072-dimension hidden size, grouped-query attention split into 24 query heads and 8 key-value heads, and relu-squared MLPs, using the same GPT-2 tokenizer and a 50,304-token vocabulary. Vergnes trained it on 65.3 billion tokens of Nvidia's Nemotron-ClimbMix dataset at a 2,048-token context length, using the Muon optimizer for matrix parameters and AdamW for embeddings, on a trapezoidal learning-rate schedule with a 5 percent warmup and a cooldown over the final half of training. He reported throughput of roughly 437,000 tokens per second. For comparison, OpenAI's GPT-2 at 1.5 billion parameters scores 0.2565 on the same metric, according to Vergnes' own numbers.

That architecture is not what Vergnes started with. He wrote that an earlier 858-million-parameter version, trained on a single Nvidia A100 over 5.8 days, scored only 0.258 CORE, a result he called a failure that forced five changes to the architecture and training recipe before the version that shipped. Debugging happened on a single consumer RTX 5090; only the final run moved to rented B200 capacity.

The numbers don't match the primary source

Vergnes benchmarks little-lm against Andrej Karpathy's nanochat, a from-scratch ChatGPT-clone pipeline Karpathy has run at price tiers including $100, $1,000 and $2,500 since 2025. Vergnes describes nanochat's $1,000-tier run, called d32, as roughly 1 billion parameters scoring 0.310 CORE.

That doesn't match Karpathy's own account. His GitHub discussion of the d32 run, posted Oct. 13, 2025, puts the model at 1,879,048,192 parameters, trained on eight H100 GPUs, scoring 0.3168 CORE on the base model. That is nearly double the parameter count Vergnes cited, and a score he rounded down by roughly four hundredths. Neither number is wrong in a way that changes Vergnes' conclusion, but the mismatch means his comparison rests on a paraphrase of the primary source rather than the source's own figures.

The comparison that does hold, and what explains it

The closer match is a different nanochat tier. Karpathy's d34 run, which cost about $2,500 on eight H100s over roughly 100 hours and finished training Nov. 20, 2025, scored 0.3382 CORE at 2.2 billion parameters. That is within a hundredth of the 0.338 CORE Vergnes reported for little-lm at a matched 1,024-token context, at less than half the price and roughly 40 percent of the wall-clock time.

The ten months between d34 and little-lm cover a full GPU generation, and that gap is a plausible explanation on its own. Nvidia's B200 carries 8 terabytes per second of memory bandwidth against the H100's 3.35 terabytes per second, a roughly 2.4-times advantage, and memory bandwidth is typically what caps how fast a small, memory-bound model moves through a fixed token budget. Judged on wall-clock hours per dollar, Vergnes' 43-hour run against Karpathy's 100-hour d34 looks like the return a hardware jump of that size would predict, the same logic behind why a chipmaker's own tokens-per-watt claims matter more than its raw throughput number, rather than evidence that training small models has gotten cheaper in some more general sense.

A second, less flattering number sits inside Vergnes' own numbers. His 65.3 billion training tokens against 3.848 billion parameters works out to a token-to-parameter ratio of about 17 to 1. Karpathy's discussion of the d34 run says he deliberately overrode nanochat's default ratio of 20 to 1, pushing it to 40 to 1 because, in his words, "the param:token ratio is overridden from 20 up to 40," meaning that model trained on roughly twice the tokens per parameter that Vergnes used and more than double what he actually ran. That little-lm matched d34's score anyway, at a lower token budget per parameter than either nanochat's default or its overridden setting, is a stronger claim about efficiency than the raw dollar figure, and one Vergnes' post does not make explicitly.

Dataset choice complicates that reading further. Vergnes trained on Nvidia's Nemotron-ClimbMix, itself built from a dataset, climbmix-400b-shuffle, that Karpathy released separately; nanochat's own runs use a different FineWeb-derived mix. A model can score higher on CORE because its pretraining data overlaps more with CORE's 22 constituent tasks, independent of anything the hardware did, and neither writeup controls for that variable.

What would change this read

Vergnes wrote that the project was "debugged on a 5090 and finished on rented B200s" but did not name the cloud provider or the per-GPU-hour rate he paid. That means the $998 figure cannot be checked against a public price list the way Karpathy's on-demand H100 numbers can. If Vergnes' rate reflected a reserved-capacity discount or a promotional credit rather than list price, the headline comparison to nanochat's runs weakens considerably, even though the CORE scores themselves would not change.

That gap between a self-reported training cost and a verifiable one is the same asymmetry OpenAI's own vendor-supplied benchmark scores faced this month when it shipped its Critical-designated model: a number that is real, sourced and still only as good as the discloser's account of how it was produced.