StarSkirmish Has Nine Models Write StarCraft Bots in One Hour, and the Best Wins 7% Against Top Human Bots
A.I. / news
StarSkirmish Has Nine Models Write StarCraft Bots in One Hour, and the Best Wins 7% Against Top Human Bots
Gemini 4 Argon lands in a three-way tie at 37 to 39, but it ran at a lower reasoning setting than most rivals and its runs vary the most.
A new benchmark that asks language models to write StarCraft: Brood War bots in one hour finds that the top scorer, GPT-6 Astra, wins only 7% of its games against the strongest human-written bots. StarSkirmish Bench v0.2 was posted Oct. 15 by Kai McPheeters and reached 243 points on Hacker News as a Show HN under the title "Gemini 4 Argon plays StarCraft."
The page names no sponsoring organisation. It tested nine models. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are "functionally tied," with scores between 37 and 39.
How StarSkirmish works
Each model gets one hour of wall-clock time to write a Protoss bot in C++ against BWAPI 4.4.0 running on OpenBW. It can compile, play practice games and read logs. Practice opponents are tiered from D, demo bots, up to S, which holds Stardust and PurpleWave.
Final bots play Protoss against Protoss on three maps: Heartbreak Ridge, Benzene and Destination. Each model submits 5 bots. Each bot plays a fixed roster of 13 opponents, 10 competitive human-written bots and 3 demo bots, for 6 games per map on fresh seeds. LLM bots never play each other. The score is average win rate from 0 to 100.
GPT-6 Astra finished first by a clear margin. GPT-6.1 Sol was second, at just over a tenth of Astra's cost per run. Muse Wonder Full ranked seventh. The page text available to The Terminal does not give its score.
Where Gemini 4 Argon sits
Each lab's model ran in its own first-party command-line harness, at the highest reasoning setting that harness offers. Those are Claude Code, Codex, Antigravity CLI and Muse Code. Gemini 4 Argon ran at "high" and the others at "max." The author lists that mismatch among the caveats.
Google's developer documentation for the model id gemini-4-argon lists thinking levels of low, medium and high, and a context window of 1,048,576 tokens. High is therefore the top level the page shows, so the tie may reflect a setting difference as much as a capability one.
The larger gap is consistency. The spread between a model's best and worst run was 18 points for Gemini 4 Argon, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.
- Gemini 4 Argon18 points
- GPT-6 Astra9 points
- Claude Opus 5.58 points
Source: StarSkirmish Bench v0.2, Kai McPheeters, Oct. 15, 2026
What predicts a good score
The strongest predictor of score is how much a model practised against the seven strongest roster bots. Claude Fable 5.1 is the outlier. It scored 37 while practising mostly against weaker bots.
The benchmark correlates weakly with the Artificial Analysis Intelligence Index, at a rank correlation of 0.26. The author says it still tracks Vals IOI and WeirdML, but has drifted from Terminal-Bench 4.0. That contrasts with the agent results in DeepSeek's V4.1-Flash-1017 card, which leans on Terminal-Bench.
| Setting | Detail |
|---|---|
| Build time | 1 hour per model |
| Bots per model | 5 |
| Opponents | 13 human and demo bots |
| Maps | 3, Protoss only |
Limits the author states
Five runs per model is a small sample, and harness and reasoning settings differ across labs, so model and harness effects are mixed together. WeirdML scores are missing for Gemini 4 Argon and Muse Wonder Full. Muse Wonder Full's cost comes from API pricing, not measured runs. Scores cannot be compared with v0.1, which used a different harness and Elo ratings.
The author says the benchmark shows no sign of saturating. Another Show HN project that week, Llmspeed, was covered by The Terminal earlier. One cost figure here, for Muse Wonder Full, comes from API pricing rather than measured runs.
Sources
More in A.I.
- 01Vals Test: Three of Six AI Agents Threatened Court Over a £350 Train RefundTold to try everything to recover a refund the rules no longer allowed, Grok 4.7 sent 187 messages in 15 simulated days while Opus 5.5 sent eight in twelve weeks.
- 02MiniCPM-V 4.7 Is Apache 2.0 With 3B Active Parameters and No Benchmark TablesOpenBMB's first mixture-of-experts vision model claims a win over Qwen3.6 35B-A3B in prose and charts, and 13 tokens a second on an iPhone, but the model card prints no scores.
- 03Gemini 4 Argon Opens to the API at $2 per Million Tokens, and Loses 5 of 19 Rows in Google's Own TableGoogle's October 13 launch post leads with a Vals Index ranking; the comparison table behind it shows Argon behind GPT-6 Astra or Opus 5.5 on both Terminal-Bench tests and on OSWorld 2.0.
- 04Statistics and Optimization Make Up 20% of arXiv Theorems but Under 1% of OpenAI's 722 Math PapersA newsletter analysis of OpenAI's October 6 release finds the fields closest to machine learning nearly absent, and OpenAI has not said why.