RuneBench Has AI Agents Grind RuneScape Skills, and Its Leaderboard Puts GPT-6 Astra on Top at 7.3%

RuneBench, an open-source benchmark that has AI coding agents play a RuneScape-like game by writing TypeScript, reached 243 points on Hacker News. The leaderboard that circulates with it lists GPT-6 Astra at its highest reasoning setting first, at 7.3 percent, according to the aggregator BenchLM.

The benchmark's own paper page carries no per-model numbers, so every score below comes from BenchLM, not from the authors.

What the agents actually do

The paper's BibTeX credits Max Bittker, Surya Narreddi, Rob Haisfield and Sean Lee. It does not list titles or affiliations. Agents play by "writing and executing TypeScript snippets against an emulated game server," the page says, and each gets a folder of markdown files from the game wiki covering skill guides, item stats, NPC locations and quest walkthroughs.

The GitHub repository is MIT-licensed, with 156 stars, 11 forks and 2 open issues at the time of reading. It runs on the Harbor agent-benchmark framework and an emulator built on the LostCity server and client, with the game running at 8x speed. The README says nothing about how that emulator relates to Jagex's official game or its terms of service.

How the scoring changed, and why the pages disagree

The authors first scored total experience points (XP) gained in a skill over a fixed window. They found that "punished exploration," because winning runs were simple grinds. They switched to the best XP rate in any 15-second window, which they say rewards higher-level strategy.

The three pages describe the task set differently:

Source	Tasks	Run length	Scoring described
GitHub README	16 skill tasks, plus 8 gold tasks	15 and 30 minutes	Peak XP/min in a 15-second window
BenchLM	16 skill tasks	30 minutes	Mean of ln(1 + XP/min)
Paper page	Skill XP tasks	Not stated	Max XP rate per 15-second window

BenchLM's leaderboard shows percentages even though the underlying metric is a log XP rate, and its page does not explain the conversion.

What BenchLM's leaderboard shows

BenchLM lists 93 models. The top five are within half a point of one another, and the bottom includes the open-weights Qwen3.8 27B.

1. GPT-6 Astra (high)7.3 %
2. GPT-6 Astra6.9 %
3. GPT-6.1 Sol (high)6.9 %
4. Gemini 4 Argon6.9 %
5. GPT-6.1 Sol6.8 %
6. Qwen3.8 27B2.8 %

Source: BenchLM RuneScape-Bench page, accessed 2026-10-21; scores not independently reproduced

BenchLM adds three caveats. It shows RuneScape-Bench for reference only and excludes it from its weighted score. Rows reflect the agent harness and gameplay strategy as well as the underlying model. And several models appear several times, once per reasoning-effort variant. The page gives no evaluation dates.

The authors say their own limits are "the long runtime" of each task, plus environment complexity and low sample sizes, which add noise and false negatives. Shorter tasks hurt results because they penalized heavy up-front planning and slow inference.

What the authors want to test next

The paper lists two follow-ups. One asks whether two bots can outperform one, either through one model scripting several characters or separate models talking in-game. The other asks whether one agent's written guide can improve another agent's performance.

RuneBench is not the only game-based test. In StarSkirmish, nine models wrote StarCraft bots in one hour and the best won 7 percent against top human bots. A separate Mercor analysis argued that 90 percent of US work hours have no real AI benchmark.

Neither the paper page nor the repository gives a date for the next results update.