Surge AI's Dayjob: Tax Gives Claude Sonnet 5.5 a 10.8% Pass Rate on 50 Assignments

Surge AI's new Dayjob: Tax benchmark gives the best published agent a 10.8 percent pass rate on 50 professional tax assignments. A separate Vals AI tax benchmark listed a top score of 80.38 percent as of Sept. 1.

Surge published Dayjob: Tax on Oct. 27. An agent receives a client request and a folder of records, then has to produce a tax return, workpaper, technical review or client advice. The post does not name an author.

The three scores in the post

The post gives pass rates for three models, all Anthropic's. It also names a model called Astra in two case studies but gives it no score in the text, and the full leaderboard sits on a separate page.

Model	Pass rate, pass@1
Claude Sonnet 5.5	10.8%
Claude Fable 5.1	9.6%
Claude Opus 5.5	9.2%

1. Claude Sonnet 5.510.8 %
2. Claude Fable 5.19.6 %
3. Claude Opus 5.59.2 %

Source: Surge AI, Dayjob: Tax, accessed 2026-10-28

A pass requires an attempt to meet at least 95 percent of a task's rubric criteria, which cover calculations, supporting evidence, consequential issues and the finished deliverable. That threshold explains much of the low number. Surge's conclusion is its own: "Agents haven’t yet passed the audit."

What the 50 assignments contain

The assignments span eight practice areas, from individual and family tax to partnerships, estates, international tax, corporate transactions and tax disputes. Together they hold 1,789 workspace files, a median of 30 per assignment and up to 123. Of those, 1,487 are PDFs, 117 are Excel workbooks and 38 are Word documents.

Surge says each task went through three layers of expert review for correctness, completeness and solvability. It describes the reviewers by background, including a tax attorney with more than 10 years at large firms and a Baker Tilly tax director, but names none of them.

Two case studies, five attempts each

Surge's post says the benchmark tests whether an agent finds the problem, not only whether it solves a problem already defined. In a condo-sale case, Opus identified a residence-requirement deadline in all five attempts. Astra never did.

In a separating-couple case, Opus based its recommendations on the assumption that the couple was unmarried across all five attempts. Astra flagged the need to verify marital status in four of five. Surge calls both cases illustrative, not representative of the full set.

Why 10.8 and 80.38 are not the same measurement

Vals AI's TaxEval page lists Muse Spark 1.2 at 80.38 percent, Muse Spark 1.1 at 79.72 percent and Claude Sonnet 4.6 at 77.11 percent, as of Sept. 1. TaxEval has more than 1,500 questions, 1,223 of them in the test set, and an LLM judge, Claude Sonnet 4.5 in non-thinking mode, scores answer correctness against ground truth. Dayjob grades a finished deliverable against a 95 percent rubric.

Vals also runs a Tax Agent Bench, described on that page only as agentic US corporate tax research. Its model page for DeepSeek V4.1-Flash-1017 lists 44.35 percent there, seventh of 78. Neither Vals figure is comparable with Surge's.

Benchmark designs can also diverge from vendor claims: in our report on Gemini 4 Argon, Vals ranked the model third where Google's card said first. Anthropic's account of Claude cutting document review time at a drug developer, covered earlier, is a customer outcome, not a benchmark.

The full Dayjob benchmark is not public. Surge asks interested parties to write to benchmarks@surgehq.ai.