Microsoft's MAI-DxO Scores 83% on NEJM Cases in Science; Physicians Got 22% Without References

Microsoft AI published its MAI-DxO diagnostic system in the journal Science on Thursday, October 22, reporting 83% accuracy on complex New England Journal of Medicine cases against 22% for physicians working without reference materials. The company's announcement says physicians reached 36% when allowed textbooks and the internet.

Microsoft AI chief Mustafa Suleyman said real-world use still requires clinical testing, Gizmodo reported. Every number below is Microsoft's. Neither the Microsoft post nor Gizmodo cites an independent evaluation.

What the benchmark measures

The test is SDBench, the Sequential Diagnosis Benchmark. It holds 304 complex NEJM cases plus a set of emergency cases from the MIMIC-IV-ED dataset. A system starts from a chief complaint, chooses follow-up questions and tests, and is scored on accuracy, evidence requested and estimated test cost.

MAI-DxO works like a panel of virtual physicians. They propose diagnoses, pick tests, challenge each other over several rounds and weigh each test against its cost.

Dataset	MAI-DxO	Physicians
NEJM cases, no references	83%	22%
NEJM cases, with references	83%	36%
MIMIC-IV-ED emergency cases	80%	47%

1. MAI-DxO83 %
2. Physicians with references36 %
3. Physicians without references22 %

Source: Microsoft AI, Building Medical Superintelligence, 2026-10-22

Who the comparison group is

Gizmodo describes the human baseline as generalist physicians. Microsoft's post gives no physician sample size and no breakdown of years in practice. It also does not name the base model behind the main results. Gizmodo says the best results used OpenAI's GPT-5, and the post says results held and improved on GPT-5.5 without changing the orchestration method.

The post says a cost-optimised configuration reached 81% on NEJM cases at less than half the cost of comparison systems. Gizmodo notes Microsoft's claim of half or a third of the cost comes from estimates on simulated cases, not actual medical bills.

The researchers expect the gap to narrow against specialists, Gizmodo reported. The headline ratio, which Microsoft puts at more than twice as accurate with references and nearly four times without, is therefore a generalist ratio.

What has not been tested

The evidence is retrospective. Dr. Dominic King, Microsoft's AI health lead, said testing at Beth Israel Deaconess Medical Center in Boston is planned within the next couple of months, according to Gizmodo. No result exists from that study.

Gizmodo also cites a 2025 study in which doctors who relied on AI during colonoscopies performed worse once the tool was removed. The same piece recalls a Harvard and Beth Israel Deaconess study in which OpenAI's o1-preview outperformed physicians in emergency triage.

Public bodies are already setting limits on chatbots at work, as when Swedish government offices blocked ChatGPT, Claude and Copilot. The hospital question is harder because accountability for an AI-assisted wrong diagnosis is unresolved, Gizmodo said. Labs are also tightening their own handling of frontier systems, as in OpenAI's move to offline training.

What Microsoft released

Microsoft open-sourced SDBench for select NEJM cases, including the Gatekeeper, Judge and cost-estimation code. It is not the full benchmark. Microsoft is also consolidating health teams, including Microsoft Research and the Dragon Copilot and PowerScribe teams, under Microsoft AI.

A companion vision paper, "The Building Blocks of Medical Superintelligence," appears in BMJ Digital Health and AI, per the post. Microsoft has not given a date for the Beth Israel results.