Microsoft's MAI-DxO reaches Science: 83% vs 36% on complex cases, but the comparison was not specialists

Pin the source first. The primary material is a Microsoft AI post titled Building Medical Superintelligence, bylined MAI Health Team, published October 22, 2026. The paper appeared in Science the same day as Sequential diagnosis with language models, DOI 10.1126/science.aeh7930. That week Microsoft Research also published The Building Blocks of Medical Superintelligence in BMJ Digital Health and AI. TechCrunch and The Verge reported the same day, and TechCrunch added cost and ablation details the official post did not include; those are attributed separately below.

What exactly is being measured. The official critique of existing benchmarks is blunt: exams such as the US Medical Licensing Exams are built to assess knowledge, not the reality of making a diagnosis. All information is presented upfront and a single answer is selected. Real clinical work is the opposite: clinicians start from limited information and decide what to ask, which test to order and how each new finding changes the likely diagnosis. Hence SDBench, using two case sets, 304 complex cases taken from New England Journal of Medicine case proceedings and a set of more typical emergency department presentations from MIMIC-IV-ED. Each becomes an interactive case where an AI model or a physician starts with limited information and chooses what to request next. The benchmark records diagnostic accuracy, evidence requested and estimated test costs, assessing the answer and the path taken.

How MAI-DxO works. Rather than one model answering immediately, it behaves like a panel of virtual physicians: each proposes diagnoses, selects useful questions or tests and challenges the emerging view, with the system coordinating that exchange over several rounds of debate, updating its assessment as evidence arrives and weighing each test's likely value against its cost. The intuition is straightforward: the value of a human consultation comes from disagreement being surfaced, not from one participant being smarter. In the paper MAI-DxO is the leading AI system on SDBench, and the sequential decision framing itself is one of the paper's contributions.

Four results as published. First, on complex NEJM cases MAI-DxO reaches 83 percent against 36 percent for physicians with external reference resources and 22 percent without, roughly 2.3 and 3.7 times. Second, on MIMIC-IV-ED emergency cases, 80 percent versus 47 percent, more than 1.7 times. Third, a cost-optimised configuration hits 81 percent on NEJM cases at less than half the cost of comparative systems. Fourth, the post states that across both datasets no other human or AI approach was both more accurate and less costly than MAI-DxO. It also stresses that repeating the evaluation with newer models including GPT-5.5 preserved and improved the gains without changing the orchestration method, meaning the benefit is not tied to one model generation.

The counterintuitive part, from TechCrunch's reporting. Much of the improvement comes from the underlying models getting stronger, with Microsoft's orchestration adding gains on top. The numbers: GPT-5 alone with a short basic prompt scored 78.9 percent on complex cases at roughly $8,400 per case; adding DxO cut cost by about $3,000 but raised accuracy by only around four points. With GPT-5.5 the results split: complex cases rose from 72 percent to nearly 87 percent at slightly lower cost, while on emergency department cases DxO delivered little improvement, roughly 85 percent with or without it at similar cost. The boost is largest for weaker models, Gemini 2.5 Flash going from 46.4 percent alone to nearly 69 with DxO. Microsoft's researchers say accuracy gains naturally shrink as models approach the theoretical maximum, a sentence that deflates the idea that an orchestration trick is what pushes AI past doctors.

Who the comparison group was, and this belongs up front. TechCrunch states plainly that cases this complex are usually handled by specialists or physician teams, and those specialists were not part of the comparison; the participants were primary care doctors and general internists working with reference materials and the internet. Microsoft is not concealing this, the official post says physicians as well, but the qualifier fixes the boundary of the conclusion. It supports the claim that in settings without specialist support this system's sequential diagnostic reasoning is substantially stronger. It does not support the claim that it beats specialists, and certainly not that it replaces physicians.

Open sourcing and real-world validation. Microsoft says it is open-sourcing the SDBench portion for select NEJM cases, including the benchmark, Gatekeeper, Judge and cost-estimation code, so researchers can reproduce the evaluation, test other models and challenge or extend the findings. On the real-world side, Microsoft has partnered with Beth Israel Deaconess Medical Center in Boston for a prospective study: patients receive standard care while consenting patients also engage with the system, and its recommendations are compared against physicians' independent assessments. Dominic King, vice president of Health at Microsoft AI, calls it the biggest study of its kind, spanning emergency and urgent care settings.

The bigger Microsoft play. The official post places diagnosis at the start of the care journey and then says diagnosis is only one part: better care also means keeping pace with medical knowledge, choosing the right treatment, understanding change over time and supporting decisions that reflect a person's values and access to care. The ambition is spelled out in The Building Blocks of Medical Superintelligence in BMJ Digital Health and AI, defined as surpassing the strongest contemporary human-led health systems in medical reasoning and coordination across the full arc of care. Two concrete paths are public: a collaboration with Mayo Clinic to train a frontier foundation model for healthcare, planned for release next year, and consolidating health teams from across Microsoft, including Microsoft Research scientists and the teams behind Dragon Copilot and PowerScribe, into Microsoft AI. Suleyman told TechCrunch he believes 2027 will bring a coding-style breakthrough for medical records, streaming predictions of how a record might unfold.

Three boundaries, stated plainly. First, this is simulation: NEJM cases are published, fully written-up complex case texts whose information density and noise profile differ sharply from a real ward, and MIMIC-IV-ED is retrospective data, so 83 percent cannot be read as eight in ten real outpatients being diagnosed correctly. Second, watch the cost definitions: the official text refers to estimated test costs, the cost of investigations, while TechCrunch reports per-case running cost, and those two are not the same thing; quoting them interchangeably produces wrong conclusions. Third, the absence of specialists from the comparison means any claim of beating doctors needs a qualifier attached. This article does not forecast when such a system reaches clinical use, nor judge whether it should.

One line worth keeping separate. What transfers from this paper is not the score but the act of reframing the task: replacing give one answer with decide what information to acquire next, and putting the cost of acquiring it into the objective. The same reframing applies to debugging, legal due diligence, investment research and procurement comparison, anywhere information costs money and choosing which piece to buy is your call. Healthcare is simply the first battlefield where Science verified it.