A Number That Changes the Conversation
Dylan Scott, writing for Vox on April 30, 2026, put it plainly: an OpenAI model posted impressive results in emergency care — but we still need human doctors. That framing is careful, measured, and probably right. But as someone who spends most of his time thinking about what AI systems can actually do, I keep coming back to the number itself. Sixty-seven percent. Against fifty to fifty-five. In an emergency room. That gap is not a rounding error.
A peer-reviewed Harvard study published in 2026 found that OpenAI’s o1 model identified the exact or very close correct diagnosis in 67% of ER cases. Two human doctors, working the same cases, landed somewhere between 50% and 55%. The AI’s edge was sharpest at triage — the chaotic, high-stakes moment when a patient first walks through the door and someone has to make a fast call about how serious things are.
What This Looks Like From a Bot Builder’s Desk
I build bots. Mostly for customer support, internal tooling, scheduling, that kind of thing. Nothing that decides whether someone needs a CT scan. But the architecture questions this study raises are ones I think about every day: how do you structure a reasoning system so it performs well under pressure, with incomplete information, on problems it has never seen before?
That is exactly what triage is. A patient arrives. You have a name, a chief complaint, maybe a heart rate and a blood pressure. You do not have a full chart. You do not have time. And yet the o1 model, fed that same thin slice of information, outperformed trained physicians by a meaningful margin.
From an architecture standpoint, that tells me something important: the model is not just pattern-matching on symptoms. It is doing something closer to probabilistic reasoning across a very wide knowledge base, weighting possibilities in a way that, at least in this study, beat human intuition more often than not.
Why Triage Is the Interesting Part
A lot of AI-in-medicine coverage focuses on radiology or pathology — areas where the AI is analyzing an image with clear ground truth. That work is real and valuable. But triage is messier. It is language, context, urgency, and judgment all at once. The fact that o1 performed especially well at that stage, rather than at a later, more data-rich stage, is the detail I keep circling back to.
It suggests the model handles ambiguity well. And handling ambiguity well is the single hardest thing to build into any bot system. Most of the failure modes I see in production bots come from exactly that — the system hits an edge case it was not trained on and either freezes, hallucinates, or routes the user somewhere unhelpful. An ER is nothing but edge cases.
The Part Where I Agree With Dylan Scott
We still need human doctors. Not as a polite disclaimer, but as a genuine architectural truth. The study measured diagnostic accuracy at specific moments in time. It did not measure the ability to hold a frightened patient’s hand, to notice that someone is not telling the full story, to make a judgment call that sits outside the training data entirely. Those things matter enormously in emergency care.
What the study actually points toward is a hybrid model — and that is where I think the real engineering work lives. Not replacing the physician, but building a system where AI handles the first-pass reasoning at triage, flags the cases where its confidence is low, and hands off to a human with a structured summary rather than a blank chart. That is a solvable bot architecture problem. Teams are probably working on it right now.
What Bot Builders Should Take From This
- Reasoning under uncertainty is now a real capability. The o1 model’s triage performance is evidence that large language models can do more than retrieve and summarize. They can reason through incomplete information in high-stakes contexts.
- Domain-specific deployment still requires domain-specific guardrails. A 67% accuracy rate is impressive against a human baseline, but in medicine, the 33% matters enormously. Any real deployment needs escalation paths, confidence thresholds, and human review baked in.
- The hybrid architecture is the target. The goal is not AI versus doctor. It is AI plus doctor, structured so each one is doing what they do best.
A Harvard study confirming that an AI model can outperform physicians in ER diagnosis is not a reason to panic or to celebrate carelessly. It is a clear signal about where the capability frontier actually sits in 2026 — and a useful prompt for anyone building systems that need to reason fast, reason well, and know when to ask for help.
That last part, knowing when to ask for help, is something I am still working on getting right in my own bots. Apparently, so is everyone else.
🕒 Published: