2025-08-24T10:20:18Z

---

Imagine this: artificial intelligence systems are acing medical exams, but what if I told you that those impressive scores might hide a troubling truth? New research published in JAMA Network Open has uncovered a startling reality about large language models (LLMs) like GPT-4o and Claude 3.5 Sonnet. These AI tools often 'pass' standardized medical tests not by reasoning through complex clinical questions, but by relying on familiar answer patterns. And when those patterns change? Their performance can tank—sometimes by over half!

The researchers behind this eye-opening study dug deep into how LLMs operate. These AI systems are designed to process and generate human-like language, trained on vast datasets including books and scientific articles. They can respond to questions and summarize information, making them seem intelligent. This led to excitement about using AI for clinical decision-making, especially as these models achieved impressive scores on medical licensing exams.

But hold on! High test scores don’t equate to true understanding. In fact, many of these models simply predict the most likely answer based on statistical patterns, raising a crucial question: are they genuinely reasoning through medical scenarios, or just mimicking answers they've previously seen? This was the dilemma explored in the recent study led by Suhana Bedi, a PhD student at Stanford University.

Bedi expressed her enthusiasm for bridging the chasm between model building and real-world application, emphasizing that accurate evaluation is vital. “We have AI models achieving near-perfect accuracy on benchmarks like multiple-choice medical licensing exam questions, but that doesn’t reflect reality,” she said. “Less than 5% of research evaluates LLMs on real patient data, which is often messy and fragmented.”

To address this gap, the research team developed a benchmark suite of 35 evaluations aligned with real medical tasks, verified by 30 clinicians. They hypothesized that most models would struggle on administrative and clinical decision support tasks because these require intricate reasoning that pattern matching alone cannot resolve—precisely the sort of thinking that matters in real medical practice.

The team modified an existing benchmark called MedQA, selecting 100 multiple-choice questions and replacing correct answers with “None of the other answers” (NOTA). This subtle yet powerful change forced the AI systems to actually reason through the questions instead of resorting to familiar patterns. A practicing clinician reviewed the modified questions to ensure medical appropriateness.

When the researchers evaluated six popular AI models, including the likes of GPT-4o and Claude 3.5 Sonnet, they were prompted to reason through each question using a method called chain-of-thought, which encourages detailed, step-by-step explanations. This approach aimed to reinforce genuine reasoning over guesswork.

The results were concerning. All models struggled when faced with the modified NOTA questions, demonstrating a notable decline in accuracy. For example, widely used models like GPT-4o and Claude 3.5 Sonnet saw their accuracy drop by over 25% and 33%, respectively. The most alarming drop occurred with Llama 3.3-70B, which got almost 40% more questions wrong when familiar answer formats were altered.

Bedi expressed her surprise at the consistent performance decline across all models, remarking, “What shocked us most was how all models struggled, including the advanced reasoning models.” This suggests that current AI systems might not be adequately equipped to tackle novel clinical situations—especially as real patients often present with overlapping symptoms and unexpected complications.

In Bedi’s own words, “These AI models aren’t as reliable as their test scores suggest.” When the answer choices were modified, performance dipped dramatically, exemplifying that some models could plummet from 80% accuracy to just 42%. It’s akin to a student breezing through practice tests only to fail when the questions are rephrased. Thus, the conclusion is clear: AI should assist doctors, not replace them.

Despite the study’s limited scope—only 68 questions—the consistent performance decline raises significant concerns. The authors stress that more research is necessary, particularly testing on larger datasets and employing varied methods to better evaluate AI capabilities.

“We only tested 68 questions from one medical exam, so this isn’t the whole picture of AI’s capabilities,” Bedi noted. “We used a specific approach to test reasoning, and there might be other methods that uncover different strengths or weaknesses.” For effective clinical deployment, more sophisticated evaluations are essential.

The research team identified three key priorities for the future: developing evaluation tools that distinguish true reasoning from pattern recognition, enhancing transparency regarding how current systems deal with novel medical issues, and creating new models that prioritize reasoning abilities over mere memorization.

“We aim to develop better tests to differentiate AI systems that genuinely reason from those that just memorize patterns,” Bedi concluded. “This research is about ensuring AI can be safely and effectively utilized in medicine, rather than just doing well on tests.” The implications are clear: impressive test scores aren’t a green light for real-world readiness in complex fields like medicine. As Bedi puts it, “Medicine is complicated and unpredictable, and we need AI that can navigate this landscape responsibly.”