As artificial intelligence (AI) models like ChatGPT continue to evolve, the idea of a healthcare system deeply intertwined with AI feels less like science fiction and more like an impending reality. Researchers are investigating how these cutting-edge tools can transform medical decision-making. A recent study published in JAMA Network Open, a medical journal by the American Medical Association, investigated the effects of large language models such as ChatGPT in improving diagnostic reasoning performance among physicians in various specialties. What they discovered surprised even the experts.
The experiment recruited 50 physicians from various hospital systems across the U.S. Participants were presented with real-world patient cases and divided into two groups. One group relied on conventional diagnostic tools, such as medical manuals and internet resources. The other group had ChatGPT at their disposal as a diagnostic aid. At first glance, the results seemed modest: the traditional group scored a median of 74%, while the AI-assisted group performed slightly better at 76%.
But the real surprise came when researchers tested ChatGPT on its own. When tasked with analyzing the same cases independently, the AI achieved a remarkable median score of 92%, outperforming both groups of physicians.
This unexpected outcome raised an intriguing question: Why didn’t the group of participants with access to ChatGPT perform better? After all, the tool at their fingertips is the same tool that showed exceptional performance on its own.
Jonathan H. Chen, the study’s senior author and an assistant professor at Stanford’s School of Medicine, suggests that the answer lies in human behavior. “What is very possible is that once a human feels like they have got a diagnosis, they do not ‘waste time or space’ on explaining more of the steps for why,” Chen explains. This insight aligns with a broader medical phenomenon: Experienced practitioners often rely on intuition or pattern recognition, sometimes limiting their willingness to explore alternative explanations—even when aided by advanced tools.
Researchers delved deeper by examining the interactions between doctors and ChatGPT. They found that many physicians used the AI model like a search engine, providing fragmented inputs rather than full patient cases. Others dismissed ChatGPT’s recommendations when they conflicted with their own diagnoses. Few doctors fully harnessed the AI’s potential, opting to treat it as a search engine rather than using it to analyze comprehensive cases and generate detailed, case-specific insights.