MIT: medical AI assistance can guide experts and mislead beginners

A new study published in Nature Medicine and covered by MIT News warns that medical artificial intelligence assistance does not affect every user in the same way. In dermatology diagnosis tests, AI explanations helped improve some results, but they also made people without clinical training more likely to trust wrong system outputs.

What happened

Researchers from MIT, Columbia University, Stanford and other institutions studied how different types of explainable AI influence decisions about skin disease. The work appears in Nature Medicine under the title “Divergent impacts of explainable AI for dermatological diagnosis on clinicians versus lay people.”

The team ran two large experiments. In the first, 623 lay users had to decide whether mole images showed melanoma or nevus. In the second, 153 primary care physicians worked on more complex differential diagnoses involving dermatological conditions. A medical-student cohort was also used to compare levels of expertise.

The support systems included a basic prediction with confidence, heat maps, retrieval of similar images and natural-language explanations generated by a multimodal model. Researchers also tested a fairness-constrained model designed to reduce performance gaps across skin tones.

What the study found

The central conclusion is not that medical AI is useless or should be avoided. The study found that AI assistance can improve the accuracy of both lay users and clinicians, and that the fairness-oriented model helped reduce skin-tone-related disparities in the tested tasks.

But the benefit came with an important cost. Among lay users, LLM-style explanations amplified deference to the system. When the AI was correct, those explanations could improve performance; when it was wrong, they could pull users toward an incorrect answer. According to MIT News, users without clinical training tended to find vague or generic explanations more convincing and could become more confident even when they were wrong.

The pattern was different for primary care physicians. Clinicians were more resilient to incorrect advice and performed best when they received only the model prediction, without an added explanation. The authors’ reading is that prior knowledge changes how the explanation is interpreted: a professional checks the suggestion against clinical judgment, while a beginner may form a judgment from the machine’s explanation.

Why it matters

The news lands at a moment when patients and consumers already use search tools, chatbots and apps to interpret symptoms, images or health questions. FDA-approved tools also exist to help identify skin conditions. The study does not evaluate every such tool or prove clinical outcomes in hospitals, but it highlights a design problem: a clear explanation does not automatically make a recommendation safer.

For teams building AI health products, the message is concrete. Adding a natural-language explanation is not enough to create trustworthy use. Presentation order, user expertise and the possibility that the AI is wrong all change the risk. The authors note that showing the AI explanation first may increase anchoring; one alternative would be to ask users to form their own hypothesis before receiving the system’s suggestion.

What it does not prove

The study focuses on image-based dermatology diagnosis in controlled tasks. It does not prove that all medical chatbots fail, or that all physicians are protected from automation bias. It also does not make AI a substitute for clinical evaluation. Its contribution is more specific: the same explanations that may help an expert review a decision can create overconfidence in a person without training.

The practical implication is cautious: medical AI should be designed around the real user, not only around the model score. In health care, the interface can be as important as the prediction.

Written by Lía Torres — Social and strategic perspective.

Sources consulted

MIT News; Nature Medicine. Exact canonical links appear in the Sources section below.