Menu
Back to AI and Diagnosis
09/01/26

Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine.

Sheppert, Alexander P; Adams, Brian; Sheppert, Andrew D; Riley, Sylvia. J Am Med Inform Assoc.

Collection Topics
AI and Diagnosis
Share

Most recent

Comparative cost of care recommended by large language models versus treating physicians.

Aran, Goldhaber-Fiebert, Many, et al., NEJM AI
10/15/26 Business Case for Excellence Tests, Screening, and Laboratory Processes AI and Diagnosis

How can we prevent clinical deskilling when using AI?

Scott, van der Vegt, Douglas, BMJ Qual Saf.
09/21/26 AI and Diagnosis Cognition and Decision Making

AI for health care quality and patient safety: scoping review of diagnostic, predictive, and decision support applications.

Xu, Veillard, Kong, J Med Intern Res.
09/21/26 AI and Diagnosis Culture, Work System, and Implementation Health IT

Summary

Claims about LLM diagnostic reasoning depend partly on whether models have encountered similar cases during training. This study compared five frontier LLMs on 2,000 clinical case reports from before and after presumed training cutoffs. Diagnostic accuracy was nearly identical between cohorts, with little lexical or semantic overlap. However, the evaluation relied partly on an AI judge, limiting how conclusively the findings establish “reasoning.”
Read the paper
Sheppert AP, Adams B, Sheppert AD, et al. Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine. J Am Med Inform Assoc. 2026;33(9):1617-1626.