Performance of a large language model on the reasoning tasks of a physician.

Clinical reasoning is a useful benchmark for LLM task performance. This study shows that OpenAI's o1-preview model outperforms other LLMs in diagnostic and management reasoning, particularly generating differential diagnoses. The study included a real-world comparison of the LLM and human clinicians in emergency department patients. The LLMs performance in probabilistic reasoning and triage remains similar to previous models, suggesting strengths in complex critical thinking but no improvement in probability-based tasks. Based on these results, the authors advocate for prospective clinical trials to measure the impact of LLMs on diagnostic errors and other patient-centered outcomes.