Scaling clinical judgment to evaluate medical AI.

Physician review of clinical reasoning is essential for evaluating medical AI but difficult to scale. This preprint introduces an LLM evaluator and physician-annotated benchmark. Despite limited examples, this study finds that conventional LLM judges often disagree with physicians, while the tested evaluator more closely matches physician scores across clinical-AI tasks, including stepwise diagnosis as case details arrived. The authors have made the code and data for the tools freely available.