Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study.
Jintao Wei, Shouyin Jiang, Ting Yin, Jiangchun Ma, Qiang Li, Yu Tian, Mengting Yan, Zhuyi Shen, Xiangkang Lv, Ximei Ma, Shanxiang Xu, Mao Zhang
OBJECTIVE: Medical domain large language models (LLMs) exhibit verified clinical decision-support capabilities in simulated case analyses and standardized tests, yet their diagnostic efficacy in real-world emergency settings remain insufficiently explored. This study evaluates the diagnostic performance of 5 mainstream LLMs (ChatGPT-4o, Gemini-2.0, Grok3, DeepSeek-V3, Doubao) against emergency department junior physicians (EDJP) on real-world emergency internal medicine cases. METHODS: A single-center retrospective analysis design was conducted. 154 anonymized emergency internal medicine patients of the Second Affiliated Hospital of Zhejiang University School of Medicine from January to May 2025 were included, covering common acute diseases of multiple systems. 15 EDJPs and 5 LLMs were selected to diagnose the cases, respectively. The main diagnostic accuracy, comprehensiveness of differential diagnosis, and response time were used as evaluation indicators. Non-parametric tests were used for statistical analysis. RESULTS: (1) Main diagnostic accuracy: DeepSeek-V3 (90.0%), ChatGPT-4o (86.0%), and Grok3 (86.0%) were significantly higher than that of EDJP (77.5%, CONCLUSION: This retrospective study shows that LLMs outperformed EDJPs in diagnostic accuracy, differential diagnosis comprehensiveness and response efficiency for emergency internal medicine diseases, demonstrating significant potential for clinical decision support. Subsequent efforts will focus on exploring how to effectively integrate into physician-led collaborative workflows to enhance emergency care quality and efficiency.
Read on ELI