Public health · 2025
Evaluation of large language models in the clinical management of multiple chronic conditions: A comparative study of DeepSeek-R1 and ChatGPT-o1.
СтатьяСравнительное исследованиеНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Zeng Haimin, Liu Guanzhuo, Liu Wenjun, Zou Siyu, Chen Bofan, Yang Ruhui, Lin Fangzhen, Peng Jie, Gao Yuan, Hao Liang
Аннотация
OBJECTIVES: To systematically compare the performance of two mainstream large language models (LLMs), DeepSeek-R1 and ChatGPT-o1, in addressing clinical questions related to multiple chronic conditions (MCCs).
STUDY DESIGN: Comparative evaluation study.
METHODS: Forty-eight clinical questions related to MCCs, covering varying levels of difficulty, were developed based on authoritative clinical guidelines and independently submitted to DeepSeek-R1 and ChatGPT-o1 in three separate rounds. A panel of five experts in clinical medicine and medical informatics conducted blinded evaluations of 288 de-identified responses across four dimensions: accuracy, comprehensiveness, clarity, and relevance. Statistical differences between models were analyzed using the Mann-Whitney U test and the Kruskal-Wallis H test. Inter-rater reliability was assessed with Fleiss' kappa.
RESULTS: Inter-rater reliability was high (Fleiss' kappa = 0.878, P < 0.05). While both models avoided major factual errors, ChatGPT-o1 significantly outperformed DeepSeek-R1 in all four evaluation dimensions. DeepSeek-R1, however, showed greater internal consistency across the three rounds of responses (Fleiss' kappa = 0.635, P < 0.05).
CONCLUSIONS: ChatGPT-o1 demonstrated superior performance in MCC-related clinical tasks, particularly in complex cases. Nevertheless, both models remain susceptible to hallucinations, underscoring the need to enhance clinical accuracy and interpretability. This study provides empirical evidence for the application of LLMs in MCC management and highlights the importance of improving model reliability, standardizing evaluation frameworks, and exploring multi-model integration to support their safe and effective use in clinical settings.
STUDY DESIGN: Comparative evaluation study.
METHODS: Forty-eight clinical questions related to MCCs, covering varying levels of difficulty, were developed based on authoritative clinical guidelines and independently submitted to DeepSeek-R1 and ChatGPT-o1 in three separate rounds. A panel of five experts in clinical medicine and medical informatics conducted blinded evaluations of 288 de-identified responses across four dimensions: accuracy, comprehensiveness, clarity, and relevance. Statistical differences between models were analyzed using the Mann-Whitney U test and the Kruskal-Wallis H test. Inter-rater reliability was assessed with Fleiss' kappa.
RESULTS: Inter-rater reliability was high (Fleiss' kappa = 0.878, P < 0.05). While both models avoided major factual errors, ChatGPT-o1 significantly outperformed DeepSeek-R1 in all four evaluation dimensions. DeepSeek-R1, however, showed greater internal consistency across the three rounds of responses (Fleiss' kappa = 0.635, P < 0.05).
CONCLUSIONS: ChatGPT-o1 demonstrated superior performance in MCC-related clinical tasks, particularly in complex cases. Nevertheless, both models remain susceptible to hallucinations, underscoring the need to enhance clinical accuracy and interpretability. This study provides empirical evidence for the application of LLMs in MCC management and highlights the importance of improving model reliability, standardizing evaluation frameworks, and exploring multi-model integration to support their safe and effective use in clinical settings.