Evaluation of large language models in the clinical management of multiple chronic conditions: A comparative study of DeepSeek-R1 and ChatGPT-o1.

OBJECTIVES

To systematically compare the performance of two mainstream large language models (LLMs), DeepSeek-R1 and ChatGPT-o1, in addressing clinical questions related to multiple chronic conditions (MCCs).

STUDY

DESIGN

Comparative evaluation study.

METHODS

Forty-eight clinical questions related to MCCs, covering varying levels of difficulty, were developed based on authoritative clinical guidelines and independently submitted to DeepSeek-R1 and ChatGPT-o1 in three separate rounds. A panel of five experts in clinical medicine and medical informatics conducted blinded evaluations of 288 de-identified responses across four dimensions: accuracy, comprehensiveness, clarity, and relevance. Statistical differences between models were analyzed using the Mann-Whitney U test and the Kruskal-Wallis H test. Inter-rater reliability was assessed with Fleiss' kappa.

RESULTS

Inter-rater reliability was high (Fleiss' kappa = 0.878, P < 0.05). While both models avoided major factual errors, ChatGPT-o1 significantly outperformed DeepSeek-R1 in all four evaluation dimensions. DeepSeek-R1, however, showed greater internal consistency across the three rounds of responses (Fleiss' kappa = 0.635, P < 0.05).

CONCLUSIONS

ChatGPT-o1 demonstrated superior performance in MCC-related clinical tasks, particularly in complex cases. Nevertheless, both models remain susceptible to hallucinations, underscoring the need to enhance clinical accuracy and interpretability. This study provides empirical evidence for the application of LLMs in MCC management and highlights the importance of improving model reliability, standardizing evaluation frameworks, and exploring multi-model integration to support their safe and effective use in clinical settings.