Evaluation of ChatGPT-4o's and DeepSeek R1's responses to urological problems: a comparative study.

BACKGROUND

Urology presents unique challenges for AI systems, requiring both extensive medical knowledge and advanced reasoning. While large language models (LLMs) like GPT-4 have shown promise in medical education and decision support, their performance in urology remains underexplored.

OBJECTIVE

To provide a time-stamped comparison of two representative large language models available at the time of evaluation, ChatGPT-4o and DeepSeek R1, in answering urology-related single-choice questions, and to evaluate their accuracy, stability, and response consistency across different response configurations.

METHODS

A total of 809 single-choice questions from the Chinese National Qualification Examination for Attending Physicians in Urology were administered to ChatGPT-4o and DeepSeek R1. Each model was tested under three configurations: basic mode, deep-thinking mode, and web-enabled retrieval. Accuracy was calculated for each configuration, and statistical comparisons were performed using McNemar's test. Stability across reasoning modes was assessed by comparing performance variability. Additional analyses examined performance differences between short-answer and case-based clinical questions.

RESULTS

ChatGPT-4o achieved accuracy rates of 78.12%, 73.79%, and 78.99% in basic, deep-thinking, and web-enabled retrieval modes, respectively. DeepSeek R1 outperformed ChatGPT-4o across all configurations, with accuracy rates of 83.19%, 81.46%, and 84.55%, respectively. All between-model differences were statistically significant (p < 0.001). DeepSeek R1 demonstrated greater internal stability across reasoning modes, whereas ChatGPT-4o showed notable variability. In subgroup analyses, DeepSeek R1 exhibited a more pronounced advantage in complex, case-based clinical questions. Both models performed consistently across urological disease categories, and findings were limited to the Chinese-language context in which the evaluation was conducted.

CONCLUSION

DeepSeek R1 showed superior performance compared with ChatGPT-4o in both accuracy and stability when answering urology-related examination questions, particularly in complex case-based scenarios. These results suggest that optimized LLMs may have potential utility in urology education and examination-style question answering, especially within Chinese-language environments. However, these findings should not be interpreted as evidence of readiness for real-world clinical decision support, and further validation in clinically realistic settings is required.