Benchmarking Large Language Models for Bladder Cancer: Comparative Knowledge Retrieval and Clinical Reasoning of DeepSeek and ChatGPT.

BACKGROUND

This study evaluates the comparative performance of DeepSeek-R1, ChatGPT-5 Pro, and urologists in the specific domains of bladder cancer knowledge retrieval and complex clinical reasoning.

METHODS

We constructed a benchmark dataset comprising 91 standardized multiple-choice questions on bladder cancer derived from MedQA, MedMCQA, and the Chinese National Medical Licensing Examination, alongside three retrospectively reconstructed real-world cases. Five advanced models, including DeepSeek (V3, R1) and OpenAI variants (ChatGPT-5, ChatGPT-5 Pro, ChatGPT-5 Mini), were evaluated. Accuracy and stability were assessed across three independent runs for standardized questions. In clinical simulations, DeepSeek-R1 and ChatGPT-5 Pro were benchmarked against human urologists. A blinded expert panel of three senior urologists evaluated responses using a 5-point Likert scale across four dimensions: readability, medical accuracy, diagnostic test appropriateness, and logical coherence.

RESULTS

In standardized testing, all models achieved >92% accuracy. ChatGPT-5 Pro ranked first (97.52%), followed closely by DeepSeek-R1 (95.04%), with both displaying superior stability. In clinical simulations, DeepSeek-R1 demonstrated logical coherence comparable to both human experts and ChatGPT-5 Pro (P > 0.05). However, DeepSeek-R1 scored significantly lower than ChatGPT-5 Pro regarding readability and "error-free" rates. Notably, human urologists significantly outperformed both AI models in diagnostic test appropriateness (P < 0.01), primarily because DeepSeek-R1 struggled with "test avoidance," tending to recommend redundant investigations (P < 0.0001).

CONCLUSIONS

DeepSeek-R1 demonstrates excellent accuracy and expert-level clinical reasoning, exhibiting competitiveness with ChatGPT-5 Pro. Although slightly inferior in readability and prone to suggesting unnecessary tests, its core reasoning capabilities remain favorable.