Comparative study on the efficacy of large language models (LLMs) in health management for patients with urological and andrological diseases: a multidimensional quality evaluation and applicability analysis.

PURPOSE: This study aims to evaluate the efficacy of large language models (LLMs) in health management for urological and andrological conditions by comparing two Chinese LLMs: DeepSeek and Kimi.

METHODS

The responses of DeepSeek and Kimi to 30 questions on six urological and andrological diseases were quantitatively assessed using quality assessment tools, expert assessments, and readability metrics.

RESULTS

In quality assessments, DeepSeek outperformed Kimi in only two instances: responses to "What are the surgical options for kidney stones?" (PEMAT-AI: P = 0.012; DISCERN-AI: P = 0.024) and "What are the treatment modalities for bladder cancer?" (PEMAT-AI and DISCERN-AI: P < 0.001). Expert assessments revealed that DeepSeek demonstrated superior accuracy and safety compared to Kimi only in addressing "Dietary considerations for kidney stone patients" (accuracy: P < 0.001; safety: P = 0.02). No statistically significant differences were found in quality assessments or expert assessments between the models for disease-specific categories. However, Kimi outperformed DeepSeek in readability evaluations.

CONCLUSION

Both DeepSeek and Kimi exhibit substantial efficacy and clinical applicability in managing urological and andrological diseases. To enhance their practical use, Kimi requires improvements in accuracy and safety, while DeepSeek should focus on improving readability.