[Evaluation of the efficacy of different large language models in solving clinical problems of early gastric cancer].

Objective

To evaluate the effectiveness of different large language models (LLM) in addressing clinical questions related to early gastric cancer (EGC).

Methods

Select four types of LLM, including ChatGPT-4o, DeepSeek-V3.2, Gemini-2.5, and Grok-3, and develop 25 EGC professional knowledge questions based on international authoritative guidelines. Ten clinical physicians evaluated the accuracy, relevance, completeness scores of the four LLM' responses. Meanwhile, based on patient interviews, 15 EGC life and health questions were extracted, and 10 EGC patients were asked to rate their understanding of the answers to the 4 LLM; The readability of LLM solution was further evaluated by clinicians. One-way analysis of variance or Friedman test was used for comparison between the groups, and paired t test corrected by Bonferroni method or Wilcoxon signed rank test corrected by Bonferroni method was used for comparison between two groups. These analyses were used to compare the accuracy, relevance, completeness, comprehension scores and readability assessment of different LLM solutions in addressing clinical questions related to EGC, thereby enabling a dual assessment of both expertise and accessibility.

Results

The accuracy scores of DeepSeek-V3.2 [M（Q1，Q3），78.0 (76.5, 82.0) points], Gemini-2.5 [80.0 (77.0, 82.5) points], and Grok-3 [82.0 (74.0, 83.0) points] evaluated by clinical physicians were lower than those of ChatGPT-4o [86.0 (83.5, 89.0) points] (all P<0.05); The relevance scores of DeepSeek-V3.2 [82.0 (77.5, 87.0) points] and Grok-3 [86.0 (82.0, 88.0) points] were both lower than those of ChatGPT-4o [88.0 (87.5, 90.5) points] (both P<0.05); The completeness scores of DeepSeek-V3.2 [75.0 (69.5, 80.0) points] and Gemini-2.5 [84.0 (81.0, 88.5) points] were both lower than those of Grok-3 [90.0 (85.5, 91.5) points] (both P<0.05). Patients' score for the comprehensibility of DeepSeek-V3.2's responses [79.0 (77.0, 90.0) points] was lower than that for ChatGPT-4o [88.0 (86.0, 90.0) points] (P<0.05). In terms of readability assessment, the simple measure of gobbledygook (SMOG) score for DeepSeek-V3.2's responses (14.1±1.6) was lower than that of ChatGPT-4o (16.4±2.3), Gemini-2.5 (15.8±2.0) and Grok-3 (15.8±1.1) (all P<0.05).

Conclusions

Different LLM have their own advantages and disadvantages in answering clinical questions about EGC. ChatGPT-4o has a better comprehensive evaluation, while DeepSeek-V3.2 has better readability in answering questions.