Assessment of accuracy, quality, and readability of large language models in kidney transplantation practice.

PURPOSE: This study aimed to comparatively evaluate the accuracy, overall quality, and readability of responses generated by three contemporary large language models (LLMs)-Grok AI, ChatGPT-5 Plus, and Gemini 2.5 Flash-to clinical questions commonly encountered in kidney transplantation (KT) practice.

METHODS

Ten open-ended clinical scenarios covering surgical techniques, complication management, immunosuppressive strategies, and perioperative care were independently assessed by two experienced physicians. LLM-generated responses were evaluated using a structured rubric assessing scientific accuracy, comprehensiveness, clarity, and clinical relevance (equally weighted), along with the Global Quality Score (GQS). Readability was analyzed using the Flesch Reading Ease Score (FRES). Inter-rater reliability was determined using Cronbach's alpha and intraclass correlation coefficients (ICC).

RESULTS

All three LLMs demonstrated consistently high performance, with mean rubric scores exceeding 9.2/10 and GQS values above 4.4/5. Grok AI achieved the highest numerical scores (rubric: 9.5 ± 0.71; GQS: 4.7 ± 0.48), followed by ChatGPT-5 Plus (9.4 ± 0.70; 4.6 ± 0.52) and Gemini 2.5 Flash (9.2 ± 1.03; 4.4 ± 0.70); however, no statistically significant differences were observed (p > 0.05). FRES values ranged from 23.85 to 32.19, indicating professional-level readability suitable for physicians. Inter-rater reliability was high across all measures (Cronbach's alpha: 0.737-0.889; ICC: p < 0.05). Key limitations include the potential for LLM hallucinations, the limited number of scenarios, and the evaluation of text-based responses at a single time point.

CONCLUSIONS

Current-generation LLMs show high and comparable performance in addressing complex KT-related clinical questions. Their consistent accuracy, quality, and readability support their potential role as adjunctive clinical decision-support tools, provided appropriate physician oversight.