Efficacy of Large Language Models in Providing Evidence-Based Patient Education for Celiac Disease: A Comparative Analysis.

Background/

Objectives

Large language models (LLMs) show promise for patient education, yet their safety and efficacy for chronic diseases requiring lifelong management remain unclear. This study presents the first comprehensive comparative evaluation of three leading LLMs for celiac disease patient education.

Methods

We conducted a cross-sectional evaluation comparing ChatGPT-4, Claude 3.7, and Gemini 2.0 using six blinded clinical specialists (four gastroenterologists and two dietitians). Twenty questions spanning four domains (general understanding, symptoms/diagnosis, diet/nutrition, lifestyle management) were evaluated for scientific accuracy, clarity (5-point Likert scales), misinformation presence, and readability using validated computational metrics (Flesch Reading Ease, Flesch-Kincaid Grade Level, SMOG index).

Results

Gemini 2.0 demonstrated superior performance across multiple dimensions. Gemini 2.0 achieved the highest scientific accuracy ratings (median 4.5 [IQR: 4.5-5.0] vs. 4.0 [IQR: 4.0-4.5] for both competitors, p = 0.015) and clarity scores (median 5.0 [IQR: 4.5-5.0] vs. 4.0 [IQR: 4.0-4.5], p = 0.011). While Gemini 2.0 showed numerically lower misinformation rates (13.3% vs. 23.3% for ChatGPT-4 and 24.2% for Claude 3.7), differences were not statistically significant (p = 0.778). Gemini 2.0 achieved significantly superior readability, requiring approximately 2-3 fewer years of education for comprehension (median Flesch-Kincaid Grade Level 9.8 [IQR: 8.8-10.3] vs. 12.5 for both competitors, p < 0.001). However, all models exceeded recommended 6th-8th grade health literacy targets.

Conclusions

While Gemini 2.0 demonstrated statistically significant advantages in accuracy, clarity, and readability, misinformation rates of 13.3-24.2% across all models represent concerning risk levels for direct patient applications. AI offers valuable educational support but requires healthcare provider supervision until misinformation rates improve.