Multidomain expert evaluation of leading large language models as providers of vaccination and preventive medicine information.

OBJECTIVES

Vaccination is one of the most effective public health interventions. Large language models (LLMs) show great capability for providing health information. This study compared the performance of ChatGPT 3.5 (G3E), ChatGPT 4o in English (G4E), Claude 3.0 (CDE), Gemini 1.5 (GME), and ChatGPT 4o in Italian (G4I) in delivering information on vaccination and preventive medicine.

STUDY

DESIGN

Mixed-method analysis evaluating large language models.

METHODS

Twenty-six expert-designed healthcare scenarios were used to evaluate each model through an adapted DISCERN-based instrument. Four experts independently rated outputs across six domains: Information Reliability, Information Quality, Medical Appropriateness, Impact on Vaccine Hesitancy, Potential for Behavioral Influence, and Overall Rating. Medians and interquartile ranges were calculated, and mixed-effects ordinal logistic regression models were applied to account for inter-rater variability.

RESULTS

G4E showed the highest performance, with significant advantages in Overall Rating (OR = 2.17, 95% CI: 1.20-3.92, p = 0.010) and Medical Appropriateness (OR = 1.86, 95% CI: 1.04-3.33, p = 0.036). G4I outperformed others in Information Quality (OR = 1.76, 95% CI: 1.06-2.93, p = 0.030) but scored lower for vaccine hesitancy and behavioral influence. GME performed weaker across qualitative domains, with occasional generation issues, while CDE and G3E yielded intermediate, consistent results.

CONCLUSIONS

Differences among LLMs reflect model architecture, training data, and language adaptation, influencing clarity, accuracy, and persuasive tone. These disparities highlight the need for domain-specific fine-tuning and language-sensitive optimization to enhance public health communication. LLMs show uneven performance in providing accurate and behaviorally effective vaccine information, underscoring the importance of evaluation and cautious integration into health communication strategies.