Performance evaluation of five major large language models in tuberculosis Q&A systems: A multidimensional assessment of readability, quality, and reliability.

BACKGROUND

Pulmonary tuberculosis (TB) is a chronic infectious disease that burdens patients and public health systems. Limited reach of traditional education and uneven online information may undermine patients' understanding, adherence, and trust. Large language models (LLMs) show promise for TB health education, but systematic evaluation is lacking.

OBJECTIVE

To evaluate five large language models in pulmonary tuberculosis Q&A (Question and Answer) scenarios and examine the effects of different large language models and TB health education themes on response quality, readability, and reliability, thereby supporting standardized artificial intelligence (AI)-assisted health education.

METHODS

This cross-sectional study was conducted from October 5 to October 11, 2025. Twenty pulmonary tuberculosis-related questions were developed with the assistance of a respiratory physician and categorized into five themes. The questions were entered into five large language models (Doubao, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and ChatGPT) to generate 100 text responses. Quality was evaluated using patient-education suitability (C-PEMAT-P) and Global Quality Score (GQS), and readability using seven indices, including Automated Readability Index (ARI) and Flesch Reading Ease Score (FRES). Statistical analyses included One-way analysis of variance (One-way ANOVA), Kruskal-Wallis, and correlation analysis.

RESULTS

GPT-5 achieved the highest C-PEMAT-P scores, followed by Doubao, while GQS scores were similar across models. Models showed significant differences on several readability indices, whereas themes had limited effects. Quality indicators were modestly associated with readability, while readability indices were strongly intercorrelated.

CONCLUSIONS

Model type is a key determinant of TB health education text quality. Quality and reading difficulty are related but relatively independent and should be jointly considered when selecting large language model (LLM)-generated materials. Further studies should include more models, diseases, and patient-reported outcomes to optimize AI-assisted health education.