Artificial Intelligence Models for Dysphonia Patient Education.

OBJECTIVE

Evaluate and compare artificial intelligence (AI) chatbot responses to queries extrapolated from statements by Dysphonia International (DI).

STUDY

DESIGN

Cross-sectional analysis.

SETTING

Online using ChatGPT, Google AI Overview (GAO), and DI.

METHODS

Two board-certified otolaryngologists and an otolaryngology resident each blindly rated ChatGPT, GAO, and DI responses to spasmodic dysphonia, muscle tension dysphonia, and vocal tremor questions. Response comprehensiveness, accuracy, appropriateness to patients, and patient safety were graded on a 5-point Likert scale. Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES), Gunning Fog Index (GFI), and average word count assessed readability. Analysis was performed with one-way analysis of variance (ANOVA) and Tukey honestly significant difference (HSD); significance was set at P < .05.

RESULTS

ChatGPT was significantly more comprehensive than both GAO (P < .01) and DI (P = .0357). ChatGPT was significantly more accurate than both GAO (P < .01) and DI (P < .01), while GAO was more accurate than DI (P < .01). ChatGPT was found to be significantly more patient-friendly than both GAO (P < .01) and DI (P < .01). ChatGPT had significantly safer recommendations than GAO (P < .01). ChatGPT had a significantly higher FKGL than GAO (P < .01), and a lower FRES than both GAO (P < .01) and DI (P < .01), indicating a more difficult reading level. All resources exceeded the recommended FKGL and GFI scores as well as the FRES of 90 to 100. Interrater reliability was 0.8.

CONCLUSION

ChatGPT performed significantly better than GAO and DI regarding comprehensiveness, accuracy, appropriateness, and was an overall safer AI resource to use than GAO. All resources were above the recommended reading grade level. AI models prove to be a formidable alternative resource for patient-centered clinical information.