Assessment of readability, reliability, and quality of large language models in addressing frequently asked questions regarding prenatal screening for fetal chromosomal anomalies.

OBJECTIVE

To evaluate the responses of large language models (LLMs) to prenatal screening questions for fetal chromosomal anomalies in terms of scientific accuracy, guideline adherence, depth of response, and clarity, as well as their potential roles in patient education and health communication.

METHODS

Responses generated by ChatGPT-4o and Gemini Advanced 1.5 Pro to frequently asked questions (FAQs) on prenatal screening for fetal chromosomal anomalies were systematically compared. Expert reviewers assessed each reply using a Likert scale across four criteria: adherence to clinical guidelines, scientific accuracy, clarity, and depth of response. Readability scores were calculated with the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) formulas.

RESULTS

LLMs were evaluated based on their responses to prenatal screening questions. ChatGPT-4o received higher proportions of favorable ratings ("good" or "excellent") across all evaluation criteria, including scientific accuracy, guideline adherence, clarity, and depth of response. The difference in average Global Quality Scale scores between ChatGPT and Gemini was statistically significant (3.87 vs. 3.70; P = 0.003). The mean FRE scores were 20.11 for ChatGPT-4o and 32.25 for Gemini Advanced 1.5 Pro, and the mean FKGL scores were 15.09 and 12.64, respectively. The differences in both FRE and FKGL scores were statistically significant (P = 0.001 and P = 0.002, respectively).

CONCLUSION

LLMs like ChatGPT-4o and Gemini Advanced 1.5 Pro can provide highly accurate responses for prenatal screening for fetal chromosomal anomalies, but they should only be used for informational purposes. A healthcare professional should always be consulted before making any final decisions.