Sticks and Stones: Bias and Readability Assessment in Large Language Model-Generated Patient Education for Anterior Cruciate Injury.

OBJECTIVE

This study aimed to (1) explore sex/gender/ethnic/socioeconomic status (SES)-related bias in large language model (LLM)-generated anterior cruciate ligament (ACL) patient education materials (PEMs) and (2) assess the accuracy, consistency, and understandability of the information provided.

DESIGN

Cross-sectional study.

METHODS

Four LLMs provided PEMs on ACL injury following 10 unique personas. Readability was assessed via Flesch-Kincaid Grade Level. Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) assessed understandability and actionability. The Linguistic Inquiry and Word Count dictionary assessed language differences to measure sex/gender/ethnic/SES-related bias. A qualitative analysis assessed accuracy and robustness.

RESULTS

Forty LLM-generated responses were analyzed. Mean Flesch-Kincaid Grade Level ranged between 9.9 (SD=0.8; Low SES persona) and 11.4 (SD=1.5; Boy persona). Thirty-six (90%) and 27 (67.5%) responses scored above the 70% adequate PEMAT-P threshold for understandability and actionability, respectively. The Kruskal-Wallis test and Mann-Whitney U test revealed no statistically significant language differences between personas, across LLMs (p>.05). Linguistic analysis revealed that 37 (92.5%) persona responses had a negative tone.

CONCLUSION

The LLM-generated PEMs were adequately understandable with minimal sex/gender/ethnic/SES-related bias and provided actionable information. The difficult reading level may limit generalizability. Crucial information was lacking from PEMs and the negative tone may unintentionally increase patient fear. JOSPT Open 2026;4(1):69-81. Epub 17 December 2025. doi:10.2519/josptopen.2025.0190.