A Comparison of Patient Information Sheets for Gestational Diabetes Created by Large Language Models and Health Professionals.

OBJECTIVE

In recent years, the exponential growth of artificial intelligence (AI) has fuelled extensive research and high expectations for its potential applications. Generative AI has shown promise in streamlining clinical work. This study aimed to evaluate the ability of ChatGPT 3.5, a publicly available generative AI tool, to generate high-quality patient information materials for gestational diabetes mellitus, a common pregnancy complication.

METHODS

This was an anonymized, within-subjects survey study comparing how obstetrical health care workers rated the understandability and actionability of 2 patient information sheets: the current standard version used at The Ottawa Hospital and an AI-derived version created using ChatGPT 3.5. Eligible obstetrical health care workers participated in the survey, reviewing the 2 versions of the patient education sheet without labels to identify which sheet was which. An adapted version of the Patient Education Materials Assessment Tool (PEMAT) was used to evaluate and score both versions on their understandability and actionability. We also collected information on the respondents' opinions on and exposure to the use of AI in health care to help describe their level of AI-familiarity.

RESULTS

A total of 70 complete responses were received and included in the analysis. Survey respondents consisted of nurses (52.9%), resident or fellow physicians (28.6%), and practising physicians (14.3%). Respondents rated the standard and AI-generated materials similarly in understandability. Although the AI-generated version scored slightly lower in actionability, both instruments achieved scores above the 70% threshold, which is generally accepted as evidence that patient education materials are sufficiently understandable and actionable.

CONCLUSIONS

These findings suggest that AI can produce patient education materials of similar quality to those currently used at The Ottawa Hospital. The actionability of the AI-generated patient education materials could be improved via prompt generation, thereby improving AI applications in clinical work and patient care.