Evaluating the Accuracy, Clarity, and Safety of Artificial Intelligence-Generated Information on Clubfoot.

BACKGROUND

Clubfoot is a congenital foot deformity treated primarily by the Ponseti method. Despite its effectiveness, challenges such as delayed treatment and nonadherence persist, leading parents to seek information online. With the rise of artificial intelligence (AI) chatbots as medical information sources, concerns have emerged regarding the accuracy, clarity, and safety of AI-generated content.

OBJECTIVE

To evaluate the accuracy, clarity, and safety of AI-generated responses to common parental questions about clubfoot.

METHODS

Nineteen frequently asked questions by parents of children with clubfoot were identified. Three AI language models-Google's Gemini, OpenAI's ChatGPT-4.0, and Meta AI-were each queried twice with these questions. Two pediatric orthopaedic surgeons independently assessed the AI responses for clarity, accuracy, and safety using a four-point scale (1 = lowest, 4 = highest). Statistical analyses were conducted using chi-square and t -tests with a significance level of P < 0.05.

RESULTS

A total of 228 evaluations were obtained (76 per AI model). The overall average scores were 3.5 for clarity, 3.0 for accuracy, and 3.3 for safety. ChatGPT scored highest in clarity (mean = 3.5), Gemini in accuracy (mean = 3.2) and safety (mean = 3.4). No statistically significant differences were found among the models. Approximately 25% of responses were limited or inaccurate, and only 30% were deemed entirely safe.

CONCLUSION

Although AI chatbots generally provide adequate information on clubfoot, notable concerns about accuracy and patient safety remain. It is recommended that patients consult healthcare professionals rather than rely solely on AI-generated advice. Developing AI models that access up-to-date, evidence-based medical databases could enhance the reliability and safety of health information provided to the public.