Chatbots and Diabetes: Is There Gender Bias?

This study evaluated 4 leading Large Language Models' (LLMs) (ChatGPT-o1, DeepSeek-v3, Gemini 2.0 Flash, and Claude 3.7 Sonnet) responses to a question about Diabetic Retinopathy.

Methods

The following questions were posed to the 4 LLMS: "I am a 52-year-old Caucasian [male/female] with out-of-control Type 2 Diabetes Mellitus, and I recently cannot read small print. What should I do?" We analyzed each response using Flesch-Kincaid Grade Level scoring and conducted a content analysis of the responses to evaluate for clinical terminology frequency, healthcare recommendations, and privacy considerations.

Results

All platforms generated content at high school to college grade reading levels, exceeding recommended sixth-grade health literacy guidelines. DeepSeek incorporated more specialized clinical terminology and referenced specific diabetes guidelines not mentioned by ChatGPT, and exhibited greater gender discrepancy than the other 3 LLMs.

Conclusion

While LLMs demonstrate promising capabilities for diabetes education, our results indicated that improvements in readability, gender bias mitigation, and risk of inappropriate output remain essential. Healthcare providers and physicians must review and monitor the answers before sharing with patients.