Can artificial intelligence models provide reliable medical counselling to fertility patients?

RESEARCH QUESTION: Can generative artificial intelligence (AI) models provide reliable counselling to fertility patients regarding real-world clinical questions?

DESIGN

In this cross-sectional study, 12 clinical questions were developed to reflect common, real-life dilemmas encountered during fertility workup and treatment. Responses to each question were generated by two experienced fertility specialists, and two AI models - ChatGPT and Gemini. Eight leading internationally recognized fertility experts, blinded to the source of each reply, independently rated all the responses on a scale from 1 (strongly disagree) to 10 (strongly agree). Ratings were compared across all four repliers using non-parametric statistical tests.

RESULTS

The replies authored by physicians received significantly higher overall scores than those generated by AI models (P < 0.001). The median scores were highest for Doctor A (9.0), followed by Doctor B (8.0), then ChatGPT (7.0) and finally Gemini, which received the lowest score (4.5). The proportion of high-scoring responses (≥8) was greatest for Doctor A (70.8%), followed by Doctor B (56.3%), then ChatGPT (47.9%) and finally Gemini (35.4%) (P < 0.001).

CONCLUSIONS

Experienced fertility specialists outperformed generative AI models in providing accurate responses to complex clinical questions. Despite the growing accessibility and sophistication of AI tools, their use for individualized fertility counselling remains limited. Continued refinement and clinical validation of AI tools are essential before they can be considered reliable for patient-specific guidance. At present, AI should be viewed as a complementary resource rather than a substitute for expert clinical judgement.