Guideline Adherence in the Diagnosis of Polycystic Ovary Syndrome Among Adolescents: Comparing Obstetricians and Gynecologists with Large Language Models.

STUDY

OBJECTIVE

Polycystic ovary syndrome (PCOS) in adolescence is a condition associated with long-term cardiovascular, metabolic, and psychological outcomes. The diagnosis of PCOS in adolescence poses diagnostic challenges for obstetricians and gynecologists due to pubertal variability and the use of criteria distinct from adult standards. Inter-clinician variability and the role of large language models (LLMs) in this context remain insufficiently explored. This study evaluated guideline adherence among physicians and LLMs using a vignette-based design.

METHODS

Ten guideline-based adolescent PCOS vignettes (ESHRE/ASRM 2023), validated by three obstetrics and gynecology specialists, were administered to obstetricians and gynecologists in Türkiye (n = 150). The same vignettes were presented to four LLMs (ChatGPT-5.2, Gemini 3 Flash, Grok 4, DeepSeek V3.2) via a standardized prompt using two independent user accounts in standard and reasoning modes. Inter-account consistency was assessed using accounts.

RESULTS

Physicians achieved a mean of 5.0 ± 1.9 correct responses out of 10 vignettes (each scored as 1 point), whereas LLMs demonstrated significantly higher performance (7.6 ± 1.1; P < .001. Physicians with national assisted reproductive technology certification achieved higher scores in difficult cases. Inter-rater agreement among physicians was low (κ = 0.127, κ = 0.032, κ = 0.027). ChatGPT-5.2 (reasoning mode) and Gemini 3 Flash (standard mode) showed high inter-account agreement (P < .001). Reasoning mode increased overall LLM accuracy.

CONCLUSION

Significant inter-clinician guideline discordance was observed in adolescent PCOS diagnosis. Strengthening this domain is critical for adolescent health. Given their nondeterministic and evolving architectures, LLMs should be positioned as adjunctive tools supporting clinical judgment rather than as independent guideline authorities.