Not all artificial intelligence large language models are created equal: heterogeneous performance in therapeutic decision-making for recurrent epithelial ovarian cancer.

OBJECTIVE

To evaluate the concordance between therapeutic recommendations generated by 3 widely used consumer large language models (Gemini, Microsoft Copilot, and ChatGPT) and the decisions of a multidisciplinary tumor board in the management of the first recurrence of high-grade serous ovarian cancer. Secondary objectives were to characterize recurrent error patterns and to assess the safety of model-generated recommendations using a 3-tier taxonomy.

METHODS

A retrospective observational analysis was conducted on 44 consecutive patients with a first recurrence of high-grade serous ovarian cancer treated at a tertiary referral center. For each case, an identical, anonymized, structured zero-shot prompt was submitted to the 3 models. The multidisciplinary tumor board decision served as the reference comparator. Overall concordance and concordance for the chemotherapy backbone, maintenance or targeted-agent class, and surgical indication were compared between models using Cochran's Q test and pairwise McNemar tests (Holm-adjusted). Discordant recommendations were categorized as true hallucinations, unsafe, or suboptimal but guideline-concordant.

RESULTS

Concordance varied substantially across models (Cochran's Q p<.001): Gemini 77.3% (95% confidence interval 62.2 to 88.5), Microsoft Copilot 36.4% (95% confidence interval 22.4 to 52.2), and ChatGPT 18.2% (95% confidence interval 8.2 to 32.7); Gemini differed significantly from the other 2 models (Holm-adjusted p<.001), whereas the difference between Microsoft Copilot and ChatGPT did not reach significance after adjustment (p=.057). Component-level analysis localized the divergence to systemic therapy: surgical-indication agreement was uniformly high (93.2% to 95.5%, p=.85). All 3 models generated unsafe recommendations for the re-use of poly (ADP-ribose) polymerase) inhibitors in patients with documented prior progression on poly (ADP-ribose) polymerase inhibitor maintenance, with marked differences in frequency (Microsoft Copilot 10, ChatGPT 3, and Gemini 2; 15 of 74 total discordances). The remaining discordances were suboptimal but guideline concordant.

CONCLUSIONS

Consumer large language models showed heterogeneous and clinically meaningful differences in reproducing Multidisciplinary Tumor Board decisions in recurrent ovarian cancer; none were free of unsafe recommendations. Their current performance does not support independent clinical use, and expert supervision is required to ensure patient safety.