Comparison of ChatGPT-4o, Google Gemini 1.5 Pro, Microsoft Copilot Pro, and Ophthalmologists in the management of uveitis and ocular inflammation: A comparative study of large language models.

PURPOSE: The aim of this study was to compare the latest large language models (LLMs) ChatGPT-4o, Google Gemini 1.5 Pro and Microsoft Copilot Pro developed by three different companies, with each other and with a group of ophthalmologists, to reveal the strengths and weaknesses of LLMs against each other and against ophthalmologists in the field of uveitis and ocular inflammation.

METHODS

Using a personal OphthoQuestions (www.ophthoquestions.com) account, a total of 100 questions from 201 questions on uveitis and ocular inflammation out of a total of 4551 questions on OphthoQuestions, including questions involving multimodal imaging, were included in the study using the randomization feature of the website. In November 2024, ChatGPT-4o, Microsoft Copilot Pro, and Google Gemini 1.5 Pro were asked the same 100 questions: 80 multiple-choice and 20 open-ended questions. Each question was categorized as either true or false. A statistical comparison of the accuracy rates was performed.

RESULTS

Among the 100 questions, ChatGPT-4o, Google Gemini 1.5 Pro, Microsoft Copilot Pro, and the human group (ophthalmologists) answered 80 (80.00%), 81 (81.00%), 80 (80.00%) and 72 (72.00%) questions, respectively, correctly. In the statistical comparisons between the groups for multiple-choice questions, no significant difference was found between the correct and incorrect response rates of the three LLMs and the human group (P=0.207, Cochran's Q test). In the statistical comparisons of responses to open-ended questions, there was no significant difference between the correct and incorrect response rates of the three LLMs and the human group (P=0.392, Cochran's Q test).

CONCLUSION

Although ChatGPT-4o, Google Gemini 1.5 Pro , and Microsoft Copilot Pro answered higher percentages of questions correctly than the human group, the LLMs were not statistically superior to each other or to the human group in the management of uveitis and ocular inflammation.