Journal of cataract and refractive surgery · 2026
Evaluating large language models vs residents in cataract and refractive surgery: comparative analysis using the American Academy of Ophthalmology Self-Assessment Program.
СтатьяСравнительное исследованиеРандомизированное исследованиеНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Wallerstein Avi, Ramnawaz Taanvee, Gauvin Mathieu
Аннотация
PURPOSE: To assess the accuracy of leading large language models (LLMs) in answering cataract and refractive surgery questions, determine whether prompt complexity affects performance, and compare their accuracy with ophthalmology residents.
SETTING: Not applicable.
DESIGN: Randomized questionnaire-based study using 100 questions from the cataract and refractive surgery section of the Basic and Clinical Science Course (BCSC) Self-Assessment Program.
METHODS: 5 LLMs (ChatGPT-4, ChatGPT-4o, Gemini, Gemini Advanced, and Copilot-Precise Mode) were tested. Data from 1983 BCSC Self-Assessment Program users served as a comparison. Each LLM underwent 2 sessions: one with a simple prompt and another with a contextualized prompt. Accuracy was defined as the proportion of correct answers.
RESULTS: Using the simple prompt, ChatGPT-4o achieved the highest accuracy at 84% (95% CI 77%-91%), followed by Gemini Advanced at 82%, Copilot at 78%, ChatGPT-4 at 77%, and Gemini at 62% ( P < .05). With the contextualized complex prompt, ChatGPT-4o again led (86%; 95% CI 79%-93%). Performance differences between simple and complex prompts were not statistically significant ( P > .05). Except for Gemini, all models' lower 95% CIs exceeded the 60% passing threshold. The mean resident score was 77% (95% CI 75%-79%). Only ChatGPT-4o significantly ( P = .04) outperformed residents, while Gemini Advanced and Copilot trended higher ( P ∼ .10).
CONCLUSIONS: ChatGPT-4o consistently outperformed ChatGPT-4, Gemini, Gemini Advanced, and Copilot, and was the only model to significantly surpass residents. Prompt complexity did not affect LLM performance. All models except Gemini exceeded the 60% accuracy threshold, indicating the potential of LLMs as tools for knowledge assessment in refractive surgery.
SETTING: Not applicable.
DESIGN: Randomized questionnaire-based study using 100 questions from the cataract and refractive surgery section of the Basic and Clinical Science Course (BCSC) Self-Assessment Program.
METHODS: 5 LLMs (ChatGPT-4, ChatGPT-4o, Gemini, Gemini Advanced, and Copilot-Precise Mode) were tested. Data from 1983 BCSC Self-Assessment Program users served as a comparison. Each LLM underwent 2 sessions: one with a simple prompt and another with a contextualized prompt. Accuracy was defined as the proportion of correct answers.
RESULTS: Using the simple prompt, ChatGPT-4o achieved the highest accuracy at 84% (95% CI 77%-91%), followed by Gemini Advanced at 82%, Copilot at 78%, ChatGPT-4 at 77%, and Gemini at 62% ( P < .05). With the contextualized complex prompt, ChatGPT-4o again led (86%; 95% CI 79%-93%). Performance differences between simple and complex prompts were not statistically significant ( P > .05). Except for Gemini, all models' lower 95% CIs exceeded the 60% passing threshold. The mean resident score was 77% (95% CI 75%-79%). Only ChatGPT-4o significantly ( P = .04) outperformed residents, while Gemini Advanced and Copilot trended higher ( P ∼ .10).
CONCLUSIONS: ChatGPT-4o consistently outperformed ChatGPT-4, Gemini, Gemini Advanced, and Copilot, and was the only model to significantly surpass residents. Prompt complexity did not affect LLM performance. All models except Gemini exceeded the 60% accuracy threshold, indicating the potential of LLMs as tools for knowledge assessment in refractive surgery.