DOI 10.1590/1413-785220263404e300345
USE OF ARTIFICIAL INTELLIGENCE FOR THE EVALUATION OF THE ENTRANCE EXAMINATION OF THE BRAZILIAN SOCIETY OF SHOULDER AND ELBOW SURGERY.
Аннотация
INTRODUCTION
AI is increasingly used for medical education and assessment, yet effectiveness on specialized exams remains underexplored. We aim to compare multiple AI models with national human averages on SBCOC entrance exams (2021-2023), evaluate answer accuracy and citation reliability, and contrast models.
METHODS
Five models-ChatGPT-4 (standard and literature-trained), ChatGPT-o1-pro, Gemini, and Meta Llama 3.1-answered official SBCOC exams (50 items/year) individually using a unified prompt and identical images. Each response included one choice (A-D) and a cited source. Outcomes were accuracy and source reliability (peer-reviewed articles/textbooks versus websites). Statistics used chi-square tests and one-way ANOVA with Tukey post-hoc (p<0.05).
RESULTS
Across 150 questions, ChatGPT-o1-pro led (66%, 62%, 68%) and exceeded human national averages each year (p<0.05). Gemini (40%, 28%, 38%) and Meta Llama 3.1 (32%, 44%, 50%) underperformed relative to humans, while both ChatGPT-4 versions hovered near the ≥50% pass threshold without significant differences. Regarding sources, ChatGPT-o1-pro and Llama predominantly cited articles or books, ChatGPT-4 alternated between literature and websites, and Gemini did not specify references.
CONCLUSIONS
ChatGPT-o1-pro outperformed the national human average and mostly used credible sources; ChatGPT-4 matched humans, while Gemini and Meta Llama 3.1 lagged. Level of evidence IV; Case series.
Подготовка и перевод публикации
Обработка статьи · этапы, ошибки и управление
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Тип публикации указан по данным источника. Уровень доказательности автоматически не присваивается.