American journal of surgery · 2026
Artificial intelligence performance in generating colorectal surgery board questions.
СтатьяНазвание и аннотация: Перевод пока не подготовлен
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Zuo Jonathan, Marty Makenna, Maniskas Seija, Akopian Gabriel, Zaghiyan Karen, Fleshner Phillip, Truong Adam
Аннотация
BACKGROUND: Large language models (LLM) can pass medical licensing and specialty board exams, but their ability to generate high-quality board-style exam questions is uncertain.
METHODS: Three LLMs each generated 20 colorectal surgery board questions in accordance with American Board of Colon and Rectal Surgery guidelines. Questions from the Colon and Rectal Surgery Educational Program (CARSEP) served as comparators. Board-certified colorectal surgeons, blinded to source, graded each question on clarity, relevance, suitability, distractor quality, and adequacy of rationale, and categorized questions as "Approved for Committee," "Author to Review," or "Not Accepted."
RESULTS: CARSEP demonstrated the highest "Approved for Committee" rate (65%), compared with ChatGPT-4o (7%), Copilot Pro (10%), and Gemini Advanced (10%). CARSEP significantly outperformed most LLMs across all evaluation domains (p < 0.001), except for question suitability, where most LLM questions received >70% very good to good ratings.
CONCLUSIONS: Although LLMs demonstrate potential, they are currently unable to consistently generate high-quality, colorectal surgery board-style questions.
METHODS: Three LLMs each generated 20 colorectal surgery board questions in accordance with American Board of Colon and Rectal Surgery guidelines. Questions from the Colon and Rectal Surgery Educational Program (CARSEP) served as comparators. Board-certified colorectal surgeons, blinded to source, graded each question on clarity, relevance, suitability, distractor quality, and adequacy of rationale, and categorized questions as "Approved for Committee," "Author to Review," or "Not Accepted."
RESULTS: CARSEP demonstrated the highest "Approved for Committee" rate (65%), compared with ChatGPT-4o (7%), Copilot Pro (10%), and Gemini Advanced (10%). CARSEP significantly outperformed most LLMs across all evaluation domains (p < 0.001), except for question suitability, where most LLM questions received >70% very good to good ratings.
CONCLUSIONS: Although LLMs demonstrate potential, they are currently unable to consistently generate high-quality, colorectal surgery board-style questions.