Journal of surgical education · 2026
Leveraging Natural Language Processing for Analysis of EPA Comment Quality by Automated QuAL Scoring.
СтатьяНазвание и аннотация: Перевод в очереди
Обработка статьи · этапы, ошибки и управление
Загрузка состояния…
Ошибки и ограничения
История · было → стало
Состояние обновляется каждые 5 секунд, пока блок открыт. Процент показывается только там, где известен объём работы. Пауза останавливает продолжение этапа; текущий запрос к источнику может завершиться.
Willis Shelby, Jenkins Phillip D, Bedrick Steven, Jesneck Jonathan, Doberne Julie W, Thanawala Ruchi M
Аннотация
OBJECTIVE: Entrustable professional activities (EPAs) are core assessment tools for competency based education. EPA assessments provide a comment section to enrich the context of the assessment that is crucial for learner growth. Previous studies developed the Quality of Assessment for Learning score (QuAL), a validated rating tool for assessing the quality of narrative comments in competency-based medical education. We applied QuAL scoring to a large assessment database to assess the quality of comments.
DESIGN: Assessment comments from a nationwide digital medical education platform were evaluated by QuAL with scores ranging from 0-5 with 0 indicating very low quality, 3 an average comment and 5 a high quality comment. This is based on three components (1) sufficient evidence about performance scored from 0-3 (2) offering suggestion for improvement, scored as 0 or 1 and (3) connection of suggestion to a described behavior, scored as 0 or 1. We leveraged previous work developing a publicly available machine learning model to score comments with QuAL. Python was used to automate the extraction and processing of comment data through the machine learning model.
PARTICIPANTS: Faculty from surgical training programs across the United States submitted EPA assessment comments for surgical residents and fellows.
RESULTS: Of 12,841 EPA assessments, 7,211 (56%) contained a comment. Assessments without comment were excluded from analysis. The mean overall comment score was 2.91 representing average quality. The average evidence was 2.4/3 (± 0.86). Only 29% of EPAs provided a suggestion with 26.5% connecting that suggestion to a described behavior.
CONCLUSIONS: The comment component of EPA assessments offers essential context to facilitate learner progress and faculty guidance of learners. Yet comments are inconsistently provided, and when they are provided, 3/4s lack suggestions for improvement and a connection to the behavior described. Future directions include faculty development initiatives regarding high quality comment utility and EPA platform driven prompts of essential comment components.
DESIGN: Assessment comments from a nationwide digital medical education platform were evaluated by QuAL with scores ranging from 0-5 with 0 indicating very low quality, 3 an average comment and 5 a high quality comment. This is based on three components (1) sufficient evidence about performance scored from 0-3 (2) offering suggestion for improvement, scored as 0 or 1 and (3) connection of suggestion to a described behavior, scored as 0 or 1. We leveraged previous work developing a publicly available machine learning model to score comments with QuAL. Python was used to automate the extraction and processing of comment data through the machine learning model.
PARTICIPANTS: Faculty from surgical training programs across the United States submitted EPA assessment comments for surgical residents and fellows.
RESULTS: Of 12,841 EPA assessments, 7,211 (56%) contained a comment. Assessments without comment were excluded from analysis. The mean overall comment score was 2.91 representing average quality. The average evidence was 2.4/3 (± 0.86). Only 29% of EPAs provided a suggestion with 26.5% connecting that suggestion to a described behavior.
CONCLUSIONS: The comment component of EPA assessments offers essential context to facilitate learner progress and faculty guidance of learners. Yet comments are inconsistently provided, and when they are provided, 3/4s lack suggestions for improvement and a connection to the behavior described. Future directions include faculty development initiatives regarding high quality comment utility and EPA platform driven prompts of essential comment components.