Quality matters: Artificial intelligence-based assessment of feedback quality predicts technical skill improvement.

INTRODUCTION

Consistently delivering high-quality feedback is critical, yet challenging, in surgical education. We hypothesized that feedback quality, as assessed by artificial intelligence (AI, Teach1) based-tools, would correlate with improvements in technical skill acquisition during simulated paracentesis training.

METHODS

Seventy-five fourth-year medical students meeting inclusion criteria (baseline Objective Structured Assessment of Technical Skills scores ≤20) performed paracentesis on simulators, received instructor feedback via a digital platform, and completed a second attempt. Teach1, developed using OpenAI Vector Embeddings and Support Vector Machine classification (area under the receiver operating characteristic curve, 0.86-0.98), evaluated 688 feedback comments on the basis of 5 established quality criteria: direct observation, skill specificity, reinforcement of strengths, targeted improvement suggestions, and actionable plans. Changes in Objective Structured Assessment of Technical Skills (Δ Objective Structured Assessment of Technical Skills) scores were analyzed in relation to feedback quality percentages.

RESULTS

Objective Structured Assessment of Technical Skills scores improved from 15.76 (±3.15) to 21.68 (±2.30) postfeedback. Feedback quality percentages positively correlated with performance improvement (Δ Objective Structured Assessment of Technical Skills) (r = 0.309, P < .001), with each 10% increase in feedback quality percentages corresponding to a 1.16-point improvement in Objective Structured Assessment of Technical Skills scores (β = 11.61, P < .001). Reinforcement of strengths and suggesting targeted improvements showed the strongest correlations with skill gains. Artificial intelligence assessments demonstrated substantial agreement with expert evaluations (κ = 0.78).

CONCLUSION

Artificial intelligence-driven assessment of feedback quality significantly correlates with technical skill improvement in a simulated procedural setting. By enhancing feedback quality, Teach1 taps into one of the strongest drivers of skill development in education, offering a scalable method to standardize feedback quality across surgical training and potentially improve patient outcomes.