AI Teaches Surgical Diagnostic Reasoning to Medical Students: Evidence from an Experiment Using a Fully Automated, Low-Cost Feedback System.

OBJECTIVE

While AI-generated feedback has shown promise in medical education, prior studies have only used AI for feedback, with question design handled by human experts, and the process required human involvement. This study aimed to evaluate the effectiveness of a fully automated AI-based system that generates both multiple-choice questions (MCQs) and personalized feedback, without any human input, on improving diagnostic reasoning in preclinical medical students.

DESIGN

A prospective, parallel-group, interventional study. The intervention group (Year-1 students) received AI-generated MCQs and feedback over 5 days using a web platform, coded via "vibe coding," with spaced repetition. The diagnoses covered included 5 abdominal pain conditions: acute appendicitis, acute cholecystitis, acute pancreatitis, acute gastroenteritis, and nephrolithiasis. Diagnostic performance was assessed via an Objective Structured Video Examination (OSVE), immediately and 2 weeks postintervention. The control group (Year-2 students) completed the OSVE once.

SETTING

Gazi University Faculty of Medicine, Ankara, Turkiye; institutional academic setting focused on undergraduate medical education.

PARTICIPANTS

Thirty-eight Year-1 medical students completed the intervention. Thirty-three Year-2 students served as a non-randomized control group. All intervention participants completed the immediate assessment; 30 completed the delayed assessment.

RESULTS

Intervention participants outperformed the control group in diagnosing the 5 abdominal pain conditions immediately after the intervention (p < 0.001) and at the 2-week follow-up (p < 0.001). Postintervention expert review confirmed the accuracy of all AI-generated questions and identified minimal issues in 0.6% of feedback statements. Total AI cost was $0.51.

CONCLUSIONS

A fully automated, low-cost AI system without human in the loop during content generation can significantly enhance illness scripts in preclinical medical students. Early engagement with such tools may help students strengthen surgical diagnostic skills and derive greater benefit from clinical environments. This kind of tools may transform how clinical reasoning is taught in resource-limited or high-volume educational settings.