Evaluation of Vision-Language Models for Surgical Narration and Critical View of Safety Assessment in Laparoscopic Cholecystectomy.

BACKGROUND

Vision-language models (VLMs) may support surgical documentation through operative video narration, but fluent narration may not capture procedural details or safety-critical anatomy. We compared VLM-generated laparoscopic cholecystectomy narrations with surgeon narration and evaluated Critical View of Safety (CVoS) assessment.

STUDY

DESIGN

Forty-eight 10-minute segments from 13 Cholec80 videos were evaluated using Gemini 2.5 Flash (Untrained VLM) and Qwen3-VL-CoA (Surgically Trained VLM) under Basic and procedure-specific prompting. VLM and surgeon narrations were compared for structure, semantic similarity, checklist-based procedural content, and CVoS recognition.

RESULTS

Attending narrations averaged 490.6 words versus 243.9-252.5 for VLMs. Specific prompting increased cosine similarity for both VLMs (0.507 to 0.593 and 0.543 to 0.584; both adjusted p<0.001), but checklist F1 improved only for the Untrained VLM (0.405 to 0.501; adjusted p=0.006), not the Surgically Trained VLM (0.337 to 0.380; adjusted p=0.233). Among 6 CVoS-positive and 42 CVoS-negative segments, Untrained Specific had sensitivity 0.33 (95% CI 0.10-0.70) and specificity 0.95 (95% CI 0.84-0.99); Surgically Trained Specific had sensitivity 0.83 (95% CI 0.44-0.97) and specificity 0.19 (95% CI 0.10-0.33).

CONCLUSIONS

Procedure-specific prompting increased similarity to attending narration without consistently improving procedural content. Divergent CVoS errors demonstrate that fluent surgical narration does not establish reliable safety assessment. VLM narration may support surgeon-reviewed documentation, but safety-critical judgments require separate task-specific validation.