Automated triage of non-informative frames in minimally invasive surgical videos: a comparative study of domain-specific vision transformers versus general-purpose multimodal models.

BACKGROUND

Minimally invasive surgery (MIS) videos support education and quality review, but raw recordings contain non-informative frames that impede review and increase storage.

METHODS

We retrospectively analyzed 300,832 frames from 120 MIS videos performed by 30 surgeons. An EndoViT-initialized Vision Transformer (ViT) was evaluated using leave-six-surgeons-out cross-validation and compared with conventional vision models. GPT-4o-mini, LLaVA, and Doubao were separately evaluated under zero- and few-shot prompting on a balanced 400-frame subset that was not sample-size matched to the ViT evaluation. We also explored the association between surgeon experience and non-informative-frame ratio.

RESULTS

In the within-dataset comparison of trained vision models, the EndoViT-initialized ViT achieved an aggregate accuracy and F1-score of 0.942. In an exploratory few-shot analysis within a separate 400-frame balanced benchmark, GPT-4o-mini achieved an accuracy of 0.890 and F1-score of 0.880; exemplar-test disjointness could not be confirmed. Because the ViT and LMM evaluations used different test sets and sample sizes, these estimates do not support a direct comparison of model superiority. The surgeon-level analysis showed no statistically significant association between experience and the non-informative-frame ratio (r = - 0.421, 95% CI - 0.83 to 0.28; p = 0.226).

CONCLUSION

The EndoViT-initialized ViT performed well in this single-institution dataset, whereas prompt-only LMM performance varied in a separate small benchmark. Matched evaluation with documented exemplar-test separation and external validation is required before determining relative performance or clinical readiness. Automated frame triage may support MIS video management with human oversight.