Toward comprehensive real-time scene understanding in ophthalmic surgery through multimodal image fusion.

PURPOSE: The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery.

METHODS

We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a Yolo-NAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence.

RESULTS

Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1 mm), the distance estimation accuracy improved from 284 μ m (OPMI only) to 33 μ m (multimodal).

CONCLUSION

Feature fusion of multimodal imaging can enhance multitask prediction accuracy compared to single-modality processing, and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multimodal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.