Projected Augmented Reality in Surgery: History, Validation, and Future Applications.

Background/

Objectives

Projected augmented reality (PAR) enables real-time projection of digital surgical information directly onto the operative field. This offers a hands-free, headset-free platform that is universally visible to all members of the surgical team. Compared to head-mounted display systems, which are limited by restricted fields of view, ergonomic challenges, and user exclusivity, PAR provides a more intuitive and collaborative surgical interface. When paired with artificial intelligence (AI), PAR has the potential to automate aspects of surgical planning and deliver high-precision guidance in both high-resource and global health settings. Our team is working on the development and validation of a PAR platform to dynamically project surgical and anatomic markings directly onto the patients intraoperatively.

Methods

We developed a PAR system using a structured light scanner and depth camera to generate digital 3D surface reconstructions of a patient's anatomy. Surgical markings were then made digitally, and a projector was used to precisely project these points directly onto the patient's skin. We also developed a trained machine learning model that detects cleft lip landmarks and automatically designs surgical markings, with the plan to integrate this into our PAR system.

Results

The PAR system accurately projected surgeon and AI-generated surgical markings onto anatomical models with sub-millimeter precision. Projections remained aligned during movement and were clearly visible to the entire surgical team without requiring wearable hardware.

Conclusions

PAR integrated with AI provides accurate, real-time, and shared intraoperative guidance. This platform improves surgical precision and has broad potential for remote mentorship and global surgical training.