Augmented reality for inner ear procedures: visualization of the cochlear central axis in microscopic videos

Purpose Visualization of the cochlea is impossible due to the delicate and intricate ear anatomy. Augmented reality may be used to perform auditory nerve implantation by transmodiolar approach in patients with profound hearing loss. Methods We present an augmented reality system for the visualization of the cochlear axis in surgical videos. The system starts with an automatic anatomical landmark detection in preoperative computed tomography images based on deep reinforcement learning. These landmarks are used to register the preoperative geometry with the real-timemicroscopic video captured inside the auditory canal. Three-dimensional pose of the cochlear axis is determined using the registration projection matrices. In addition, the patient microscope movements are tracked using an image feature-based tracking process. Results The landmark detection stage yielded an average localization error of 2.18 +/- 1.44mm(n = 8). The target registration error was 0.31 +/- 0.10mm for the cochlear apex and 15.10 +/- 1.28. for the cochlear axis. Conclusion We developed an augmented reality system to visualize the cochlear axis in intraoperative videos. The system yielded millimetric accuracy and remained stable throughout the experimental study despite camera movements throughout the procedure in experimental conditions.