November 21, 2019

Mixed reality displays are entering the operating room, first as surgical planning tools and soon as real-time guides to help doctors zap tumors.

Imagine maneuvering your car through a dark tunnel that bulges unexpectedly in places and then turns sharply through a maze-like passage. The perilous journey feels safer with a light and camera showing the way ahead. It’s even better if digital lines lay out a track, assuring you stay in your lane.

In a rudimentary way, that scenario illustrates the advantage mixed reality (or augmented reality) is bringing to surgery, starting with the delicate pathways of the sinus. “When you’re doing minimally invasive surgery in a narrow corridor, you want to stay in the corridor, do whatever needs to be done, and not disturb anything around you,” says Martin J. Citardi, a rhinologist at the University of Texas Health Science Center at Houston, who in 2018 became the first in the United States to perform sinus surgery using a blended view of real and digital imagery.

Digital markings superimposed on real-world camera feeds have become commonplace in everyday life, as in the rear-view cameras on cars and the first-down lines in televised football games, but it’s a new phenomenon in surgery. In 2017, the guidance system Citardi used became the first to gain approval from the US Food and Drug Administration for sinus surgery. Two different surgical planning apps designed for HoloLens, Microsoft’s mixed reality headset, have since also garnered FDA approval. Coming soon are the shimmering holograms of sci-fi imagination in surgery itself: A clinical trial is now evaluating whether HoloLens headsets help surgeons and radiologists zap tumors.

These emerging tools are “a dramatic step forward,” says Jenny Chen, a neuroradiologist and founder of 3D Heals, which promotes innovation in medical 3D printing. “The next step is to see how companies fine-tune these products to be more user-friendly and clinically more effective.”

The HoloLens headset, whose second iteration is due for release by the end of the year, is finding a niche among surgeons and biomedical engineers. “When you put on the HoloLens, it’s like having X-ray vision,” says Karl West, an inventor and biomedical engineer at The Cleveland Clinic who developed a HoloLens app that is licensed to the Cleveland-based startup MediView. “You can see the anatomy inside the patient.” There are no actual X-rays; the images come from pre-operative CT scans or MRIs, projected holographically through a head-mounted display and coupled with ultrasound and electromagnetic tracking devices. […read more]