glass

A U G M E N T E D A N D V I RT U A L R E A L I T Y I N P L A S T I C S U R G E RY

Jonathan Kanevsky, MDMirko Gilardino, MDSamuel Lin, MD

O U T L I N E

DEVICE DEMO

REALITY SPECTRUM AR VR HARDWAREAR VR APPLICATIONS

Franz Robaud – 1812-Battle of Borodino

1838: Charles Wheatstone

Motronheilig.com

Motronheilig.com

S P E C T RU M O F R E A L I T Y

Motronheilig.com

W H AT I S M I XE D R E A L I T Y ?

MR devices combine input from multiple sensors in a wearable platform

An on-board computer integrates this information between the device, the Internet, and the outside world

The processed data is then relayed to the user through a visual display

W H AT I S V I RT U A L R E A L I T Y ?

W H AT I S V I RT U A L R E A L I T Y ?

W H AT I S A U G M E N TE D R E A L I T Y ?

• Spatial AR• Hand Held• Virtual Retina Display• Contact Lens• Head Mounted Display

W H AT I S A U G M E N TE D R E A L I T Y ? Spatial Augmented Reality

W H AT I S A U G M E N TE D R E A L I T Y ? Hand Held AR

W H AT I S A U G M E N TE D R E A L I T Y ? Contact lens AR

http://www.extremetech.com/computing/126043-us-military-developing-multi-focus-augmented-reality-contact-lenses

W H AT I S A U G M E N TE D R E A L I T Y ? Virtual Retina Display

http://www.extremetech.com/computing/126043-us-military-developing-multi-focus-augmented-reality-contact-lenses

W H AT I S A U G M E N TE D R E A L I T Y ?

www.dailymail.co.ukwww.dailymail.co.uk

Head Mounted Display

Google glass

Meta

H O LO L E N SMI C R OSOFT

H O LO L E N SMI C R OSOFT

A PPL I C AT I O N S I N PL A ST I C S U R G E RY

Voice activated head-mounted camera optimal for hands free recording form surgeon POV

Generate teaching libraries for plastic surgery eduction or record resident technique

S U R G I C A L E D U C AT I O N :

www.ucsf.eduGaster and Lin, 2014

PL A S T I C SU R G E RY A P PL I C AT I ON S

Augmented reality display for:

- Diagrams - Anatomical

markings- -AI applications

S U R G I C A L E D U C AT I O N :

Gaster , Lin, Kanevsky, Corban 2015

S U R G I C A L E D U CAT I O N :

Ma M, Fallavollita P, Seelbach I, et al. Personalized augmented reality for anatomy education. Clinical anatomy (New York, N.Y.).

2016;29(4):446-453

S U R G I C A L E D U CAT I O N :

S U R G I C A L E D U C AT I O N :

S U R G I C A L E D U C AT I O N :

Transmit visual information for diagnosis and surgical planning

Trainees can call a senior surgeon to discuss a diagnosis

Virtual assistance during procedures with other surgeons in HUD

T EL E M E D I C I N E :

internetmedicine.com

P L A S T I C S U R G E RY A P P L I C AT I O N S

http://1.bp.blogspot.com/-mVbtvRKhCJQ/UpKIR3hlw-I/AAAAAAAAHgk/sNRLYQ0n4SI/s1600/VIPAAR_Google_Glass_Virtual_Surgeon.jpg

TE L E M E D I C I N E :

DATA OV E R L AY A ND D O CU M E N TAT I O N :

internetmedicine.com

Accessing patient information: CT scans, MRIs, X-rays, patient notes, labs or vital signs all while remaining sterile in the operating room.

Generating new notes: OR dictations with embedded video during surgery

Peri-operative utility: Pre-operative markings, Post-operative photos

P L A S T I C S U R G E RY A P P L I C AT I O N S

C ON S U LTAT I O N :

http://www.illusioimaging.com/

P L A S T I C S U R G E RY A P P L I C AT I O N S

P E R I O - O P E RAT I V E :

Mezzana P, Scarinci F, Marabottini N. Augmented reality in oculoplastic surgery: first iPhone application. Plast Reconstr Surg.

2011;127(3):57e-58e.

P L A S T I C S U R G E RY A P P L I C AT I O N S

PAT I E NT C A R E :

internetmedicine.com

P L A S T I C S U R G E RY A P P L I C AT I O N S

PAT I E NT C A R E :

internetmedicine.com

Perioperative Pain and Anxiety

P L A S T I C S U R G E RY A P P L I C AT I O N S

G AT E C O N T R O L T H E O RY

L I M I TAT I O N S

Two broad areas for which AR devices can be used to interface with the EMR:

• HIPAA compliance

• Distraction of Surgeon – Delayed response time an decreased awareness

• Cost effective innovation

TH E N E XT S T E P

Two broad areas for which AR devices can be used to interface with the EMR:

Adoption by the surgical community is a major hurdleIntegration with artificial intelligence and big data

Tracking analysis

3D virtual surgical planning software (i.e. oculus rift )

www.pcworld.com

A R D E V I C E N E E DSIntegrate patient data in visual field pre, intra and post-op

Enhance consultations in real-time

Make surgical education more visual and hands- on

Synthesize images and video into medical documen-tation

"Of course nobody knows anything. You just getwhat information you can through the windows of your five senses, and then make your guesses.

When they're wrong, you pay the penalty."

R E F E R E N C E S

Two broad areas for which AR devices can be used to interface with the EMR:

1. Armstrong, D. G., Rankin, T. M., Giovinco, N. A., Mills, J. L. & Matsuoka, Y. A Heads-Up Display for Diabetic Limb Salvage Surgery A View Through the Google Looking Glass. J. Diabetes Sci. Technol. 8, 951–956 (2014).2. Berger, A. J., Gaster, R. S. & Lee, G. K. Development of an Affordable System for Personalized Video-Documented Surgical Skill Analysis for Surgical Residency Training. Ann. Plast. Surg. (2013). doi:10.1097/SAP.0b013e31827e513c3. Waxman, B. P. Medicine in small doses. ANZ J. Surg. 84, 907–907 (2014).4. Assad-Kottner, C., Hakeem, A., Fontenot, E. & Uretsky, B. F. ‘Tele-mentoring’: an interventional procedure using a wearable computer: first-in-man. J. Am. Coll. Cardiol. 63, 1022 (2014).5. Carenzo, L. et al. Disaster medicine through Google Glass. Eur. J. Emerg. Med. Off. J. Eur. Soc. Emerg. Med. (2014). doi:10.1097/MEJ.00000000000002296. Schreinemacher, M. H., Graafland, M. & Schijven, M. P. Google glass in surgery. Surg. Innov. 21, 651–652 (2014).7. Lindeque, B. G. P. et al. Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Orthopedics 37, 751–757 (2014).8. Jeroudi, O. M. et al. Accuracy of Remote Electrocardiogram Interpretation With the Use of Google Glass Technology. Am. J. Cardiol. 0,9. Shakil, I., Tran, P. & Dehy, R. System and method for augmenting healthcare-provider performance. (2014). at <http://www.google.com/patents/US20140222526>10. Emergency providers see big potential for Google Glass. ED Manag. Mon. Update Emerg. Dep. Manag. 26, 55–58 (2014)