Research  Award  Brief  

For  more  information,  please  contact  Dr.  Nassir  Navab  (camp@jhu.edu).  

 

Magic  Mirror:  A  Novel  Human  Anatomy  Education  Environment  with  Augmented  Reality  Technology  (2016  -­‐  2018)  

PI:  Nassir  Navab,  Ph.D.  Research  Professor    Department  of  Computer  Science  Whiting  School  of  Engineering  

Co-­‐Investigators:  

Gregory  D.  Hager,  Ph.D.    Professor    Department  of  Computer  Science  Whiting  School  of  Engineering  

Greg  M.  Osgood,  M.D.  Chief  Orthopaedic  Trauma  Department  of        Orthopaedic  Surgery  School  of  Medicine  

Roghayeh  Barmaki,  Ph.D.  Postdoctoral  Researcher  Dept.  of  Computer  Science  Whiting  School  of  Engineering  

Research  Question:    Can  augmented  reality  help  medical  students  learn  human  anatomy?    Interdisciplinary  Approach:  This  project  bridges  educational  theory,  augmented  reality  novel  gamification  techniques,  and  multimodal  machine  learning  to  develop  and  refine  a  new  augmented  reality  learning  tool  for  human  anatomy  education.    Potential  Implications  of  Research:  This  project  will  produce  a  new  personalized  educational  tool  for  pre-­‐medical  and  medical  students  learning  human  anatomy  and  inform  the  development  of  similar  augmented  reality  learning  tools  in  other  domains.    

Learning  human  anatomy   is   critical   to  medical   students,  but   the   learning  process   itself   can  be   challenging.  Students   spend   countless   hours   participating   in   traditional   cadaver   labs   and   studying   physical   anatomy   models,  atlases  of  anatomy,  and  digitalized  image  banks.  Such  traditional  education  practices  have  changed  relatively  little  in  the  last  few  decades,  and  while  these  practices  have  advantages,  they  also  have  significant  drawbacks.  The  cadaver-­‐based   learning   approach   has   seen   a   decline   due   to  practical,  ethical,  and  cost  issues.  As  a  result,  anatomical  education   has   relied   more   on   physical,   diagram,   and  image   models.   The   model   approach   introduces  additional   spatial   learning   challenges   for   students,  including   difficulties   in   (1)   translating   two-­‐dimensional  static   images,   diagrams,   or   photographs   in   medical  illustrations  to  three-­‐dimensional  human  bodies  and  (2)  visualizing  dynamic  processes   in  a   living  organism,   such  as   the   activity   of   working   muscles,   from   inanimate  physical  models.    

In  this  project,  we  will  develop  and  refine  a  new  interactive,   augmented   reality   (AR)   learning   tool   for  anatomy  education.  The  tool  combines  the  Magic  Mirror  system   and   Balaur   Display   Wall   to   produce   “in-­‐situ”  visualizations  of  medical   information  directly  on  top  of  the  student’s  own  body.  A  gesture-­‐based  user  interface  (UI)  allows  the  student  to  directly  interact  with  medical  illustrations.    As  seen  in  Figure  1,  a  student  standing  in  front  of  a  large  display  screen  sees  an   image  of  the  small   intestine  projected  onto  his  own  torso  (organ   is  outlined   in  yellow)  and  a  more  detailed  cross-­‐section  of  the  organ  is  displayed  in  the  right  top  corner.    The  student  can  select  an  icon  on  the  right  side  of  the  screen  via  hand  gestures  to   learn  more  about  the  organ’s   functions  or  view  other  organs.  The  tool  will  create  a  personalized,  self-­‐paced,  immersive  learning  environment  for  students.  

Undergraduate  and  graduate  students  enrolled  in  anatomy  courses  will  participate  in  a  series  of  educational  sessions   with   the   AR   learning   tool.   We   will   assess   students'   learning   by   examining   changes   in   their   anatomy  knowledge  over  time  using  knowledge  assessments  and  course  grades.  We  will  examine  the  usability  of  the  tool  (e.g.,  ease,  functionality)  by  exploring  full-­‐body  tracking  data,  system  event  logs  (i.e.,  learners'  interactions  in  the  system),  and  time  required  to  complete  assigned  tasks  in  the  system  relative  to  their  learning  outcomes.  Multimodal  machine  learning  methodologies  will  be  applied  to  analyze  the  data.  This  project  will  produce  a  new  personalized  educational  tool  for  students  learning  human  anatomy  and  will  be  the  basis  for  future  educational  intervention  work  in  this  area.  Moreover,   this  work  has  potential   implications   for   the  development  of   similar  AR   learning   tools   in  other  domains,  such  as  rehabilitation  exercise,  and  for  other  student  populations  (e.g.,  high  school).    

Figure   1.   An   augmented   reality   learning   tool   for   medical   students  studying  human  anatomy.