A Survey of Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

April 21st 2014

mark.billinghurst@hitlabnz.org

Highlights from:

M. Billinghurst, A. Clark, and G. Lee. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction. Vol. 8, No. 1 (2015) 1–202, 2015

http://www.nowpublishers.com/article/Details/HCI-049

Introduction

1977 – Star Wars

Augmented Reality Definition !  Defining Characteristics [Azuma 97]

! Combines Real and Virtual Images -  Both can be seen at the same time

!  Interactive in real-time - The virtual content can be interacted with

! Registered in 3D - Virtual objects appear fixed in space

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

2008 - CNN

!  Put AR pictures here

Augmented Reality Examples

Virtual Reality

!  Immersive VR ! Head mounted display, gloves !  Separation from the real world

AR vs VR

Making Interfaces Invisible

Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.

From Reality to Virtual Reality

Ubiquitous Computing Augmented Reality Virtual Reality

Milgram’s Reality-Virtuality continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

Metaverse !  Neal Stephenson’s “SnowCrash” !  The Metaverse is the convergence of:

!  1) virtually enhanced physical reality !  2) physically persistent virtual space

!  Metaverse Roadmap !  http://metaverseroadmap.org/

Summary !  Augmented Reality has three key features

! Combines Real and Virtual Images !  Interactive in real-time !  Registered in 3D

!  AR can be classified alongside other technologies !  Invisible Interfaces ! Milgram’s Mixed Reality continuum !  Stephenson’s MetaVerse

History

Pepper’s Ghost (1862)

!  Dates back to Giambattista della Porta (1584)

Early HMDs and HUDs (1960’s)

Sutherland / Sproull’s see-through HMD (1965)

Bucaneer HUD (1958)

AR History !  1960’s – 80’s: Early Experimentation

•  Military, Academic labs

!  1980’s – 90’s: Basic Research •  Tracking, Displays

!  1995 – 2005: Tools/Applications •  Interaction, Usability, Theory

!  2005 - : Commercial Applications •  Games, Medical, Industry, Mobile

Important Milestones !  1992: T. Caudell coined term “AR” (Boeing) !  1996: First collaborative AR systems !  1999: ARToolKit released – tracking library !  1997: Feiner First outdoor AR system !  1997 – 2001: Mixed Reality Systems Lab (Japan) !  1999 – 2003: ARVIKA project (Germany) !  2002: ISMAR conference started

AR Enters Mainstream (2007 -)

!  Magazines !  MIT Tech. Review (Mar 2007)

-  10 most exciting technologies !  Economist (Dec. 2007)

-  Reality, only better !  Games

!  Sony “Eye of Judgement” -  300,000+ units shipped

!  Broadcast TV !  Sports broadcasting

Google Searches for AR

Browser Based AR (2008 - ) !  Flash + Camera + 3D graphics !  High impact

!  High marketing value

!  Large potential install base !  1.6 Billion web users

!  Ease of development !  Lots of developers, mature tools

!  Low cost of entry !  Browser, web camera

Mobile AR (2005 - ) ! Mobile Phones

! Camera, processor, display ! Computer vision based AR

!  Advertising ! HIT Lab NZ (2007) !  AR print advertisement !  Txt to download app

Mobile Outdoor AR (2009 - ) !  Mobile phones with GPS !  Tag real world locations

!  GPS + Compass input !  Overlay graphics data on live video

!  Applications !  Travel guide, Advertising, etc

!  Wikitude, Layar, Junaio, etc.. !  Android/iOS based, Public API released

Motorola Droid

AR Today ! Key Technologies Available

- Robust tracking (Computer Vision, GPS/sensors) - Display (Handheld, HMDs) -  Input Devices (Kinect, etc) - Developer tools (Qualcomm, Metaio, Wikitude)

! Commercial Business Growing - Gaming, GPS/Mobile, Online Advertisement

•  >$5 Billion USD by 2016 (Markets andMarkets) •  >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

AR Business Today

!  Marketing ! Web-based, mobile

!  Mobile AR ! Geo-located information and service

!  Gaming ! Mobile, Physical input (Kinect, PS Move)

!  Upcoming areas ! Manufacturing, Medical, Military

Summary !  Augmented Reality has a long history going

back to the 1960’s !  Interest in AR has exploded over the last few

years and is being commercialized quickly !  AR is growing in a number of areas

! Mobile AR ! Web based AR ! Marketing experiences

Technology

Augmented Reality Definition !  Defining Characteristics

! Combines Real and Virtual Images - Display Technology

!  Interactive in real-time -  Interaction Technology

! Registered in 3D - Tracking Technology

Display

Display Technologies

" Types (Bimber/Raskar 2003) " Head attached

•  Head mounted display/projector " Body attached

•  Handheld display/projector " Spatial

•  Spatially aligned projector/monitor

Display Taxonomy

Types of Head Mounted Displays

Occluded See-thru

Multiplexed

Optical see-through HMD

Virtual images from monitors

Real World

Optical Combiners

Epson Moverio BT-200

▪  Stereo see-through display ($700) ▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪  Android Powered, separate controller ▪  VGA camera, GPS, gyro, accelerometer

Strengths of optical see-through !  Simpler (cheaper) !  Direct view of real world

!  Full resolution, no time delay (for real world) !  Safety !  Lower distortion

!  No eye displacement !  see directly through display

Video see-through HMD Video cameras

Monitors

Graphics

Combiner

Video

Vuzix Wrap 1200DXAR

▪  Stereo video see-through display ($1500) ■ Twin 852 x 480 LCD displays, 35 deg. FOV ■ Stereo VGA cameras ■ 3 DOF head tracking

Strengths of Video See-Through !  True occlusion

!  Block image of real world !  Digitized image of real world

!  Flexibility in composition ! Matchable time delays ! More registration, calibration strategies

!  Wide FOV is easier to support ! wide FOV camera

Multiplexed Displays !  Above or below line of sight !  Strengths

!  User has unobstructed view of real world !  Simple optics/cheap

!  Weaknesses !  Direct information overlay difficult

•  Display/camera offset from eyeline !  Wide FOV difficult

Google Glass

▪  Monocular see-through multiplexed display ▪  640 x 360 microprojector, 15 degree FOV ▪  5 MP camera, gyro, accelerometer

Display Technology !  Curved Mirror

!  off-axis projection !  curved mirrors in front of eye !  high distortion, small eye-box

!  Waveguide !  use internal reflection !  unobstructed view of world !  large eye-box

See-through thin displays

!  Waveguide techniques for thin see-through displays !  Wider FOV, enable AR applications !  Social acceptability

Opinvent Ora

Lumus DK40

Spatial Augmented Reality

!  Project onto irregular surfaces !  Geometric Registration !  Projector blending, High dynamic range

!  Book: Bimber, Rasker “Spatial Augmented Reality”

Projector-based AR

Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo

Projector

Real objects with retroreflective covering

User (possibly head-tracked)

Example of projector-based AR

Ramesh Raskar (MIT Media Lab)

Head Mounted Projector

!  NVIS P-50 HMPD !  1280x1024/eye !  Stereoscopic !  50 degree FOV !  www.nvis.com

HMD vs. HMPD

Head Mounted Display Head Mounted Projected Display

CastAR - http://technicalillusions.com/

!  Stereo head worn projectors !  Interactive wand !  Rollable retro-reflective sheet

!  Designed for shared interaction

Video Monitor AR

Video cameras Monitor

Graphics Combiner

Video

Stereo glasses

Examples

Handheld Displays !  Mobile Phones

! Camera ! Display !  Input

Tracking

Objects Registered in 3D

!  Registration !  Positioning virtual object wrt real world

!  Tracking ! Continually locating the users viewpoint

-  Position (x,y,z), Orientation (r,p,y)

Tracking Technologies !  Active

•  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location

!  Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision

•  Marker based, Natural feature tracking

!  Hybrid Tracking •  Combined sensors (eg Vision + Inertial)

Tracking Types

Magnetic Tracker

Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

Mechanical Tracker

Example: Marker tracking !  Available for more than 10 years !  Several open source solutions exist

!  ARToolKit, ARTag, ATK+, etc

!  Fairly simple to implement !  Standard computer vision methods

!  A rectangle provides 4 corner points !  Enough for pose estimation!

Marker Based Tracking: ARToolKit

http://artoolkit.sourceforge.net/

Coordinate Systems

Tracking challenges in ARToolKit

False positives and inter-marker confusion (image by M. Fiala)

Image noise (e.g. poor lens, block coding /

compression, neon tube)

Unfocused camera, motion blur

Dark/unevenly lit scene, vignetting

Jittering (Photoshop illustration)

Occlusion (image by M. Fiala)

Markerless Tracking

Magnetic Tracker Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

!  No more Markers! #Markerless Tracking

Mechanical Tracker

Natural Feature Tracking

!  Use Natural Cues of Real Elements !  Edges !  Surface Texture !  Interest Points

!  Model or Model-Free !  No visual pollution

Contours

Features Points

Surfaces

Texture Tracking

Edge Based Tracking !  RAPiD [Drummond et al. 02]

!  Initialization, Control Points, Pose Prediction (Global Method)

Line Based Tracking !  Visual Servoing [Comport et al. 2004]

Model Based Tracking !  OpenTL - www.opentl.org

!  General purpose library for model based visual tracking

Marker vs. natural feature tracking !  Marker tracking

!  ++ Markers can be an eye-catcher !  ++ Tracking is less demanding !  -- The environment must be instrumented with markers !  -- Markers usually work only when fully in view

!  Natural feature tracking !  -- A database of keypoints must be stored/downloaded !  ++ Natural feature targets might catch the attention less !  ++ Natural feature targets are potentially everywhere !  ++ Natural feature targets work also if partially in view

Example: Outdoor Hybrid Tracking

!  Combines !  computer vision

-  natural feature tracking !  inertial gyroscope sensors

!  Both correct for each other !  Inertial gyro - provides frame to frame

prediction of camera orientation !  Computer vision - correct for gyro drift

Robust Outdoor Tracking

!  Hybrid Tracking !  Computer Vision, GPS, inertial

!  Going Out !  Reitmayr & Drummond (Univ. Cambridge)

Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.

Handheld Display

Interaction

!  Interface Components ! Physical components ! Display elements

- Visual/audio !  Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Interface Elements

AR Design Space

Reality Virtual Reality

Augmented Reality

Physical Design Virtual Design

Interface Design Path

1/ Prototype Demonstration

2/ Adoption of Interaction Techniques from other interface metaphors

3/ Development of new interface metaphors appropriate to the medium

4/ Development of formal theoretical models for predicting and modeling user actions

Desktop WIMP

Virtual Reality

Augmented Reality

Interaction Development !  Information Browsing

! Camera movement !  Limited interaction

!  3D AR Interaction ! HMD, hand tracking !  3D UI/VR techniques !  Specialized input devices

Tangible User Interfaces (Ishii 97)

!  Augmented Surfaces !  Rekimoto 1998 ! Multiple projection surfaces !  Tangible prop interaction

!  i/O Brush (2004) !  Ryokai, Marti, Ishii !  Sensor enhanced real brush

Other Examples !  Triangles (Gorbert 1998)

!  Triangular based story telling

!  ActiveCube (Kitamura 2000-) ! Cubes with sensors

Lessons from Tangible Interfaces !  Benefits

!  Physical objects make us smart (affordances) ! Objects aid collaboration ! Objects increase understanding (cognitive artifacts)

!  Limitations ! Difficult to change object properties !  Limited display capabilities (2D surface) !  Separation between object and display

Orthogonal Nature of AR Interfaces

Tangible AR Interaction

!  AR overcomes limitation of TUIs !  enhance display possibilities ! merge task/display space !  provide public and private views

!  TUI + AR = Tangible AR !  Apply TUI methods to AR interface design

Tangible AR Design Principles !  Tangible AR Interfaces use TUI principles

!  Physical controllers for moving virtual content !  Support for spatial 3D interaction techniques !  Support for multi-handed interaction ! Match object affordances to task requirements !  Support parallel activity with multiple objects !  Allow collaboration between multiple users

VOMAR - Tangible AR Interface !  Use of natural physical object

to control virtual objects

!  Physical objects ! Catalog book:

-  Turn over the page

!  Paddle operation: -  Push, shake, incline, hit, scoop

Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object manipulation on a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium on (pp. 111-119). Ieee.

Wrap-up !  Browsing Interfaces

!  simple (conceptually!), unobtrusive

!  3D AR Interfaces !  expressive, creative, require attention

!  Tangible Interfaces !  Embedded into conventional environments

!  Tangible AR ! Combines TUI input + AR display

Typical Applications

!  Web based AR !  Flash, HTML 5 based AR !  Marketing, education

!  Outdoor Mobile AR !  GPS, compass tracking !  Viewing Points of Interest in real world !  Eg: Junaio, Layar, Wikitude

!  Handheld AR !  Vision based tracking !  Marketing, gaming

!  Location Based Experiences !  HMD, fixed screens !  Museums, point of sale, advertising

Typical AR Experiences

CityViewAR Application

!  Visualize Christchurch before the earthquakes

User Experience

!  Multiple Views

! Map View, AR View, List View

!  Multiple Data Types !  2D images, 3D content, text, panoramas

Demo: Warp Runner

!  Puzzle solving game !  Deform real world terrain

Demo: colAR

!  Turn colouring books pages into AR scenes ! Markerless tracking, use your own colours..

!  Try it yourself: http://www.colARapp.com/

What Makes a Good AR Experience? !  Compelling

!  Engaging, ‘Magic’ moment

!  Intuitive, ease of use ! Uses existing skills

!  Anchored in physical world !  Seamless combination of real and digital

Research Opportunities

Previous Research Themes !  Zhou et. al. (2008) analyzed all ISMAR papers

Zhou, F., Duh, H. B. L., & Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings ISMAR 2008, (pp. 193-202). IEEE Computer Society.

Possible Research Directions !  Tracking

! Wide area, Reliable indoor, Ubiquitous tracking !  Interaction

!  Intelligent systems, Gesture, Collaborative systems

!  Displays ! Wide FOV, Retinal Scanning, Contact Lens

!  Social Acceptance ! Wearable AR, Handheld AR, Social Interactions

Wide Area Tracking

!  Process !  Combine panorama’s into point cloud model (offline) !  Initialize camera tracking from point cloud !  Update pose by aligning camera image to point cloud !  Accurate to 25 cm, 0.5 degree over wide area

Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking.In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.

Gesture Based Interaction

!  Use free hand gestures to interact !  Depth camera, scene capture

!  Multimodal input !  Combining speech and gesture

HIT Lab NZ Microsoft Hololens

Meta SpaceGlasses

Wide FOV Displays

!  Wide FOV see-through display for AR !  LCD panel + edge light point light sources !  110 degree FOV

Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014 Emerging Technologies (p. 20). ACM.

Social Acceptance

!  People don’t want to look silly !  Only 12% of 4,600 adults would be willing to wear AR glasses !  20% of mobile AR browser users experience social issues

!  Acceptance more due to Social than Technical issues !  Needs further study (ethnographic, field tests, longitudinal)

Conclusions

Conclusion !  AR seamlessly blends real and virtual imagery

!  Interactive in real time, fixed in space

!  AR has developed into a mass market technology !  Education, engineering, entertainment

!  The technologies to create AR are available ! Display, tracking, interaction

!  There are opportunities for significant research

Gartner Hype Cycle (2014)

Market Forecast

Up to $120B by 2020 – 5 x VR market

User Forecast

More Information

•  Mark Billinghurst –  Email: mark.billinghurst@hitlabnz.org

– Twitter: @marknb00

•  Website –  http://www.hitlabnz.org/