COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

July 23rd 2014

Lecture 2: AR Technology

mark.billinghurst@hitlabnz.org

Recap

Augmented Reality Definition   Defining Characteristics [Azuma 97]

 Combines Real and Virtual Images -  Both can be seen at the same time

  Interactive in real-time -  Virtual content can be interacted with

  Registered in 3D -  Virtual objects appear fixed in space

What is not Augmented Reality?

  Location-based services   Barcode detection (QR-codes)   Augmenting still images   Special effects in movies   …   … but they can be combined with AR!

AR vs VR   Virtual Reality: Replaces Reality

  Scene Generation: requires realistic images  Display Device: fully immersive, wide FOV   Tracking and Sensing: low accuracy is okay

  Augmented Reality: Enhances Reality   Scene Generation: minimal rendering okay  Display Device: non-immersive, small FOV   Tracking and Sensing: high accuracy needed

Milgram’s Reality-Virtuality continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

A Brief History of AR (1)   1960’s: Sutherland / Sproull’s

first HMD system was see-through

History Summary   1960’s – 80’s: Early Experimentation   1980’s – 90’s: Basic Research

  Tracking, displays

  1995 – 2005: Tools/Applications   Interaction, usability, theory

  2005 - : Commercial Applications  Games, Medical, Industry

Google Searches for AR

2008 - Browser Based AR   Flash + camera + 3D graphics   High impact

  High marketing value

  Large potential install base   1.6 Billion web users

  Ease of development   Lots of developers, mature tools

  Low cost of entry   Browser, web camera

2005 - Mobile Phone AR  Mobile Phones

  camera   processor   display

  AR on Mobile Phones   Simple graphics  Optimized computer vision  Collaborative Interaction

2009 - Outdoor Information Overlay   Mobile phone based   Tag real world locations

  GPS + Compass input   Overlay graphics data on live video

  Applications   Travel guide, Advertising, etc

  Wikitude, Layar, Junaio, etc..   Android based, Public API released

AR Today  Key Technologies Available

- Robust tracking (Computer Vision, GPS/sensors) - Display (Handheld, HMDs) -  Input Devices (Kinect, etc) - Developer tools (Qualcomm, Metaio, ARTW)

 Commercial Business Growing - Gaming, GPS/Mobile, Online Advertisement

•  >$5 Billion USD by 2016 (Markets andMarkets) •  >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

Sample AR Applications

Applications

  Medicine   Manufacturing   Information overlay   Architecture   Museum   Marketing   Gaming

Applications: medical   “X-ray vision” for surgeons   Aid visualization, minimally-invasive operations.

Training. MRI, CT data.  Ultrasound project, UNC Chapel Hill.

Courtesy UNC Chapel Hill

Medical AR Trials   Sauer et al. 2000 at Siemens

Corporate Research, NJ   Stereo video see through

F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial, MICCAI 2002

Assembly and maintenance

© 1993 S. Feiner, B. MacIntyre, & D. Seligmann, Columbia University

© 1996 S. Feiner, B. MacIntyre, & A. Webster, Columbia University

PS3 - Eye of Judgment (2007)   Computer Vision Tracking   Card based battle game   Collaborative AR   October 24th 2007

AR Books – Markerless Tracking

AR Annotations

Columbia University

HRL

© 1993 S. Feiner, B. MacIntyre, M. Haupt, & E. Solomon, Columbia University

© 1997 S. Feiner, B. MacIntyre, T. Höllerer, & A. Webster, Columbia University

Broadcast TV

Interactive Museum Experiences   BlackMagic

  Virtual America’s Cup   410,000 people in six months

  MagicPlanet   TeManawa science museum   Virtual Astronomy   Collaborative AR experience

  AR Volcano   Interactive AR kiosk   Scienceworks museum, Melbourne

Digital Binocular Station

http://www.DigitalBinocularStation.com/

Museum Archeology   LifePlus (2002-2004)

 Natural feature tracking   Virtual characters  Mobile AR system

  Archeoguide (2000-2002)  Cultural heritage on-site guide  Hybrid tracking   Virtual overlay

Sales and Marketing   Connect with brands and branded objects   Location Based Experiences

  Lynx Angels

  Web based   Rayban glasses

  Mobile   Ford Ka campaign

  Print based   Red Bull Magazine

Summary   AR technology can be used to develop a wide

range of applications   Promising application areas include

 Games   Education   Engineering  Medicine  Museums   Etc..

AR Experience Design

“The product is no longer the basis of value. The

experience is.”

Venkat Ramaswamy The Future of Competition.

Experience Economy

experiences

services

products

components

Valu

e

Sony CSL © 2004

Gilmore + Pine: Experience Economy

Function

Emotion

Good Experience Design   Reactrix

  Top down projection   Camera based input   Reactive Graphics   No instructions   No training

Improve the experience of picking up rubbish?

World’s Deepest Rubbish Bin

  The Fun Theory – http://www.funtheory.com

Improve the experience of walking up stairs?

Musical Stairs

  The Fun Theory – http://www.funtheory.com

Apple: The Value of Good Design

  Good Experience Design Dominates Markets

iPod Sales 2002-2007

Using the N-gage

SideTalking   http://www.sidetalkin.com

Interaction Design

“Designing interactive products to support people in their everyday and working lives” Preece, J., (2002). Interaction Design

  Design of User Experience with Technology

  Higher in the value chain than product design

  Interaction Design involves answering three questions:   What do you do? - How do you affect the world?   What do you feel? – What do you sense of the world?   What do you know? – What do you learn?

Interaction Design is All About You

  Users should be involved throughout the Design Process

  Consider all the needs of the user

Interaction Design Process

experiences

applications

tools

components

Building Compelling AR Experiences

Tracking, Display

Authoring

Interaction

Usability

Summary   In order to build AR applications you need to

focus on the user experience   Great user experience is based on

  Low level AR component technology   Authoring tools   Application/Interaction design  User experience texting

AR Technology

experiences

applications

tools

components

Sony CSL © 2004

Building Compelling AR Experiences

Display, Tracking

Core Technologies  Combining Real and Virtual Images

•  Display technologies  Interactive in Real-Time

•  Input and interactive technologies  Registered in 3D

•  Viewpoint tracking technologies Display

Processing

Input Tracking

AR Displays

AR Displays

e.g. window reflections

Virtual Images seen off windows

e.g. Reach-In

Projection CRT Display using beamsplitter

Not Head-Mounted

e.g. Shared Space Magic Book

Liquid Crystal Displays LCDs

Head-Mounted Display (HMD)

Primarily Indoor Environments

e.g. WLVA and IVRD

Cathode Ray Tube (CRT) or Virtual Retinal Display (VRD)

Many Military Applications & Assistive Technologies

Head-Mounted Display (HMD)

e.g. Head-Up Display (HUD)

Projection Display Navigational Aids in Cars

Military Airborne Applications

Not Head Mounted (e.g. vehicle mounted)

Primarily Outdoor (Daylight) Environments

AR Visual Displays

Display Technologies

 Types (Bimber/Raskar 2003)  Head attached

•  Head mounted display/projector  Body attached

•  Handheld display/projector  Spatial

•  Spatially aligned projector/monitor

Display Taxonomy

Head Mounted Displays

Head Mounted Displays (HMD) -  Display and Optics mounted on Head -  May or may not fully occlude real world -  Provide full-color images -  Considerations

•  Cumbersome to wear •  Brightness •  Low power consumption •  Resolution limited •  Cost is high?

Key Properties of HMD   Field of View

  Human eye 95 degrees horizontal, 60/70 degrees vertical

  Resolution   > 320x240 pixel

  Refresh Rate   Focus

  Fixed/manual

  Power   Size

Types of Head Mounted Displays

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

The image cannot be displayed. Your computer

Occluded See-thru

Multiplexed

Immersive VR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

Non see-thru!Image source

& optics

virtual object

Virtual World

See-thru AR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

see-thru!combiner

Virtual Image superimposed!over real world object

real world

Image source

Optical see-through head-mounted display

Virtual images from monitors

Real World

Optical Combiners

Optical See-Through HMD

Epson Moverio BT-200

▪  Stereo see-through display ($700) ▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪  Android Powered, separate controller ▪  VGA camera, GPS, gyro, accelerometer

View Through Optical See-Through HMD

The Virtual Retinal Display

  Image scanned onto retina   Commercialized through Microvision

  Nomad System - www.mvis.com

Strengths of optical see-through AR   Simpler (cheaper)   Direct view of real world

  Full resolution, no time delay (for real world)   Safety   Lower distortion

  No eye displacement   but COASTAR video see-through avoids this

Video AR Architecture

Head!Tracker

Host !Processor

Graphics!renderer

Digital!Mixer Frame!

Buffer

head position/orientation

to network Display!Driver

Non see-thru!Image source

& optics

Head-mounted camera aligned to

display optics

Video!Processor

Video image of real world

Virtual image inset into video of real world

Video see-through HMD Video cameras

Monitors

Graphics

Combiner

Video

Video See-Through HMD

Vuzix Wrap 1200DXAR

▪  Stereo video see-through display ($1500) ■ Twin 852 x 480 LCD displays, 35 deg. FOV ■ Stereo VGA cameras ■ 3 DOF head tracking

View Through a Video See-Through HMD

Strengths of Video See-Through AR   True occlusion

  Virtual images can block view of real world

  Digitized image of real world   Flexibility in composition  Matchable time delays  More registration, calibration strategies

  Wide FOV is easier to support

Optical vs. Video AR Summary   Both have proponents   Video is more popular today?

  Likely because lack of available optical products

  Depends on application?  Manufacturing: optical is cheaper  Medical: video for calibration strategies

Eye multiplexed AR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

Virtual Image inset into!real world scene

real world

Opaque!Image source

Virtual Image ‘inset’ into real

Google Glass

View Through Google Glass

Vuzix M-100

▪  Monocular multiplexed display ($1000) ■ 852 x 480 LCD display, 15 deg. FOV ■ 5 MP camera, HD video ■ GPS, gyro, accelerometer

Display Types

▪  Curved Mirror ▪  off-axis projection ▪  curved mirrors in front of eye ▪  high distortion, small eye-box

▪  Waveguide ▪  use internal reflection ▪  unobstructed view of world ▪  large eye-box

See-through thin displays

▪  Waveguide techniques for thin see-through displays ▪  Wider FOV, enable AR applications ▪  Social acceptability

Opinvent Ora

Waveguide Methods

See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable

Holographic Hologram diffracts light Limited FOV Colour bleeding

Diffractive Slanted gratings Total internal reflection Costly, small FOV

Waveguide Methods

See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable

Clear-Vu Reflective Several reflective elements Thinner light guide Large FOV, eye-box

Reflective Simple reflective elements Lower cost Size is function of FOV

Comparison Chart