COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

July 12th 2013

Lecture 1: Introduction

Mark Billinghurst   PhD Electrical Engineering

  University of Washington

  Interaction Design   Museum experiences   Tools for designers

  Augmented Reality   Mobile AR, Evaluation,   Multimodal Interfaces, Collaborative

  Collaboration   Enhanced FtF and remote collaboration   Social networking

Overview   One two hour lecture a week

  Friday 11am – 1pm

  You will learn   Introduction to Augmented Reality   Augmented Reality technology   AR Interaction techniques   Interaction Design   AR authoring tools   Research directions in AR

  Complete a simple project

Course Outline   Wk 1 (July 8th): Introduction to Augmented Reality (AR)   Wk 2 (July 15th): AR Technology   Wk 3 (July 22nd): AR Developer Tools   Wk 4 (July 29th): AR Interaction Techniques   Wk 5 (Aug 5th): AR Applications   Wk 6 (Aug 12th): Outdoor and Mobile AR

  Wk 9 (Sept 2nd): Collaborative AR   Wk 10 (Sept 9th): AR research Directions   Wk 11 (Sept 16th): Final Project Presentations

Assessment

  Research project – 40%  Group work (2-4 people)  Due Sept 27th

  Two Class Assignments – 20 %   Programming assignments, individual work

  Final Exam – 40%   Exam week Oct 3rd – 14th

Introduction

A Brief History of Time

  Trend   smaller, cheaper, more functions, more intimate

  Technology becomes invisible   Intuitive to use   Interface over internals   Form more important than function   Human centered design

A Brief History of Computing

  Trend   smaller, cheaper, faster, more intimate, intelligent objects

  Computers need to become invisible   hide the computer in the real world

-  Ubiquitous / Tangible Computing

  put the user inside the computer -  Virtual Reality

Invisible Interfaces

Jun Rekimoto, Sony CSL

Graphical User Interfaces

  Separation between real and digital worlds  WIMP (Windows, Icons, Menus, Pointer) metaphor

Ubiquitous Computing

  Computing and sensing embedded in real world   Particle devices, RFID, motes, arduino, etc

Virtual Reality

  1985…

Virtual Reality

  Immersive VR  Head mounted display, gloves   Separation from the real world

1977 – Star Wars

Augmented Reality Definition   Defining Characteristics [Azuma 97]

 Combines Real and Virtual Images -  Both can be seen at the same time

  Interactive in real-time -  The virtual content can be interacted with

  Registered in 3D -  Virtual objects appear fixed in space

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

2008 - CNN

  Put AR pictures here

Augmented Reality Examples

AR vs VR   Virtual Reality: Replaces Reality

  Scene Generation: requires realistic images  Display Device: fully immersive, wide FOV   Tracking and Sensing: low accuracy is okay

  Augmented Reality: Enhances Reality   Scene Generation: minimal rendering okay  Display Device: non-immersive, small FOV   Tracking and Sensing: high accuracy needed

Milgram’s Reality-Virtuality continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

Augmented Virtuality

  VR with windows into the real world

Metaverse   Neal Stephenson’s “SnowCrash”   The Metaverse is the convergence of:

  1) virtually enhanced physical reality   2) physically persistent virtual space

  Metaverse Roadmap   http://metaverseroadmap.org/

Metaverse Dimensions • Augmentation technologies that layer information onto

our perception of the physical environment. • Simulation refers to technologies that model reality • Intimate technologies are focused inwardly, on the

identity and actions of the individual or object; • External technologies are focused outwardly, towards

the world at large;

Metaverse Components   Four Key Components

  Virtual Worlds   Augmented Reality  Mirror Worlds   Lifelogging

Mirror Worlds   Mirror worlds are informationally-enhanced

virtual models of the physical world.  Google Earth, MS Street View, Google Maps

LifeLogging   Technologies record and report the intimate

states and life histories of objects and users  Nokia LifeBlog, Nike+, FitBits

  Steve Mann - LifeLogging

Gordon Bell: LifeLogging

1 TB to store 65 years of data

Summary   Augmented Reality has three key features

 Combines Real and Virtual Images   Interactive in real-time   Registered in 3D

  AR can be classified alongside other technologies   Invisible Interfaces  Milgram’s Mixed Reality continuum  MetaVerse

AR History

A Brief History of AR (1)   1960’s: Sutherland / Sproull’s

first HMD system was see-through

A Brief History of AR (2)

F16 – Head Up Display

A Brief History of AR (3)

1960 - 70’s: US Air Force helmet mounted displays (T. Furness)

A Brief History of AR (4)

1970 - 80’s: US Air Force Super Cockpit (T. Furness)

A Brief History of AR (5)

  Early 1990’s: Boeing coined the term “AR.” Wire harness assembly application begun (T. Caudell, D. Mizell).

A Brief History of AR (6)

  1994: Motion stabilized display [Azuma]   1995: Fiducial tracking in video see-through [Bajura / Neumann]   1996: UNC hybrid magnetic-vision tracker

A Brief History of AR (7)

  1996: MIT Wearable Computing efforts   1998: Dedicated conferences begin (ISMAR)   Late 90’s: Collaboration, outdoor, interaction   Late 90’s: Augmented sports broadcasts

History Summary   1960’s – 80’s: Early Experimentation   1980’s – 90’s: Basic Research

  Tracking, displays

  1995 – 2005: Tools/Applications   Interaction, usability, theory

  2005 - : Commercial Applications  Games, Medical, Industry

2007 - AR Reaches Mainstream

  MIT Technology Review  March 2007   list of the 10 most

exciting technologies   Economist

 Dec 6th 2007  Reality, only better

Gartner Hype Cycle

2009 - AR in Magazines   Esquire Magazine

 Dec 2009 issue   12 pages AR content

  Many Others  Wired  Colors   Red Bull   Etc

Google Searches for AR

2008 - Browser Based AR   Flash + camera + 3D graphics   High impact

  High marketing value

  Large potential install base   1.6 Billion web users

  Ease of development   Lots of developers, mature tools

  Low cost of entry   Browser, web camera

Impact of Web-based AR   Boffswana Living Sasquatch

  http://www.boffswana.com/news/?p=605

  In first month   100K unique visits   500K page views   6 minutes on page

2005 - Mobile Phone AR  Mobile Phones

  camera   processor   display

  AR on Mobile Phones   Simple graphics  Optimized computer vision  Collaborative Interaction

AR Advertising (HIT Lab NZ 2007)

  Txt message to download AR application (200K)   See virtual content popping out of real paper advert   Tested May 2007 by Saatchi and Saatchi

2008: Location Aware Phones

Nokia Navigator Motorola Droid

2009 - Outdoor Information Overlay   Mobile phone based   Tag real world locations

  GPS + Compass input   Overlay graphics data on live video

  Applications   Travel guide, Advertising, etc

  Wikitude, Layar, Junaio, etc..   Android based, Public API released

Layar (www.layar.com)   Location based data

 GPS + compass location  Map + camera view

  AR Layers on real world  Customized data   Audio, 3D, 2D content

  Easy authoring   Android, iPhone

AR Today  Key Technologies Available

- Robust tracking (Computer Vision, GPS/sensors) - Display (Handheld HMDs) -  Input Devices (Kinect, etc) - Developer tools (Qualcomm, Metaio, ARTW)

 Commercial Business Growing - Gaming, GPS/Mobile, Online Advertisement

•  >$5 Billion USD by 2016 (Markets andMarkets) •  >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

  Web based AR   Flash, HTML 5 based AR   Marketing, education

  Outdoor Mobile AR   GPS, compass tracking   Viewing Points of Interest in real world   Eg: Junaio, Layar, Wikitude

  Handheld AR   Vision based tracking   Marketing, gaming

  Location Based Experiences   HMD, fixed screens   Museums, point of sale, advertising

Typical AR Experiences

AR Business Today   Marketing

 Web-based, mobile

  Mobile AR  Geo-located information and service  Driving demand for high end phones

  Gaming  Mobile, Physical input (Kinect, PS Move)

  Upcoming areas  Manufacturing, Medical, Military

Some Commercial AR Companies   ARToolworks (http://www.artoolworks.com/)

  ARToolKit, FLARToolKit, SDKs

  Metaio (http://www.metaio.com/)   Marketing, Industry, SDKs

  Total Immersion (http://www.t-immersion.com/)   Marketing, Theme Parks, AR Experiences

  Qualcomm (http://www.vuforia.com/)   Mobile AR, Vuforia SDK

  Many small start-ups (String, Ogmento, etc)

Summary   Augmented Reality has a long history going

back to the 1960’s   Interest in AR has exploded over the last few

years and is being commercialized quickly   AR is growing in a number of areas

 Mobile AR  Web based AR  Marketing experiences

Sample AR Applications

Applications

  Medicine   Manufacturing   Information overlay   Architecture   Museum   Marketing   Gaming

Applications: medical   “X-ray vision” for surgeons   Aid visualization, minimally-invasive operations.

Training. MRI, CT data.  Ultrasound project, UNC Chapel Hill.

Courtesy UNC Chapel Hill

Medical AR Trials   Sauer et al. 2000 at Siemens

Corporate Research, NJ   Stereo video see through

F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial, MICCAI 2002

Assembly and maintenance

© 1993 S. Feiner, B. MacIntyre, & D. Seligmann, Columbia University

© 1996 S. Feiner, B. MacIntyre, & A. Webster, Columbia University

PS3 - Eye of Judgment (2007)   Computer Vision Tracking   Card based battle game   Collaborative AR   October 24th 2007

AR Books – Markerless Tracking

AR Annotations

Columbia University

HRL

© 1993 S. Feiner, B. MacIntyre, M. Haupt, & E. Solomon, Columbia University

© 1997 S. Feiner, B. MacIntyre, T. Höllerer, & A. Webster, Columbia University

Broadcast TV

Interactive Museum Experiences   BlackMagic

  Virtual America’s Cup   410,000 people in six months

  MagicPlanet   TeManawa science museum   Virtual Astronomy   Collaborative AR experience

  AR Volcano   Interactive AR kiosk   Scienceworks museum, Melbourne

Digital Binocular Station

http://www.DigitalBinocularStation.com/

Museum Archeology   LifePlus (2002-2004)

 Natural feature tracking   Virtual characters  Mobile AR system

  Archeoguide (2000-2002)  Cultural heritage on-site guide  Hybrid tracking   Virtual overlay

Sales and Marketing   Connect with brands and branded objects   Location Based Experiences

  Lynx Angels

  Web based   Rayban glasses

  Mobile   Ford Ka campaign

  Print based   Red Bull Magazine

Summary   AR technology can be used to develop a wide

range of applications   Promising application areas include

 Games   Education   Engineering  Medicine  Museums   Etc..

Things to Do..   Find your favourite YouTube video showing

an AR interface – send to mark   Read – Feiner Scientific American article   Find a friend for the project   Think about project ideas