Software Development ForumEmerging Technology SIG

Weds. Sept 9 2009

Augmented Reality

Clark DodsworthOsage Associates

clark@dodsworth.com

Maribeth BackFX Palo Alto Laboratory

back@fxpal.com

Theoretical foundations

Scope

My background

Brief AR history

Directions for development, revenue + evolution

Maribeth Back, FX-PAL:

“High Tech Chocolate: Exploring Mixed Reality applications for Industry”

Ribbit.com demo by Kristi Wells and Brendan Lee

Tonight’s Plan

Augmenting Reality and Revenue Connecting Mobile, Sensors, Location & Layers

Clark Dodsworthclark@dodsworth.com

Alfred North Whitehead,first augmented reality theorist:

#1: “Civilization advances by extending the number of important operations which we can perform without thinking of them.” ...effective AR doesn’t increase the cognitive load

#2: We think in generalities, but we live in detail.” ...AR tasks: managing detail

#3: “The art of progress is to preserve order amid change and to preserve change amid order.” ...contextually dynamic data management & display via AR

Scope - General

AR = subset of Ubiquitous Computing: same infrastructure and issues as UbiComp and its personalization.

AR = enhance personal awareness (as needed) of: physical & non-physical context, personal state, opportunities & threats...local & distant.

AR = deliver personalized info with less effort, more space & time precision, and fewer seams than before, without increasing distraction.

Scope - General

AR = subset of Ubiquitous Computing: same infrastructure and issuesas UbiComp and its personalization.

AR = enhance personal awareness (as needed) of: physical & non-physical context, personal state, opportunities & threats...local & distant.

AR = deliver personalized info with less effort, more space & time precision, and fewer seams than before, without increasing distraction.

Augmented Context + Information Salience = Value

Scope - Model

1st plateau of AR = augmented perception of internal, external, & distant reality. Smartphone and headset AR that’s non-intrusive.

Beyond that, the value of well-evolved AR has significant parallels to the value of consciousness:

2nd plateau of AR = augmented decision-making. Integrated analysis, evaluation, and management (informatics) of data streams, constantly and dynamically evaluating the salience of each stream in context of the user’s ever-changing current situation and the user’s historical paths through n dimensions of data.

Then displaying the most salient data...when appropriate: Contextual evaluation driving contextual augmentation; software ‘awareness’ augmenting the decision process via delivery of filtered salient aspects of the user’s situation. At least 2 generations further.

Scope - Model

The value of well-evolved AR has significant parallels to the value of consciousness, AKA:

...the Default Mode Network of the brain

• One of the hardest-working systems in the brain

• Active when the brain is aware, but not focused on a particular task

• Reduces activity when you concentrate on a specific task

• 2 linked brain areas: A) Imagination, empathy for the thoughts and intent of othersB) Personal memories, visualization of future actions, scenarios

Scope - Model

The value of well-evolved AR has significant parallels to the value of consciousness, AKA:

...the Default Mode Network of the brain

• One of the hardest-working systems in the brain

• Active when the brain is aware, but not focused on a particular task

• Reduces activity when you concentrate on a specific task

• 2 linked brain areas: A) Imagination, empathy for the thoughts and intent of othersB) Personal memories, visualization of future actions, scenarios

DFM = constant process of considering how to be & what to do next.

Getting and dealing with the data upon which to make all those decisions is the constant background task of 2nd-plateau AR apps.

Scope - Phone

AR hot again as part of the smartphone ecosystem.

Smartphone AR (SAR?) = low-hanging fruit. 2nd is games.

SAR & LBS* = features, not an industry. Startups will fail/be acquired.

Privacy / security issues will be more extreme than currently.

Acronym Fusion: AR + LBS + GIS + Gesture = POTS... “my computer”

*LBS = Location-Based Services

Scope - farther

AR hot again as part of the smartphone ecosystem.

Smartphone AR (SAR?) is the low-hanging fruit.

SAR & LBS = features, not an industry; startups will fail / be acquired.

Privacy / security issues will be more extreme.

Acronym Fusion: AR + LBS + GIS = POTS... “my computer”

After that: Contextual Personalization + More Sensors + onboard DBs + Projectors

Background

My work Product strategy, dev., design, improvement, & IP evaluation, all based on User-Centered Design. For emerging technologies and markets in software, consumer electronics, and entertainment.

My perspective “Product as Tool” Tool should adapt to (fit) the user and be its own user manual.

Scythes are an excellent model system for tool design, though we don’t have ~2000 years to iterate.

AR-related projects:

VideoCart, ‘92: Location-aware mobile retail touch-screen kiosk system with data-driven path mapping.

Live Wall, ‘98: 100-meter, rear-projection, 1:1 scale videophone.

“Ambient Intelligence” strategy,’98 for Philips Consumer Electronics.

Royal Tyrell Museum, Alberta, ‘06: Concept & spec: permanent LBS-based virtual paleontological exhibit complex across 11 remote

archaeological sites, for Reich + Petch.

Dubailand theme parks, ‘08: Concept evaluation & high-level system spec, strategic & tech partners: 24/7 multi-application smartphone AR experience.

Background

VideOcart

VideoCart: mobile location-aware touchscreen retail, ‘92

Live Wall: semi-mobile videophone design for Philips, ‘98

“Ambient Intelligence” strategy for Philips, ‘98

all photos: Philips

services across all hotel, leisure, retail, restaurant, and transit in Dubailand, and to

follow the visitors home.

Dubailand AR project, ‘08

• Concept > smartphone AR • Technology identification, assessment, & qualification • High-level system definition • Strategic partnering • Broadened scope to include

c. Lynne LaCascia, all rights reserved

Part of Dubailand model:

Dubailand AR project, ‘08

c. Lynne LaCascia, all rights reservedAll images © Chimera Design 2007-2008

AR Timeline

Sailboat tell-tales

~1900

Rear-view mirrors

1st jet fighter heads-up display

1958 1970s

Commercial aviation use

Helmet-mounted displays

“Augmentedenvironments,” sensor nets

1974-8

US DoD air-combat testing

1988

1st car HUDOlds Cutlass Supreme

Now 20??

Battery energy-density resolved

1992+ 1999

Horsecarriages

Tom Caudell &David Mizell

ARToolkitUW HITLab

Prehistory,soon after invention of the sail.

Dedicated tools begin:

“UbiComp”:Mark Weiser

2006

Useful smart- phone AR

Nokia’s MARA

Hands-up displays

Reed sails on boats in the prehistoric Middle East had inherent sail-edge tell-tales.

Example of sensor/display fusion

photo c. Luxury Peru Tours & Travel

Marmon Wasp racecar, 1911

Non-see-through HUD display: Rear-view mirror is just above line of sight

MIG21 heads-up display (installed): circa 1960s

Cintel “Buccaneer” aircraft cockpit heads-up display device, UK 1958

photo: Rochester Avionics Archives, UK

photo: Will Hise cc

HUDs

HUD: modern commercial aviation AR

HMDs Helmet-mounted displays

• High-priority military R&D investment

• Tethered to power and computer

• Obstructs part of the field of view

• Delicate to align, adjust, maintain

• Size and weight unacceptable

• Adds complexity/distraction to user situation

• Current state of the art = net loss of context awareness

c. Michael Edwards,2007

Rockwell-Collins Sim Eye SR100A, 2009

HMDs Helmet-mounted displays

• High-priority military R&D investment

• Tethered to power and computer

• Obstructs part of the field of view

• Delicate to align, adjust, maintain

• Size and weight unacceptable

• Adds complexity/distraction to user situation

• Current state of the art = net loss of context awareness

Not AR

c. Michael Edwards,2007

Rockwell-Collins Sim Eye SR100A, 2009

photo: Mobilizy GmbH

Mobilizy's original product, “Wikitude AR (Travel) Guide,” for Android

Hands-Up Displays

Some Players

Mobilizy • Wikitude World Browser, • new Wikitude Drive (Android)

Nokia • Point & Find, Image Space, MARA

Layar • Layar Reality Browser 2.1 (Android) • Enables adding layers easily • Example: Trulia.com • Working on iPhone dev now

Studierstube ES • Rich development framework • Client and middleware • 3D object tracking

Copyright Network World, Inc. All rights reserved.

photo: Mobilizy GmbH

Total Immersion • D’Fusion Studio, Pro, Mobile

Metaio • Unify Design: no programming

Tonchidot • Sekai Camera (iPhone & Android)

Augmento

....

Studierstube ES

Copyright Network World, Inc. All rights reserved.

photo: Mobilizy GmbH

Now

• GPS• Accelerometers• Light sensors• Proximity sensors• Compass• Touch (2- or 2.5-D)• Bluetooth

Need

• Near-field RFID (“touch,” “wand”)

• Constant visual awareness + recording

• Constant audio awareness + recording

• Galileo-grade location resolution

• Ultrasonic & IR spatial analysis

• Laser projector(s) with geometric awareness & beam stabilization for object augmentation

• ...

Hardware

Next---1. Constant, robust markerless 2D & 3D feature I.D. and object recognition (Nokia Point & Find, ...) as a general

feature

2. Motion analysis + evaluation linked to the object recognition

3. General-purpose indoor position sensing

4. More nuanced gestural input interpretation + integration with voice

5. Constant audio + visual awareness with flagging & interpretation

1. Allostatic control of data delivery: apps “learn the user” over time, to drive what to display (‘awareness driving awareness,’ ‘context drives the augmentation of context’)

1. Constant, opportunistic acquisition and evaluation of available sensor-net feeds.

Software

Still further out...

Rob Cook’s award talk at Siggraph ‘09:

“...the next holy grail for the computer graphics community should be realist augmented reality.”

http://media.siggraph.org/s2009podcasts/RobCook_AwardExport.mov

Software

Augmenting Revenue

Assumptions• Full OS citizenship: no tethering to a computer • More coverage & bandwidth• CPU & memory follow Moore’s Law• Something magic happens to battery energy density.

Work• Extend existing software onto the mobile platform with AR where

appropriate. Includes Augmenting Security for public locations. • Building AR into new apps as a normal part of software develop-

ment: Like the shift to web from desktop & enterprise.• Multi-modal interface differentiation:

Handheld vs. plugged into a desktop screen or large projector

Platforms and toolkits

Platforms iPhone 3.9 Android 1.5 Palm WebOS RIM BlackBerry OS Symbian S60 5th Edition Windows Mobile 6.5 (Samsung LinuxOS?)

Tools Studierstube ES Layar ARToolkit NYARToolkit FLARToolkit Tinmith ...

clark@dodsworth.com

end OA logo

AR-related conferences

Ubicomp 2009, Orlando, FL. Sept. 30-Oct. 3, 2009 www.ubicomp.org/ubicomp2009/ Int’l Symposium on Mixed and Augmented Reality (ISMAR09) - Orlando, FL, October 19-22, 2009 http://ismar09.org 8th Int’l Conference on Mobile and Ubiquitous Multimedia (MUM 2009) 22-25 Nov., Cambridge, UK www.mum2009.org8th IEEE Int’l Conf. on Pervasive Computing & Communications (PerCom) Mannheim (Mar 29-April 2, 2010)

www.percom.org7th IEEE Workshop on Ubiquitous Communications & Services: www.ubiquitous-management.org/mucs/2010/index.phpIntSymposium on Wearable Computers (ISWC'09) - Linz,  Austria, September 4-7, 2009. www.iswc.net/register

IEEE VR March 20-26, 2010 in Waltham, Massachusetts http://conferences.computer.org/vr/2010/IEEE Symposium on 3D User Interfaces (3DUI), March 20-21 2010, Waltham, Mass

http://conferences.computer.org/3dui/3dui2010/Symposium on Haptic Interfaces March 25-26 2010, Waltham, Mass., www.hapticssymposium.org/next_conference.htmInt’l Conference on Mobile, Wireless and Optical Communication (MWOC 2009), Dec. 09 Korea

www.iacsit.org/mwoc/index.htm Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) Nov. 30-Dec. 3 Orlando, FL  www.iitsec.org/SVC Wireless Annual Conference  Sep 25-26, 2009, Mountain View www.svcwireless.org/svcw/ac2009-homeLocation Intelligence Conference 2009  Oct 5-7 2009, Broomfield, CO www.locationintelligence.net/3rd Int’l Conference on Mobile Ubiquitous Computing, Systems, Services & Technologies UBICOMM October 11-16, 2009 -

in Sliema, Malta www.iaria.org/conferences2009/UBICOMM09.htmlMobiquitous 2010 www.mobiquitous.org/2nd ACM Int’l Workshop on Mobile Opportunistic Networking Feb. 22-23, 2010, Pisa,  http://cnd.iit.cnr.it/mobiopp20102nd Int’l Conference on Mobile Lightweight Wireless Systems (MOBILIGHT 2010) www.mobilight.org5th Int’l Conference on Body Area Networks September (BodyNets) Sept. 10-12, 2010 Corfu, Greece www.bodynets.org/IMMERSECOM 2010 http://immerscom.org/ (2009 was at Berkeley)1st Int’l ICST Conference on User Centric Media  (UCMedia 2009), 9-11 Dec. 09, Venice www.usercentricmedia.org

Credits

• Default Mode Network paper: Gusnard, DA, Raichle, ME: “Searching for a baseline: the resting human brain from the perspective of functional imaging.” Nature Reviews Neuroscience, 2001

• “iLamps: Geometrically Aware and Self-Configuring Projectors,” Rascar, et al. 2003

• Commercial aviation HUD image courtesy Todd Lapin, cc

• MARA image courtest Nokia Research

• Handheld AR.net (http://handheldar.net/stbes.php) for Studierstube ES diagram

• Fighter jet HUD device image cc www.rochesteravionicarchives.co.uk

• Photo of sail’s tell-tales courtesy Bill Gracey, cc

• Modern helmet-mounted see-through display courtesy Rockwell-Collins: "SIM EYE SR100-A”

• Photo of Ray Harroun’s Marmon Wasp racecar with mirror, cc The359

• Vuzix Wrap 920AV and Tac-Eye GMD images c. Vuzix Inc.

• Theme park illustrations © Chimera Design, 2007-2009

• Photo of Dubailand model © Lynne LaCascia, 2009

Software Development ForumEmerging Technology SIG

Augmented Reality

back@fxpal.com

Maribeth Back, Fuji-Xerox Palo Alto Labs“High Tech Chocolate:

Exploring Mixed Reality applications for Industry”