COSC 426 Lect. 5 - Mobile AR
189
Lecture 5: Mobile AR Lecture 5: Mobile AR Mark Billinghurst k billi h t@hitl b mark.billinghurst@hitlabnz.org HIT Lab NZ University of Canterbury
-
Upload
mark-billinghurst -
Category
Technology
-
view
2.850 -
download
5
description
Lecture 5 of the COSC 426 graduate course on Augmented Reality. This lecture provides an overview of Mobile Augmented Reality. The Lecture is given by Mark Billinghurst of the HIT Lab NZ at the University of Canterbury
Transcript of COSC 426 Lect. 5 - Mobile AR
Lecture 5: Mobile ARLecture 5: Mobile AR
Mark Billinghurstk billi h t@hitl [email protected]
HIT Lab NZ University of Canterbury
AR on mobile phonesLow cost, widely spread platform
Billions of phones deployedPeople know how to use themStrong demand from commercial sideHuge chance for AR!
Target practical applicationsEasy to useQuality graphicsRobust tracking15-30 Hz overall frame rate
Why would you use phones?Robust and fool-proof
People know how to use their mobile phones…
Variety of supported devicesSelf-contained operationSelf contained operation
Enough processing power to do everything natively
Ultra mobileUltra-mobileLow cost devices
!Even better: people already own the target hardware!
A very unique chance for bringing AR to the masses
Why would you not use phones?Compared to PC-based setups
Less processing power and memoryp g p yHarder to program, debug and deploy
Hardware difficult or impossible to extendHardware difficult or impossible to extendSmall number of available librariesT i ll l li l i i h Typically only little experience in research groupsSo many different devices and operating systems
Other limitations of handheld ARUsual limitations in mobile HCI
Small screen- Less information possible to display- Less immersion
Limited inputLimited input…
Other limitationsOther limitationsYou see through the camera and not through the phoneSwitch attention between background and phoneSwitch attention between background and phoneStrain factor of holding the phone upSocial issues with pointing the phone at peoplep g p p p
Mobile AR HistoryMobile AR History
E l f M b l AREvolution of Mobile AR
Camera phoneCamera phone
Thin client ARWearable AR Camera phone
- Self contained AR
WearableComputers
- Thin client AR
Handheld AR Displays
Self contained ARPDAs-Thin client AR
p yPDAs-Self contained AR
1995 1997 2001 2003 20041995 1997 2001 2003 2004
Handheld DisplaysTethered Applications
Fitzmaurice Chameleon (1994)( )Rekimoto’s Transvision (1995)Tethered LCDPC Processing and Tracking
Handheld AR Display - Tethered
1995, 1996 Handheld ARARPad CameleonARPad, CameleonRekimoto’s NaviCam, TransvisionTethered LCDTethered LCDPC Processing and Tracking
Mobile AR: Touring Machine (1997)Columbia University
Feiner, MacIntyre, Höllerer, Webster
Combines See through head mounted displayGPS t kiGPS trackingOrientation sensorBackpack PC (custom)p ( )Tablet input
MARS View
Vi t l t l id th l ldVirtual tags overlaid on the real world“Information in place”
Backpack/Wearable AR
1997 Backpack ARFeiner’s Touring MachineAR Quake (Thomas)Tinmith (Piekarski)MCAR (Reitmayr)Bulky, HMD based
Mobile AR - Hardware
GPSAntenna
RTK correction AntennaRTK correction AntennaRTK correction AntennaRTK correction Antenna Example self-built workingsolution with PCI-based 3D graphics
PCI 3D Graphics BoardHMDHMD
ControllerControllerHMDHMD
ControllerController
solution with PCI based 3D graphics
PC104 Sound Card
TrackerController
DC to DCCPU
PC104 PCMCIA
DC to DCConverter
Battery
WearableComputer
Hard Drive
Serial
GPS RTK correction
Radio
Ports
Columbia Touring Machine
1997 Philip Kahn invents camera phoneSharp J-SH04
1997 Philip Kahn invents camera phone1999 First commercial camera phone
Millions of Camera Phones
1000
1200
800
1000
DSC
400
600DSCPhone
0
200
2002 2003 2004 2005 2006 2007 2008 2009 2010
Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge)PDA used as I/O deviceWi l i k i Wireless connection to workstation Room-scale ultrasonic tracking (Bat)
2001 AR-PDA (C Lab)PDA thin graphics clientRemote image processingwww.ar-pda.com
Mobile Phone AR – Thin Client
2003 ARphone (Univ. of Sydney)Transfer images via Bluetooth (slow – 30 sec/image)g ( g )Remote processing – AR Server
Early Phone Computer Vision Apps2003 – Mozzies Game - Best mobile gameOptical motion flow detecting phone orientationSi SX1 S bi 120Mh VGA CSiemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH)Wi f N ki ' S i 60 Ch ll 2005 Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)
Computer Vision on Mobile PhoneCameras and Phone CPU sufficient for computer vision applications Pattern Recognition (Static Processing)g ( g)
QR CodeShotcode (http://www.shotcode.com/)
M i Fl (2D I P i )Motion Flow (2D Image Processing)GestureTek
- http://www.gesturetekmobile.com/
TinyMotion
3D Pose CalculationAugmented RealityAugmented Reality
Handheld AR – Self Contained2003 PDA-based AR
ARToolKit port to PDAStudierstube ported to PDAStudierstube ported to PDAAR Kanji Educational App.Mr Virtuoso AR characterWagner’s Invisible Train
- Collaborative AR
Mobile Phone AR – Self Contained2004 M bil Ph AR2004 Mobile Phone AR
Moehring, BimberHenrysson (ARToolKit)Henrysson (ARToolKit)Camera, processor, display together
Location Aware Phones
Nokia NavigatorMotorola Droid
Real World Information OverlayTag real world locations Tag real world locations
GPS + Compass inputOverlay graphics data on live videoOverlay graphics data on live video
Applications Travel guide, Advertising, etc
Eg: Mobilizy Wikitude (www.mobilizy.com)Android based, Public API released
Other companiesLayar, AcrossAir, Tochnidot, RobotVision, etc
Layar – www.layar.com
HIT Lab NZ Android AR PlatformArchitectural ApplicationLoads 3D models
a OBJ/MTL format
Positions content in spacepGPS, compass
Intuitive user interfaceIntuitive user interfacetoolkit to modify the model
Connects to back end model databaseConnects to back end model database
HIT Lab NZ Mobile Outdoor AR
History of Handheld and Mobile AR1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quakeg
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR Self contained: Invisible Train2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian
Mobile AR by Weight1996
2003 2007
Scale it down:Vesp‘R [Kruijff ISMAR07]:
Scale it down more:S t h $500
Backpack+HMD:5 8k
Vesp R [Kruijff ISMAR07]:…Sony UMPC 1.1GHz…1.5kg
till >$5K
Smartphone…$500…All-in-one…0.1kg
…5-8kg …still >$5K …billions of units
2011 S f h A2011 State of the ArtHandheld Hardware available
PDA, mobile phones, external camerasSensors: GPS, accelerometer, compass
Software Tools are AvailableTracking: ARToolKitPlus, QCARGraphics: OpenGL ESGraphics: OpenGL ESAuthoring: Studierstube, Layar, Wikitude, Unifye
What is needed:High level authoring toolsContent development toolsNovel interaction techniquesUser evaluation and usability
Mobile AR CompaniesMobile AR
GPS + compass
M C iMany CompaniesLayarWikitudeWikitudeAcrossair PressLiteYelpRobot visionEEtc..
$2 million USD in 2010$732 million USD in 2014
Handset ManufacturersQualcommQualcomm
$100 million USD investment
N kiNokia25+ people in NRC
SamsungExploring the space
Apple586 AR Applications on App Store
GoogleGoogle goggles/Android AR Applicationsg g gg pp
Qualcomm
Acquired Imaginationq gOctober 2010 - Releases free Android AR SDKComputer vision tracking - marker, markerless p gIntegrated with Unity 3D rendererhttp://developer.qualcomm.com/arp p q
Rock-em Sock-em
Sh d AR DShared AR DemoMarkerless tracking
Apple
iPhone 4 SDK supports direct camera accessppLaunches AR theme on App Store
> 500 AR apps on App Store
Developing AR applications
Mobile AR TechnologyInvolves
Tracking gContent Loading
Rendering/3D graphicsg g pUser Interface Application Design Application Design Evaluation
Scientific challengesAR requires (unlike related disciplines)AR requires (unlike related disciplines)
Strict real time operation (30Hz)- Unlike Ubicomp or mobile information systemsp y
High spatial precision (1cm, 1 degree)- Unlike location-based services
Robustness for operation by human user- Unlike many computer vision methods in automation etc.
M bil h AR i (i ddi i )Mobile phone AR requires (in addition)No thin client!S l l f f d k ARSame level of performance as desktop AR- New algorithms must be orders of magnitude more efficient
No unrealistic assumptions about HWNo unrealistic assumptions about HW
How does a basic AR application work?Main loopMain loop
Get a video frame from the cameraEstimate the position and the orientation
- computer vision, sensor input (GPS, compass)p p ( p )
Render the augmented scene (video + virtual content))Render GUIProcess user inputProcess user inputUpdate application status
Studierstube ES Framework
Typical AR application framework
Applications
Developed at TU Graz
StudierstGraz tube Software Staack
Platform
End User Application
Programming Lib i
pp
Libraries
OS/Low Level API
Hardware
Th S d b ES f kThe Studierstube ES frameworkA
pplicatioonsStudieContent
User Interface - Application
erstube Software S
Graphics
Content
StackTracking
Platform
Platform
Platform
What are the challenges?Experiences with embedded platforms requiredM l f ( i )Many platforms (operating systems)Slow CPUs (low clock rates, often no FPU)
Difficulties with trackingDifficulties with visualizations that require a lot of data
iprocessing
Slow memory accessNo or weak hardware 3D acceleration
Difficulties with graphics
Processing power of mobile phones
Weak processing power~1GHz Single core~1GHz, Single coreOften no floating point unit- Floating point code ~40x slower than integer codeoat g po t co e 0 s owe t a tege co e- Fixed point problematic for many algorithms- Requires good math library
Code optimized for phones runs 5-10x slower on ahigh-end phone than on an average PCNot going to change dramatically due to battery power
So what are the common problems?Bad camera quality under low lighting
Noise, motion blur,Strongly varies with different phones
Small memorySmall memoryKeeping large databases in memory is problematic
Slow memorySlow memoryLow clock rate- Processing large memory areas is prohibitiveg g y p- Typical CV building blocks (SVD, image processing) are too demanding
Slow data transfers between CPU and GPU
Pl f f h dh ld ARPlatforms for handheld ARPros Cons
Windows Mobile
Easy to program and debugLots of devices
Camera drivers are not always good
SymbianLargest installed basisGood devices
Hard to program and debug, SDK changes often, usually slow CPUs
Very nice hardware Camera API only with OS4iPhone
Very nice hardwareHype factor
Camera API only with OS4Objective C as main language
AndroidIncreasing number of devices
Java as main languageAndroidHype factor
Java as main language
Linux (LiMo, M )
Full Linux support, limitedh d f AR
Large set of different librariesMaemo) handsets for AR
Large set of different libraries
RIM Blackberry
Widely spread in the US Java onlyBlackberry
Palm WebOS
Nice hardwareOnly very few devices so farNo native SDK so far
Worldwide smartphone market share
1Q101Q09
Symbian
RIMRIM
iPhoneiPhone
Android
Source: Gartnerhttp://www.gartner.com
2011 US Market Share
What makes a device interesting for AR?
Open and easy to programGood cameraGood cameraFast CPU, FPU is a plusGood H/W 3D supportLarge installed basisg
Easy access to devices
GPS, accelerometer, compassGPS, accelerometer, compassEnough memory/storage
Typical Smart Phone HardwareCPUCPU
300-800+ Mhz
GPUGPUNone, or Power VR Chip (OpenGL ES1.0/2.0)
InputpTouch screen, keyboard, keypad
SensorsGPU l (1 3 5 b )GPU, compass, accelerometer, camera (1.3-5mb+)
NetworkingBluetooth Wifi 3GBluetooth, Wifi, 3G
Screen320x240 up to 800x480
HTC HD2HTC HD2Windows MobileWindows MobileFast CPU (1GHz)Big screen
4.3”, 800x480
GPS, compass and accelerometerGood cameraGood camera
Depends on lighting conditions
Hardware 3DHardware 3DSlow texture upload: slow video background renderingslow video-background rendering
HTC DHTC DesireAndroidAndroidFast CPU (1GHz)Smaller screen
3.7”, 800x480
Multi-touchHardware 3DHardware 3DGPS, compass and accelerometeraccelerometer
Ph 4iPhone 4Apple iOS 4 0Apple iOS 4.0Faster CPU (1.2GHz)Hi h l iHigh screen resolution
3.5”, 960x640
(Finally) camera APIMulti-touchHardware 3DGPS compass accelerometerGPS, compass, accelerometerand gyroscope
Hardware SensorsC ( l i f )Camera (resolution, fps)
Maker based/markerless trackingVid lVideo overlap
GPS (resolution, update rate)Outdoor locationOutdoor location
Compass Indoor/outdoor orientationIndoor/outdoor orientation
AccelerometerMotion sensing relative tiltMotion sensing, relative tilt
Programming Environments
Mobile Development EnvironmentsNot like developing for desktops
Wide range of different OSLimited CPU, low memory, poor graphics, no floating point
Popular Mobile OSS bi (S bi C++ C bid IDE)Symbian (Symbian C++, Carbide IDE)iPhone (Objective C)Android (Java, Native NDK wrapper) (J pp )Windows Mobile (C#, C++, Visual Studio)
OtherPalm OS, Blackberry, Linux
Programming Windows MobileVery similar to desktop Windows
Almost identical APIU d f lUnicode functions only
Development toolsEmbedded Visual C++ C#Embedded Visual C++, C#
- Deprecated, not suggested
Visual Studio 2005Visual Studio 2008
- Required for FPU usage
F i f b l k tFor overview of camera bugs look athttp://studierstube.org/handheld_ar/camera_phones.php
Programming SymbianDevelopment tools
Carbide.c++Commercial version required for on-device debugging (important since emulator is bad…)SDK f dSDK appropriate for your device
Many quirksC i l d C++ tCrippled C++ supportWriting to static variables not allowed/recommendedCleanup StackpAPI includes ~1500 classes
Moving to Qtg
Programming iPhoneHarsh restrictions from Apple
Apps have to go through the apps store
X d IDE f d lXcode IDE for developmentNice development tools
Objective CObjective-CRequired for application developmentCan call into C/C++ codeCan call into C/C code
Camera API support in iOS 4+Can overlay on live videoy
AndroidHardware creatorsHardware creators
HTC, LG, Samsung, Motorola
Wid l il bl hWidely available phoneDifferent form factors – tablets, phones, PC, etc
Multiple versions and fragmentationAndroid 1.0, 1.6Android 2.0, 2.1
Free Tools Eclipse DevelopmentApp Integrator
Mobile Graphics
Computer Graphics on Mobile PhonesSmall screen, limited input optionsLimited support for accelerated graphics
Most phones have no GPU Most phones have no GPU
Mobile Graphics LibrariesOpenGL ES (1.0, 2.0)p ( , )
- Cross platform, subset of OpenGL- C/C++ low level library
Java M3GJava M3G- Mobile 3D graphics API for J2ME platform - Object importer, scene graph library- Support from all major phone manufacturerspp j p
O GL ESOpenGL ESSmall footprint subset of OpenGLSmall-footprint subset of OpenGL
OpenGL is too large for embedded devices!
P f l l l l API f ll f i li f 3D Powerful, low-level API, full functionality for 3D gamesCan do almost everything OpenGL can (but only one way)A l bl ll k l fAvailable on all key platformsSoftware and hardware implementations available
F ll iblFully extensibleExtensions like in OpenGL
No redundancy!Convenience functions removed
OpenGL ESSLIDE 68
OpenGL ES vs. OpenGL (1.x)OpenGL OpenGL ES
glBegin/glEnd 11
Primitive Types all no quads & polygons
Data Types float double int etc float fixedData Types float, double, int, etc… float, fixed
glDraw/Read Pixels glReadPixels only
T tTextures 1D, 2D, 3D, cube 2D
Stencil optional
Window Bindings WGL, GLX, etc… EGL
1: Except for Security Critical profile
OpenGL ESSLIDE 69
OpenGL ES on mobile devices
Java Applications C++ Applications
S h API G MiddlScenegraph APIM3G (JSR 184)
GameEngine
Middleware Engines
OpenGL ES Performance
( )Faster graphics (esp. hardware accelerated)Longer batter performance (> 10%)
VVersionsTwo major tracksTwo major tracks
Not compatible, parallel rather than competitive
OpenGL ES 1 xOpenGL ES 1.xFixed function pipelineSuitable for software implementationsSuitable for software implementationsAll 1.x are backwards compatible
OpenGL ES 2 xOpenGL ES 2.xVertex and pixel shaders using GLSL ESAll 2 x will be backwards compatibleAll 2.x will be backwards compatible
F d F (1 )Fixed Function (1.x)
http://www.khronos.org/opengles/2_X/
P bl (2 )Programmable (2.x)
http://www.khronos.org/opengles/2_X/
OpenGL ES 1.x vs 2.0
Tracking
Mobile Augmented Reality’s goal
Create an affordable, massively multi-user, widespread platform
© Tinmith, U. of South Australia
Tracking is…Estimating the device‘s pose (position and orientation) Strictly in real time (30Hz)Strictly in real time (30Hz)
With high spatial precision (1cm, 1 degree)Robustly for operation by human userRobustly for operation by human userNo unrealistic assumptions about HWLeaving enough power to other tasks (interaction graphics)Leaving enough power to other tasks (interaction, graphics)
Tracking requirements for AR on phonesFast and efficientFast and efficientForm factor: light and robustTrack simultaneously
A large number of objectsBy a large number of users
Requires little or no …qDevice modificationManual calibration (targeting non-technical users)( g g )Instrumentation of the physical environment
Low costsLow costs
Tracking on mobile phonesVision-based tracking
Marker-based trackingModel-based natural feature trackingNatural feature tracking in unknown environments
Sensor trackingGPS, inertial compass, gyroscope
Tracking for Handheld ARSLIDE 80
Backpack-based 1.
Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)( ), ( ), y g ( )
Laptop, HMDEnhanced GPS (DGPS / RTK) + inertial sensor for viewpoint trackingEnhanced GPS (DGPS / RTK) + inertial sensor for viewpoint trackingHand tracking w/ fiducial markers
Tracking for Handheld ARSLIDE 81
Backpack-based 2.
Kalkusch et al., 2002Video see-through HMD w/ cameragViewpoint Tracking w/ inside-out computer vision using markersARToolKit markers on walls installed and surveyed manuallyy y
Tablet PC / UMPC-based 1.
Schall et al., 2006Hybrid tracking on UMPC
Camera fiducial marker trackinggWhen no marker in view inertial sensor + UWB tracking
Tablet PC / UMPC-based 2.CAMERA
LEDs
Klein & Drummond, 2004Combining outside in (LED tracking for low accuracy robust pose) & Combining outside-in (LED tracking for low accuracy, robust pose) & inside-out (edge-based tracking for high accuracy) computer vision
PDA-based 1.
BatPortal (Newman et al., 2001) SHEEP (MacWilliams et al., 2003)( , )
PDA as thin client (rendering & tracking on server + VNC)
Tracking by ART (external IR cameras + retroreflective target)
Ultrasonic tracking Projection-based AR environ.
PDA-based 2.
Signpost on PDA (Wagner & Schmalstieg, 2003)
First “truly” handheld AR platform: PDA + cameraStandalone self-contained AR systemStandalone, self contained AR systemOptimized fiducial marker tracking library
History of non-AR Tracking on Phones (1)
AR-PDA (Gausemeier et al., 2003)
Model based tracking
Kick Real (Paelke et al., 2004)
Edge detection of real foot + collisionModel-based trackingPDA = thin client
tracking off-loaded to serverNot real-time
Edge detection of real foot + collision detection w/ virtual ball2D tracking and limited interaction(tailored to game)
History of non-AR Tracking on Phones (2)
PhoneGuide (Bruns et al., 2005) LightSense (Olwal, 2006)
Neural network for recognizing visual features of museum artifactsCombined w/ BT “tracker”O
External camera tracks cell phone LEDSingle user, only coarse position tracking, no orientationBorder case of AROnly object recognition Border-case of AR
History of non-AR Tracking on Phones (3)
Mosquito Hunt (Siemens, 2003)Marble Revolution (BitSide, 2004)Pi i (VTT 2006)Pingis (VTT, 2006)Game control w/ optical flow techniques
TinyMotion (Wang et al., 2006)GUI control & input on cell phones
w/ image differencing & block correlationflow techniques w/ image differencing & block correlation
ARToolKit Tracking (Kato)
ARToolKit - Computer vision based marker tracking libraries
http://artoolkit.sourceforge.net/
History of AR Tracking on Phones (1)
2003ARToolKit on PDAARToolKit on PDAWagner et at.
200420043D Marker on PhoneMöhring et al.g
2005ARToolKit on SymbianHenrysson et al.
Tracking for Handheld ARSLIDE 91
Fiducial marker tracking on handhelds
Möhring et al., 2004 Henrysson et al., 2006Wagner et al., 2003
Rohs, 2006Bucolo et al., 2005
History of AR Tracking on Phones (2)
20052005Visual CodesRohs et atRohs et at.
2008Advanced Marker TrackingAdvanced Marker TrackingWagner et al.
2008Natural Feature TrackingWagner et al.
What can we do on today‘s mobile phones?
Typical specs600+ MHz~5MB of available RAM160x120 - 320x240 at 15-30 Hz camera
Possible to doMarker tracking in 5-15msNatural feature tracking in 20-50ms
Handheld AR Interfaces
Handheld HCI
Consider your userFollow good HCI principlesAdapt HCI guidelines for handheldsAdapt HCI guidelines for handheldsDesign to device constraintsRapid prototyping
User evaluation
Consider Your User■ They are probably mobile■ They are probably mobile
able to use the interface with one hand
■ They want quick access to application content■ They want quick access to application content.Want enhanced interaction with the real world
Interaction with the real world is the main focusInteraction with the real world is the main focus
■ They expect to be able to multitaskstart phone call, use another application, etcp pp
Norman’s Principles of Good PracticeEnsure a high degree of visibility
- allow the user to work out the current state of the system and the f ti ibl range of actions possible.
Provide feedback- continuous, clear information about the results of actions.continuous, clear information about the results of actions.
Present a good conceptual model- allow the user to build up a picture of the way the system holds
h h l i hi b i diff d h together, the relationships between its different parts and how to move from one state to the next.
Offer good mappingsg pp g- aim for clear, natural relationships between actions the user performs
and the results they achieve.
Hi h L l D i G id liHigh Level Design GuidelinesFrom Shneiderman’s 8 desktop design guidelines:p g g
Enable Frequent Users to Use ShortcutsOffer Informative FeedbackDesign Dialogs to Yield Closure
G d T i h’ id liGong and Tarasewich’s guidelines:Design for Small Devices Design for Limited and Split AttentionDesign for speed and recovery Design for speed and recovery Allow for personalizationDesign for EnjoymentDesign for Enjoyment
UI Device Constraints
Comparing Desktop to Handheld Interfaces
Screen Size Input Operation Multimedia Connectivity
Desktop > 1024 x 768 MouseKeyboard
Two handedStationary
Millions of coloursGraphics accel.5.1 Audio
WiredAlways On
Handheld < 640 x 480 StylusTouchButtons
One handedMobile
65K coloursNo graphics accel.Stereo audio
WirelessMaybe On
E m l O2 A ti MExample: O2 Active Menu
Highly visualUse PDA buttons for inputLarge icons and easy to read textVisually indicate which tabs are scrollableApplication UI looks different from device UIApplication UI looks different from device UI
iPhone Guidelines
Minimize required user input. Avoid unnecessary interactivity.y yProvide feedback when necessaryProvide fingertip sized target areasProvide fingertip-sized target areas.Avoid clutter and busy backgrounds. Express essential information succinctly.Make it obvious how to use your content.y
iPhone Interface
Designing for Device ConstraintsInput Device
Touch, stylus, keyboard, buttons, keypady y yp
ScreenSize, resolution,
SensorsCamera – frame rate image resolutionCamera frame rate, image resolutionGPS – resolution, coverageCompass - accuracyCo pass accu acy
Sample Handheld AR InterfacesCleanLarge Video ViewLarge Icons Large Icons Text Overlay
Twitter 360
www.twitter-360.comiPhone applicationppSee geo-located tweets in real worldTwitter com supports geo taggingTwitter.com supports geo tagging
Wikitude – www.mobilizy.com
BlahBlahBl h
BlahBlah
Bl h
BlahBlahBlah
BlahBlah
Bl h
Blah
Blah Blah
Blah
BlahBlahBlah
Blah
BlahBlahBlah
BlahBlah
BlahBlah
Information Filtering
Information Filtering Information Filtering (Julier et al ’00)Information Filtering (Julier et al. ’00)
• Remove clutter by goal- and distance based filtering • User’s task is route finding: Sniper and relevant buildings are displayed; bj t hi h d t i d t b dobjects, which are determined to be unnecessary, removed
W bl AR
HMD vs Handheld AR InterfaceHandHeld ARWearable AR
Output:Output:Display Input &
Output
Input
Handheld Interface MetaphorsTangible AR Lens Viewing
Look through screen into AR scene Interact with screen to interact with AR Interact with screen to interact with AR content
- Eg Invisible Train
Tangible AR Lens ManipulationSelect AR object and attach to device Use the motion of the device as input
- Eg AR Lego
Handheld Display vs Fixed Display
Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasmaUsers performed (1) navigation task, (2) selection taskUsers performed (1) navigation task, (2) selection taskMoving handheld display provided greater perceived FOV, higher degree of presence, faster completion time
J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
Search Task Completion Time
FOV, Presence and ImmersionPerceived FOV and Actual FOV (deg. marked by subjects)
5852
64 606070
33 31
52
30 30 30
45
2030405060
Perceived FOVActual FOV
01020
Motionbased hh
Buttonbased hh
Smallscreen
17' screen 42' screen5.45.76
7
based hh based hh screen4.3 4.1 4.3
4.74.5 4.34.7 4.9
3
4
5
PresenceImmersion
0
1
2
0Motion
based hhButton
based hhSmall
screen17' screen 42' screen
Rapid Prototyping
Speed development time by using quick hardware mockupsp p y g q phandheld device connected to PCLCD screenUSB phone keypadCamera
Can use PC development toolsCan use PC development toolsFlash, Visual Basic, etc
Mobile Physical PrototypingBug Labshttp://www.buglabs.net/
Open source hardware modules, each producing one or more services. p g
Modules snap together physically and the i h l i ll services connect together logically to
enable users to easily build applications.
Software PrototypingPython Symbian (HIT Lab NZ)
stbTracker wrapperstbTracker wrapperAccess to SMS, Bluetooth, GPSRapid developmentRapid development
import e32import appuifwfrom gles import *
if e32.s60_version_info>=(3,0):import imp
t i l d d i ('M t' ' \\ \\bi \\M t d')magnet=imp.load_dynamic('Magnet', 'c:\\sys\\bin\\Magnet.pyd')else:
import Magnetfrom Magnet import
#Define Model
def frameback(num_markers):if (num markers > 1):if (num_markers > -1):
glMatrixMode(GL_PROJECTION)#Draw graphics…
appuifw.app.orientation = 'landscape‘ # Use full frameSetCameraCallback(frameback) # Register callbackcreateCamera() # Define cameraInitGLES() # Start Open GLInitGLES() # Start Open GLTrackerInit() # Start trackerInitCamera() # Start camera
Design GuidelinesApply handheld HCI guidelines for on screen contentApply handheld HCI guidelines for on-screen content
- large buttons, little text input, etcDesign physical + virtual interface elementsPick appropriate interface metaphorpp p p
- “handheld lens” approach using handheld motionTangible AR for AR overlay- Tangible AR for AR overlay
Build prototypesContinuously evaluate application
AR BrowsersAR Browsers
AR BrowsersCommercial outdoor AR applications
Junaio, Layar, Wikitude, etc
All have their own language specificationsWikitude – ARMLJunaio - XML
Need for common standardBased on existing standards for geo-located content etcSupport for dynamic/interactive contentEasier to author mobile AR applicationsEasy to render on AR browsers
Layar
Junaio
Hello World Example" " " " "echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<results><poi id=\"1\" interactionfeedback=\"none\">
<name><![CDATA[Hello World POI]]></name><description><![CDATA[[This is my first POI.]]></description> <l>48.1385,11.5750,0</l>, ,<o>0,0,0</o>
<mime-type>text/plain</mime-type><thumbnail>http://www junaio com/publisherDownload/tutorial/icon jpg</<thumbnail>http://www.junaio.com/publisherDownload/tutorial/icon.jpg</thumbnail><icon>http://www.junaio.com/publisherDownload/tutorial/icon_map.png</icon> icon> </poi></results>"
KHARMA + Argon: A KML/HTML Architecture and Browser for AR Applications and Games
Bl i M I d Al HillBlair MacIntyre and Alex HillSchool of Interactive Computing,
145Georgia Institute of Technology 145
KHARMAKHARMA: KML/HTML Augmented Reality Mobile Architecturemedia hackersresearch tools {
skilled computationalistsour past focus {savvy technical designers
general public
{{
breadth of adoption
general publiccurrent focus{Ygrasil IDE
Designers AR Toolkit
Unity AR Toolkit
146146Designers AR Toolkit
KHARMA
KHARMA: KML/HTML Augmented Reality Mobile ArchitectureProblem: limited authoring tools for mobile ARProblem: limited authoring tools for mobile AR
limited expression (coord, name, desc, link) vs higher hurdle of 3Dno accepted standard and proprietary client protocollimited client-side interactivity is more akin to Web 1 0limited client-side interactivity is more akin to Web 1.0
Solution: HTML with KML combines What? and How? with Where?
• allows extensive client side (albeit 2D) interactivity and expressivityallows extensive client side (albeit 2D) interactivity and expressivity• two the most broadly adopted standards for presentation and geo location• HTTP server distribution, CSS and Javascript allow for true Web 2.0 content
++KML
147
HTML
KHARMA Architecture with four componentsArchitecture with four components
Channel servers- delivering individual channels of AR content,
Tracking servers Tracking servers - providing content related to location
Tracking infrastructure serversTracking, infrastructure servers- delivering information about the physical environment,
Mobile client - for generating the resulting augmentations
KML already supports HTML• description tag accepts CDATA enclosed markup
- but no global styling of scripting support• no control over balloon styling• no control over balloon position and orientation• no relative positioning <Placemark id="culc_center">
<name>CULC Visualization</name><description><
<Placemark id="culc_center"><name>CULC Visualization</name><description><
GPS Located Content<Placemark>
GeospotSurveyed location
<Placemark><name>GeoSpot</name><description>a surveyed GeoSpot </description><Camera> <!-- camera element for GeoSpot -->
Moves camera to fixed location
p<longitude>-84.394135</longitude> <!- GPS coordinates --><latitude>33.76083</latitude><altitude>0</altitude><TimeStamp> <! when GeoSpot was surveyed ><TimeStamp> <!-- when GeoSpot was surveyed -->
<when>1997-07-16T10:30:15+03:00</when></TimeStamp><Icon>
<href>http://geospot.imtc.gatech.edu/image/03_icon.png</href></Icon>
</Camera><Point> <! location displayed on map or within browser view ><Point> <!-- location displayed on map or within browser view -->
<coordinates>-84.394135,33.76083</coordinates></Point>
</Placemark>
For more information, please visit:http://www argon gatech edu/http://www.argon.gatech.edu/
Developing AR Experiences
Sony CSL © 2004
Game Case Study
Resources
ResourcesBooksBooksMobile Interaction Design
M J d G M dMatt Jones and Gary MarsdenDesigning for Small Screens
Studio 7.5Handheld Usability
Scott WeissDesigning the Mobile User Experience
Barbara Ballard
Developer Guidelines
Palmhttp://www.access-company.com/developers/documents/docs/ui/UI_Design.html
Zen of Palm guidelineshttp://www.access-company.com/developers/documents/docs/zenofpalm.pdf
Motorolahttp://developer.motorola.com/docstools/developerguides/
iPhone Human Interface Guidelineshttp://developer.apple.com/documentation/iPhone/Conceptual/iPhoneHIG/
Handheld HCI Design WebsitesDo’s and Don’ts of PocketPC designhttp://www.pocketpcmag.com/_archives/Nov04/Commandements.aspx
U bili i l i h dh ld biliUsability special interest group – handheld usabilityhttp://www.stcsig.org/usability/topics/handheld.html
Usable Mobile websitehttp://www.smartgroups.com/groups/usablemobile
Mobile Coders Websitehttp://www mobilecoders com/Articles/mc-01 asphttp://www.mobilecoders.com/Articles/mc-01.asp
Univ of Waikato Handheld Grouphttp://www.cs.waikato.ac.nz/hci/pdas.html
Mobile Interaction Websitehttp://www.cs.waikato.ac.nz/~mattj/mwshop.html
Platform – Recommended readingLots of low level information on the complete ARM familyp yValuable tool for driver andframework developersframework developersNot that important for pure application developersapplication developers
Platform – Recommended readingVery low level and targeted for PCsMost information outdated on PCMost information outdated on PCEffective memory usage one of the most importantof the most importantoptimization strategies on mobile devices!on mobile devices!
Tracking – Recommended readingLots of the basics on theComputer Vision you will p yneed for AR trackingSeveral code and pseudo-codeSeveral code and pseudo codesnippets
Tracking – Recommended readingAll about the geometryyou will need for a tracking system
Camera modelsProjectionEpipolar geometryH hiHomographies…
Graphics – Recommended reading
Mobile 3D Graphics( ll b t O GL ES 1 )(all about OpenGL ES 1.x)OpenGL ES 2.0
GProgramming GuideOpenGL ES 2.0 Man Pageshttp://www.khronos.org/opengles/sdk/docs/man/ShaderX7Chapter on “ Augmented Reality on Mobile Phones”
OpenGL ES ResourcesKhronos Group OpenGL ES Page
http://www.khronos.org/opengles/
OpenGL ES 2.0 Bookhttp://www.opengles-book.com/
AMDs OpenGL ES 2.0 Emulator
