COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

August 2nd 2013

Lecture 4: Designing AR Interfaces

Key Points from Lecture 3

Tracking Requirements

  Augmented Reality Information Display   World Stabilized   Body Stabilized   Head Stabilized

Increasing Tracking Requirements

Head Stabilized Body Stabilized World Stabilized

Tracking Types

Magnetic Tracker

Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

Mechanical Tracker

Marker Based Tracking: ARToolKit

http://artoolkit.sourceforge.net/

Markerless Tracking

Magnetic Tracker Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

  No more Markers! Markerless Tracking

Natural Feature Tracking   Use Natural Cues of Real Elements

  Edges   Surface Texture   Interest Points

  Model or Model-Free   ++: no visual pollution

Contours

Features Points

Surfaces

Combining Sensors and Vision   Sensors

-  Produce noisy output (= jittering augmentations) -  Are not sufficiently accurate (= wrongly placed augmentations) -  Gives us first information on where we are in the world,

and what we are looking at   Vision

-  Is more accurate (= stable and correct augmentations) -  Requires choosing the correct keypoint database to track from -  Requires registering our local coordinate frame (online-

generated model) to the global one (world)

Outdoor AR Tracking System

You, Neumann, Azuma outdoor AR system (1999)

The Registration Problem   Virtual and Real must stay properly aligned   If not:

  Breaks the illusion that the two coexist   Prevents acceptance of many serious applications

Sources of registration errors   Static errors

 Optical distortions  Mechanical misalignments   Tracker errors   Incorrect viewing parameters

  Dynamic errors   System delays (largest source of error)

-  1 ms delay = 1/3 mm registration error

Reducing static errors   Distortion compensation   Manual adjustments   View-based or direct measurements   Camera calibration (video)

Reducing dynamic errors (1)

  Reduce system lag   Faster components/system modules

  Reduce apparent lag   Image deflection   Image warping

Reducing System Lag

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

Faster Tracker Faster CPU Faster GPU Faster Display

Predictive Tracking

Time

Position

Past Future

Can predict up to 80 ms in future (Holloway)

Now

AR Design

experiences

applications

tools

components

Building Compelling AR Experiences

Tracking, Display

Authoring

Interaction

Usability

3. Discovering User Needs

Mark Billinghurst

Interaction Design Process

Interaction Design is User Centered

A methodology for ensuring good user experiences with products by getting feedback from users to inform the design.

50

Discovering Needs

• Why: Requirements definition: the stage where failure occurs most commonly

Getting requirements right is crucial

What, How and Why? • What

1. Understand users, task, context 2. Produce a stable set of requirements

• How: Data gathering activities Data analysis activities Expression as ‘requirements’

• Why: Requirements definition: failure occurs most commonly

Getting requirements right is crucial

Iterate

What Are the Requirements?   Functional   What should the system do? What has it done?

  Data   What type of data?

  Environment or Context of use   Physical: dusty? noisy? light? heat? humidity? ….   Social: sharing of files, of displays, work alone ..   Organizational: IT support, hierarchy, training..

  Users: Who are they?   Usability: learnability, throughput, flexibility..

Who REALLY are your Users/Stakeholders?

• Not as obvious as you think: —  those who interact directly with the product —  those who manage direct users —  those who receive output from the product —  those who make the purchasing decision —  those who use competitor’s products

• Three categories of user (Eason, 1987): —  primary: frequent hands-on

—  secondary: occasional or via someone else

—  tertiary: affected by its introduction, or will influence its purchase

Who are the Stakeholders? Check-out operators

Customers Managers and owners

• Suppliers • Local shop owners

Consider the Whole User

Data Gathering

Mark Billinghurst

Data Gathering Techniques (1)

  Questionnaires   Looking for specific information   Qualitative and quantitative results   Good for getting data from a large, dispersed group

  Interviews   Good for exploring issues, using props   Structured, unstructured or semi-structured   But are time consuming and difficult to visit everyone

Data Gathering Techniques (2)

  Workshops or focus groups   Group interviews/activities   Good at gaining a consensus view and/or highlighting areas

of conflict   Observations   Spending time with users in day to day tasks   Good for understanding task context   requires time and can result in a huge amount of data

  Documentation   Procedures and rules written down in manuals

A day in the Life of.. Cultural Probes.. Role Playing..

Case Study: Equator Domestic Probes

4. Design and Prototyping Mark Billinghurst

Interaction Design Process

Elaboration and Reduction   Elaborate - generate solutions. These are the opportunities   Reduce - decide on the ones worth pursuing   Repeat - elaborate and reduce again on those solutions

Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons

The Design Funnel   Alternate generation of ideas and convergence until resolution

Modified from Pugh, S. (1990) Total design: Integrated methods for successful products engineering. Addison-Wesley. P. 75

Design Inspiration

Unmet Needs

  Digital Cameras for Apple Late 80s – What would happen if we married

a computer with a digital camera?

Many studies of current photography use. Uncovering of unmet needs in traditional experience

- not enough photos - not knowing whether photo good or not - ability to add sound - ability to organize

Features determined through research and designed Iteratively.

Brainstorming

  Best with interdisciplinary team

Other Products

  Notice all the iPod look-alikes?

Tools for Effective Design

 Personas  Scenarios  Storyboards  Wireframes and Mock-ups  Prototypes

Persona Technique •  Personas are a design tool to help visualize who you are

designing for and imagine how person will use the product •  A persona is an archetype that represents the behavior and

goals of a group of users •  Based on insights and observations from customer research •  Not real people, but synthesised from real user characteristics •  Bring them to life with a name, characteristics, goals, background •  Develop multiple personas

Gunther the Ad Guy

Gunther is from Germany. He Travels extensively for work and As he is an advertising executive he needs to present concepts to clients quickly and easily. He is a person very well-versed in new technologies and wishes he had easier portable solutions for his presentations…..

Scenarios Usage Scenarios are narrative descriptions of how the product

meets the needs of a persona

Short (2 pages max) Focus on unmet needs of persona Concrete story Set of stories around essential tasks, problems... Use to test ideas

A business woman travels from SF to Paris on a business trip. On her way to the airport she narrowly misses a traffic delay. She avoids the jam because her Smartphone beeps and send her a warning text message on her route from the office to the airport.

Upon arrival the location-sensitive Smartphone notifies the airline that she‘ll check in shortly and an airline employee finds her immediately and takes her baggage. Her display shows that her flight is on time and provides a map to her gate. On the way she downloads tourist information (maps, events) for Paris.

Once found her seat, she begins to review the downloaded information. She books a ticket for an opera she wants to see. Her Smartphone makes the booking using her credit card number stored in memory. The security software of the Smartphone protects her against fraud.

The Smartphone stores the opera booking along with emails written on the plane. As soon as she steps off the plane, it makes the calls and sends the emails. As she leaves the airport, a map appears on the display and guides her to her hotel

Storyboarding

Sequence of sketches showing use of system in everyday use context

Concrete example Easier (faster) to grasp than text based stories Means of communication with users and system

developers Sketches, not drawings... Use to test interaction and make sure design works

Example Storyboard

Example Storyboard p.2

Sketching Mark Billinghurst

Sketching is about design Sketching is not about drawing It is about design.

Sketching is a tool to help you: -  express -  develop, and -  communicate design ideas

Sketching is part of a process: -  idea generation, -  design elaboration -  design choices, -  engineering

Why is sketching useful?   Early ideation   Think through ideas   Force you to visualize how things come together   Communicate ideas to others to inspire new

designs   Active brainstorming

Figure 41: A Sketch of a Dialogue with a Sketch

The “conversation” between the sketch (right bubble) and the mind (left

bubble). A sketch is created from current knowledge (top arrow). Reading,

or interpreting the resulting representation (bottom arrow), creates new

knowledge. The creation results from what Goldschmidt calls “seeing

that” reasoning, and the extraction of new knowledge results from what

she calls “seeing as.”

sketchrepresentation

mind(new) knowledge

Create(seeing that)

Read(seeing as)

54

Sketchbook examples Industrial designer

  ideas   variations   annotations

Source: sketch page from student Industrial designer Samnang Eav

Form factors for wearable computer watch

Sketchbook examples Idea variations

Source: Nicolai Marquardt sketchbook, with permission.

Four different versions of an idea

Sketch examples – design variations

From Carloyn Snyder’s Book: Paper Prototyping (2003) Morgan Kaufmann, p350

Sketch examples – storyboard transitions

From Carloyn Snyder’s Book Paper Prototyping (2003) Morgan Kaufmann, p11

Techniques: Annotations   Augment a sketch

 Directly on sketch   As layer

-  Tracing -  Photoshop layer

 Over dynamic video

Technique: Annotations   Textual notes

 Name and/or explain thinks   Add details   List of items  Questions/issues about design

  Graphical marks  Connect notes to sketch elements   Relates sketch elements   Show dynamics of elements or interaction over time

Technique: Tracing   Basic Idea

-  Copy/trace the fixed elements of an existing interface/screen

  Technology -  Camera, photograph, tracing

paper or.. -  Photoshop or equivalent (trace

over image on a separate layer)

 Drawing skill required -  Almost zero

Technique Hybrid Sketches   Mixes full and low fidelity elements   High fidelity parts:

  Fixed design elements

  Low fidelity parts:   Speculative components

  Contrast  Makes certain parts of the sketch stand out

Your Turn   Design a mobile AR user interface

  Viewing virtual buildings in Christchurch

  Views (Should be able to swap between)  Map View   AR View   List View

Prototyping Mark Billinghurst

68 www.id-book.com

What is a prototype? In interaction design it can be (among other things):

•  a series of screen sketches •  a storyboard, i.e. a cartoon-like series of scenes •  a Powerpoint slide show •  a video simulating the use of a system •  a lump of wood (e.g. PalmPilot) •  a cardboard mock-up •  a piece of software with limited functionality

69 www.id-book.com

Why Prototype? •  Evaluation and feedback are central to interaction design •  Stakeholders can see, hold, interact with a prototype

more easily than a document or a drawing •  Team members can communicate effectively •  You can test out ideas for yourself •  It encourages reflection: very important aspect of design •  Prototypes answer questions, and support designers in

choosing between alternatives

Sketch vs. Prototype Sketch   Prototype  

Invite   A)end  

Suggest   Describe  

Explore   Refine  

Ques;on   Answer  

Propose   Test  

Provoke   Resolve  

Tenta;ve,  non  commi)al   Specific  Depic;on  

The primary differences are in the intent

Types of Prototypes

Low Fidelity – quick and dirty, easy access materials like cardboard and paper.

High Fidelity – more involved electronic versions similar in materials to final product.

RAPID Prototyping

  Fast and inexpensive   Identifies problems before they’re coded   Elicits more and better feedback from users   Helps developers think creatively   Gets users and other stakeholders involved early

  Fosters teamwork and communication   Avoids opinion wars   Helps decide design directions

Paper Prototyping (Low Fidelity)

Quick and simple means of sketching interfaces Use office materials Easier to criticize, quick to change Creative process (develop in team) Can also use for usability test (focus on flow of interaction rather than visuals) Used a lot to test out concepts before real design begins.

Paper Prototyping

Paper Proto: Create Widgets

Paper Prototyping with Hardware

Blank Model Prototypes

Technique: State Transition Diagrams

  Create transition diagram   key interaction steps   branch points

  For each transition   Sketch the screen   Include the transitional diagram   Label the transition with what triggered the

transition (typically user input of set of system responses)

Technique: State Transition Diagrams

Wireframes and Page Semantics

Half-Functional Prototypes

More hi-fi

HTML (or Powerpoint) ...

Mostly precoded flow, but allows clicking and experiencing flow

Wireframing Tools   Pidoco (websites, mobile, desktop)  No programming   http://www.pidoco.com

  WireframeSketcher   Plug-in for Eclipse   http://wireframesketcher.com/

  Omnigraffle (Mac)  Digramming tool   http://www.omnigroup.com/products/omnigraffle/

Screen MockUp Prototypes

HTML Wireframes

Transitions

Technique: Make storyboards come alive   An interactive foam

core and paper sketch/storyboard

Mobile Rapid Prototype Tools

  Android: App Inventor   http://appinventor.mit.edu/

Interactive Role Play

High-fidelity prototyping •  Uses materials that you would expect to be in the

final product. •  Prototype looks more like the final system than a

low-fidelity version. •  For a high-fidelity software prototype common

environments include Macromedia Director, Visual Basic, and Smalltalk.

•  Danger that users think they have a full system…….see compromises

Interactive Role Play in Environment

Tethered Prototypes (High Fidelity)

Goal: Communicate Vision

HIGH FIDELITY PROTOTYE

Rapid Prototyping

  Speed development time with quick hardware mockups   handheld device connected to PC   LCD screen, USB phone keypad, Camera

  Can use PC development tools for rapid development   Flash, Visual Basic, etc

‘Wizard-of-Oz’ Prototyping • The user thinks they are interacting with a

computer, but a developer is responding to output rather than the system.

• Usually done early in design to understand users’ expectations

>Blurb blurb >Do this >Why?

User

Typical Iterations for HW/SW Product:

1.  Paper prototype to evaluate conceptual model 2.  Interactive computer-based prototype with rough

screens to evaluate feature placement 3.  Tethered prototype to evaluate button + screen

interactions 4.  Real device prototyped with major features

working 5.  Real device prototyped with all features working

and graphic design implemented

AR Design Case Study

ITERATIVE DESIGN PROCESS

Design  (redesign)  

Prototype  Test  Design  with  Users  (Note  

problems)  Evaluate  (Fix  Issues)  

BUNRATTY FOLK PARK

BUNRATTY FOLK PARK

 Irish visitor attraction run by Shannon Heritage  19th century life is recreated  Buildings from the mid-west have been relocated to the 26-land surrounding Bunratty Castle  30 buildings are set in a rural or village setting there.

AUGMENTED REALITY

In Bunratty Folk Park:  Allows the visitor to point a camera at an

exhibit, the device recognises it by it’s location and layers digital information on to the display

 3-dimensional virtual objects can be positioned with real ones on display

 Leads to dynamic combination of a live camera view and information

NAVIGATIONAL AID

Smartphone Platform Most people carry mobile phones and are

comfortable with them Ideal Augmented Reality Technology  Global tracking tools  Wireless communication capabilities  Location based computing  Large display for interaction

DESIGNING FOR THE VISITOR

HUMAN CENTRED DESIGN

Goal of the Navigational Aid  Easy to use, clear and understandable  Useful to visitors  Creating interaction between the visitor and the aid

through the user interface

Engage the visitor  It is necessary to understand the visitor to Bunratty  Identify visitor motives and goals while going through

the Folk Park.

HUMAN CENTRED DESIGN

Understanding the technology and Related Works  Literature  Similar Technologies  Electronic Tours in Museum Settings  Interactions design in Outdoor Museums

Understanding the User over time  Observations  Interviews

HUMAN CENTRED DESIGN

Findings  Most visitors do not use the map  Most visitors have mobile phones  Visitors want more information  View the Folk Park at their own pace  Information should be straight to the point  Large social interaction within groups

NEXT STEPS FROM RESEARCH

 Define Visitors Goals  Define Functionalities of the Aid

 Develop Personas – visitors who use the Aid  Develop Scenarios – how the persona uses the navigational aid in the Folk Park  Draw up Storyboards on scenarios

FUNCTIONALITY  View Options  Camera View  Map View  List View

 Sub-Options  Places  Events  Restaurants

 AR Features  Text Information  3D Objects  3D Tour Guide  3D Placement of Buildings

STORYBOARD

ITERATIVE DESIGN PROCESS

ITERATIVE DESIGN PROCESS

Prototyping and User Testing  Low Fidelity Prototyping  Sketches  Paper Prototyping  Post-It Prototyping  PowerPoint Prototyping

 High Fidelity Prototyping  Wikitude

INITIAL SKETCHES - Photoshop

Pros: •  Good for idea generation •  Cheap •  Concepts seem feasible

Cons: •  Not great feedback gained •  Photoshop not fast enough for making changes

POST IT PROTOTYPING

First  DraJ  

Camera  View  with  3D  

Second  DraJ   Third  DraJ  •   Selec;on  highlighted  in  blue   •   Home  bu)on  added  for  easy  

naviga;on  to  main  menu  

POWERPOINT PROTOTYPING Benefits •  Used for User Testing •  Interactive •  Functionalities work •  Quick •  Easy arrangement of slides

User Testing •  Participants found •  15 minute sessions screen captured •  ‘Talk Allowed’ technique used •  Notes taken •  Post-Interview

WIKITUDE

 Popular AR browser  Mapping  Point of Interest abilities  Multiplatform  Shows points of interest  Markers can be selected in and

information pop-up appears

WIKITUDE

User Testing  Application well received

 Understandable

 Participants playful with the technology

FINAL CONCEPT DESIGN

FINAL DESIGN CONCEPT

Key Issues   Fix issues found in previous sessions

 Design with guidelines in mind

 Appealing to the Mental Model   Icon Design

 Aesthetic Design  Colour/Font  Buttons  Look

VIDEO PROTOTYPE

 Flexible  tool  for  capturing  the  use  of  an  interface  

 Elaborate  simula;on  of  how  the  naviga;onal  aid  will  work  

 Does  not  need  to  be  realis;c  in  every  detail  

 Gives  a  good  idea  of  how  the  finished  system  will  work  

Project List   Mobile

  Hybrid Tracking for Outdoor AR   City Scale AR Visualization   Outdoor AR Authoring Tool   Outdoor AR collaborative game   AR interaction for Google Glass

  Non-Mobile   AR Face Painting   AR Authoring Tool   Tangible AR puppeteer studio   Gesture based interaction with AR content

BuildAR Assignment   Download and install Build AR (or HIT Lab NZ)

  http://www.buildar.org/

  Build two AR scenes  Marker based

-  at least 3 markers, 3D models, same theme

  Image based -  at least 2 images, 3D model, text, image/video

  Due August 16th

Project Assignment   Design/Related work exercise   Individual

  Each person find 2 relevant papers/videos/websites  Write two page literature review

  As a team - prototype design   Sketch out the user interface of application  Design the interaction flow/Screen mockups   3 minute Presentation in class August 16th