LECTURE 4: PROTOYPING FOR AUGMENTED REALITY

Mark Billinghurst

AR Summer School February 15th – 19th 2016 University of South Australia

Interaction Design Process

Three Phase Model

Elaboration and Reduction

•  Elaborate - generate solutions. These are the opportunities •  Reduce - decide on the ones worth pursuing •  Repeat - elaborate and reduce again on those solutions

Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons

How can we quickly prototype Augmented Reality experiences

with little or no coding?

Why Prototype?

▪  Quick visual design ▪  Capture key interactions ▪  Focus on user experience ▪  Communicate design ideas ▪  “Learn by doing/experiencing”

Prototyping Tools ▪  Static/Low fidelity ▪  Sketching ▪  User interface templates ▪  Storyboards/Application flows

▪  Interactive/High fidelity ▪  Wireframing tools ▪  Mobile prototyping ▪  Native Coding

Typical Development Steps

▪  Sketching ▪  Storyboards ▪  UI Mockups ▪  Interaction Flows ▪  Video Prototypes ▪  Interactive Prototypes ▪  Final Native Application

Increased Fidelity & Interactivity

Sketching Sketching is not about drawing It is about design. Sketching is a tool to help you:

•  express •  develop, and •  communicate design ideas

Sketching is part of a process:

•  idea generation, •  design elaboration •  design choices, •  engineering

Figure 41: A Sketch of a Dialogue with a Sketch

The “conversation” between the sketch (right bubble) and the mind (left

bubble). A sketch is created from current knowledge (top arrow). Reading,

or interpreting the resulting representation (bottom arrow), creates new

knowledge. The creation results from what Goldschmidt calls “seeing

that” reasoning, and the extraction of new knowledge results from what

she calls “seeing as.”

sketchrepresentation

mind(new) knowledge

Create(seeing that)

Read(seeing as)

“Sketching is about the activity not the result” – Bill Buxton

10

From Sketches to Prototypes •  Sketches: early ideation stages of design •  Prototypes: capturing /detailing the actual design

Sketch vs. Prototype Sketch   Prototype  Invite   A)end  Suggest   Describe  Explore   Refine  Ques;on   Answer  Propose   Test  Provoke   Resolve  Tenta;ve,  non  commi)al   Specific  Depic;on  

The primary differences are in the intent

Sketched Interfaces

▪  Sketch + Powerpoint/Photoshop/Illustrator

GlassSim – http://glasssim.com/

▪  Simulate the view through Google Glass ▪  Multiple card templates

GlassSim Card Builder ▪  Use HTML for card details ▪  Multiple templates ▪  Change background ▪  Own image ▪  Camera view

GlassSim Samples

Application Storyboard

▪  http://dsky9.com/glassfaq/google-glass-storyboard-template-download/

Application Flow

Limitations ▪  Positives ▪  Good for documenting screens ▪  Can show application flow

▪  Negatives ▪  No interactivity/transitions ▪  Can’t be used for testing ▪  Can’t deploy on wearable ▪  Can be time consuming to create

Transitions

▪ Series of still photos in a movie format. ▪ Demonstrates the experience of the product ▪ Discover where concept needs fleshing out. ▪ Communicate experience and interface ▪ You can use whatever tools, from Flash to iMovie.

Video Sketching

UI Concept Movies

Pop - https://popapp.in/

• Combining sketching and interactivity on mobiles

• Take pictures of sketches, link pictures together

Using Pop

Interactive Wireframing ▪  Developing interactive interfaces/wireframes ▪  Transitions, user feedback, interface design

▪  Web based tools ▪  UXpin - http://www.uxpin.com/ ▪  proto.io - http://www.proto.io/

▪  Native tools ▪  Justinmind - http://www.justinmind.com/ ▪  Axure - http://www.axure.com/

Proto.io - http://www.proto.io/ ▪  Web based mobile prototyping tool ▪  Features ▪  Prototype for multiple devices ▪  Gesture input, touch events, animations ▪  Share with collaborators ▪  Test on device

Proto.io - Interface

Wireframe Limitations ▪  No access to sensor data ▪  Camera, orientation sensor

▪  No multimedia playback ▪  Audio, video

▪  Simple transitions ▪  No conditional logic

▪  No networking

Hardware Prototyping

Fake Display

3D print Thingiverse model see http://www.thingiverse.com/thing:65706

Have the social impact of Google Glass without the cost

Build Your Own Wearable

▪  MyVu display + phone + sensors

Beady-i

▪  http://www.instructables.com/id/DIY-Google-Glasses-AKA-the-Beady-i/

Rasberry Pi Glasses

▪  Modify video glasses, connect to Rasberry Pi

▪  $200 - $300 in parts, simple assembly ▪  https://learn.adafruit.com/diy-wearable-pi-near-eye-kopin-video-

glasses

Physical Input Devices ▪  Can we develop unobtrusive input

devices ? ▪  Reduce need for speech, touch pad input ▪  Socially more acceptable

▪  Examples ▪  Ring, ▪  pendant, ▪  bracelet, ▪  gloves, etc

Prototyping Platform

Arduino Kit Bluetooth Shield Google Glass

Example: Glove Input

▪ Buttons on fingertips ▪ Map touches to commands

Example: Ring Input

▪ Touch strip, button, accelerometer ▪ Tap, swipe, flick actions

How it works

Bracelet

Armband

Gloves

1,2,3,4

Values/output

Summary ▪  Prototyping for wearables is similar to mobiles

▪  Tools for UI design, storyboarding, wireframing

▪  Android tools to create interactive prototypes ▪  App Inventor, Processing, etc

▪  Arduino can be used for hardware prototypes ▪  Once prototyped Native Apps can be built

▪  Android + SDK for each platform

Other Tools ▪  Wireframing ▪  pidoco ▪  FluidUI

▪  Rapid Development ▪  Phone Gap ▪  AppMachine

▪  Interactive ▪  App Inventor ▪  WearScript

App Inventor - http://appinventor.mit.edu/

▪  Visual Programming for Android Apps ▪  Features ▪  Access to Android Sensors ▪  Multimedia output

▪  Drag and drop web based interface ▪  Designer view – app layout ▪  Blocks view – program logic/control

App Inventor Designer View

App Inventor Blocks View

Orientation Demo

▪  Use wearable orientation sensor

Prototyping Case Study

Google Glass

Google Glass Prototyping

The Road to Glass

AR Design Case Study

ITERATIVE DESIGN PROCESS

Design  (redesign)  

Prototype  Test  Design  with  Users  (Note  

problems)  Evaluate  (Fix  Issues)  

MOBILE AUGMENTED REALITY FOR SPATIAL NAVIGATION

Sharon Brosnan

0651869

Bachelor of Science in Digital Media Design

BUNRATTY FOLK PARK Sharon Brosnan

0651869

Bachelor of Science in Digital Media Design

BUNRATTY FOLK PARK

• Irish visitor attraction run by Shannon Heritage • 19th century life is recreated • Buildings from the mid-west have been relocated to the 26-land surrounding Bunratty Castle

• 30 buildings are set in a rural or village setting there.

AUGMENTED REALITY In Bunratty Folk Park: •  Allows the visitor to point a camera at an exhibit, the

device recognises it by it’s location and layers digital information on to the display

•  3-dimensional virtual objects can be positioned with real ones on display

•  Leads to dynamic combination of a live camera view and information

NAVIGATIONAL AID

Smartphone Platform Most people carry mobile phones Ideal Augmented Reality Technology •  Global tracking tools •  Wireless communication capabilities •  Location based computing •  Large display for interaction

DESIGNING FOR THE VISITOR

HUMAN CENTRED DESIGN

Goal of the Navigational Aid •  Easy to use, clear and understandable •  Useful to visitors •  Creating interaction between the visitor and the aid

through the user interface

Engage the visitor •  It is necessary to understand the visitor to Bunratty •  Identify visitor motives and goals while going through

the Folk Park.

HUMAN CENTRED DESIGN Understanding the technology and Related Works !  Literature

! Similar Technologies ! Electronic Tours in Museum Settings !  Interactions design in Outdoor Museums

Understanding the User over time !  Observations !  Interviews

HUMAN CENTRED DESIGN

Findings •  Most visitors do not use the map •  Most visitors have mobile phones •  Visitors want more information •  View the Folk Park at their own pace •  Information should be straight to the point •  Large social interaction within groups

NEXT STEPS FROM RESEARCH

• Define Visitors Goals

• Define Functionalities of the Aid

• Develop Personas – visitors who use the Aid

• Develop Scenarios – how the persona uses the navigational aid in the Folk Park

• Draw up Storyboards on scenarios

FUNCTIONALITY • View Options

•  Camera View •  Map View •  List View

•  Sub-Options •  Places •  Events •  Restaurants

! AR Features ! Text Information ! 3D Objects ! 3D Tour Guide ! 3D Placement of Buildings

STORYBOARD

ITERATIVE DESIGN PROCESS

ITERATIVE DESIGN PROCESS

Prototyping and User Testing •  Low Fidelity Prototyping

•  Sketches •  Paper Prototyping •  Post-It Prototyping •  PowerPoint Prototyping

•  High Fidelity Prototyping •  Wikitude

INITIAL SKETCHES - Photoshop

Pros: •  Good for idea generation •  Cheap •  Concepts seem feasible Cons: •  Not great feedback gained •  Photoshop not fast enough for making changes

POST IT PROTOTYPING

First  DraJ  

Camera  View  with  3D  

Second  DraJ   Third  DraJ  •   Selec;on  highlighted  in  blue   •   Home  bu)on  added  for  easy  

naviga;on  to  main  menu  

POWERPOINT PROTOTYPING Benefits •  Used for User Testing •  Interactive •  Functionalities work •  Quick •  Easy arrangement of slides

User Testing •  Participants found •  15 minute sessions screen captured •  ‘Talk Allowed’ technique used •  Notes taken •  Post-Interview

WIKITUDE

• Popular AR browser • Mapping • Point of Interest abilities • Multiplatform •  Shows points of interest

• Markers can be selected in and information pop-up appears

FINAL CONCEPT DESIGN

FINAL DESIGN CONCEPT

Key Issues •  Fix issues found in previous sessions

•  Design with guidelines in mind

•  Appealing to the Mental Model •  Icon Design

•  Aesthetic Design •  Colour/Font •  Buttons •  Look

VIDEO PROTOTYPE

" Flexible  tool  for  capturing  the  use  of  an  interface  

" Elaborate  simula;on  of  how  the  naviga;onal  aid  will  work  

" Does  not  need  to  be  realis;c  in  every  detail  

" Gives  a  good  idea  of  how  the  finished  system  will  work  

www.empathiccomputing.org

@marknb00

mark.billinghurst@unisa.edu.au