University of Toronto

earPod:

Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback

Shengdong Zhao

Pierre Dragicevic

Mark Chignell

Ravin Balakrishnan

Patrick Baudisch

Menu Selection

on

Mobile Devices Often Requires

Visual Feedback

Why Eyes-free?

Why Eyes-free?

Eyes-free capable

Eyes-free capable

Marking Menus

Eyes-free capable

Marking Menus (Expert Use)

Eyes-free capable

Marking Menus (Expert Use)

True eyes-free

• TouchPlayer (Pirhonen et al, 2002)

True eyes-free

• TouchPlayer (Pirhonen et al, 2002)

• Head gesture-based auditory pie menu (Brewster et al, 2003)

earPod

13

earPod

14

earPod Design

15

earPod Walkthrough

16

Browsing

Fast browsing

18

Fast browsing

Very fast browsing and cancel

20

Very fast browsing and cancel

Expert use

Expert use

Interactive Voice Response System

24

Interactive Voice Response System

25

Slow

Painful

“Touchtone Hell”

(Yin and Zhai, 2006)

Characteristics of IVR

• Inpatient user – Urgent; want to talk to real person

• Lack of user control– It’s both bad and good

• Typically narrow but deep phone tree, resulting long sentences

Visual vs. auditory menu

27

Visual Linear Menu

Visual vs. auditory menu

28

Visual Linear Menu IVR System

Problems of IVR system

• Speech is serial and slow (Slowiaczek and Nusbaum 1985)

Problems of IVR system

• Synchronous vs. asynchronous communication

Synchronous

Visual Outputs

Mouse Input

time

User

System

Problems of IVR system

• Synchronous vs. asynchronous communication

Synchronous

Visual Outputs

Mouse Input

time

User

System

Asynchronous

time

Audio

PressKey

User

System

WaitingPressKey

Waiting

Audio

Problems of IVR system

• Transition to expert use

Skill development

Ski

ll le

vel

Novice(Voice menu: recognition)

Problems of IVR system

• Transition to expert use

Skill development

Ski

ll le

vel

Expert(direct code access: recall)

Novice(Voice menu: recognition)

Problems of IVR system

• Transition to expert use

Skill development

Ski

ll le

vel

Expert(direct code access: recall)

Novice(Voice menu: recognition)

earPod’s approach

• Problem: speech is serial and slow

earPod’s approach

• Problem: speech is serial and slow– By-pass audio

earPod’s approach

• Problem: speech is serial and slow– By-pass audio – Use interruptible audio feedback

earPod’s approach

• Problem: speech is serial and slow– By-pass audio – Use interruptible audio feedback– Use non-speech audio

earPod’s approach

• Problem: asynchronous communication

earPod’s approach

• Problem: asynchronous communication– Synchronous communication (reactive

audio feedback)

Audio Outputs

Touchpad Input

time

User

earPod’s approach

• Problem: steep novice to expert transition

• Learning menu structure

Smooth transition

• Learning menu structure

• Learning expert gesture

Smooth transition

Target Item

• Learning menu structure

• Learning expert gesture

Smooth transition

Target Item Novice

• Learning menu structure

• Learning expert gesture

Smooth transition

Target Item Novice Novice/Expert

• Learning menu structure

• Learning expert gesture

Smooth transition

Target Item Novice Novice/Expert Expert

47Skill development

Ski

ll le

vel

Expert(direct tapping: recall)

Novice(glide browsing: recognition)

Support Single-hand Usage

48

(Karlson et al, 2007)

Haptic guidance

Haptic guidance

earPod Design Summary

• Touch Input

51

earPod Design Summary

• Touch Input• Reactive & Interruptible Audio

52

earPod Design Summary

• Touch Input• Reactive & Interruptible Audio• Non-speech Audio

53

earPod Design Summary

• Touch Input• Reactive & Interruptible Audio• Non-speech Audio• Input/Output Mapping

54

earPod Design Summary

• Touch Input• Reactive & Interruptible Audio• Non-speech Audio• Input/Output Mapping• Direct Item Access

55

earPod Design Summary

• Touch Input• Reactive & Interruptible Audio• Non-speech Audio• Input/Output Mapping• Direct Item Access• Transition to Expert Use

56

Experiment

57

• earPod vs. iPod-like linear menu

Experiment – visual condition

58

Experiment – audio condition

59

Experimental design

• Within-subject, counter-balanced• 12 participants x• 2 techniques (audio and visual) x• (40+80) items for the 2 menu

configurations (8 and 8x8) x • 4 blocks• = 11,520 menu selections in total.

60

Results

61

Block number Block number

Results

62

Block number Block number

Results

63

Block number Block number

Results

64

Block number Block number

Results

65

Block number Block number

Results

• Detailed Log Analysis– 9/12 reached expert use (1 level menu)– 8/12 reached expert use (2 level menu)

66

Results

• Detailed Log Analysis– 9/12 reached expert use (1 level menu)– 8/12 reached expert use (2 level menu)

• Subjective Feedback – 4 preferred earPod– 4 preferred iPod-like visual menu– 4 had no preferences– 11/12 suggest combine audio & visual

67

earPod limitations

• Work with structured menu of limited breadth

68

earPod limitations

• Work with structured menu of limited breadth

69

Conclusion

• earPod: eyes-free menu selection for structured menu of limited breadth

• Performance comparable to iPod-like linear menus

• Support single-handed interaction• Design for mobile use

70

Future Work

• Long/dynamic menus

71

Future Work

• Long/dynamic menus

• Dual channel menu

72

Future Work

• Long/dynamic menus

• Dual channel menu

• Dichotic listening (Ranjan et al., 2006)

73

Future Work

• Long/dynamic menus

• Dual channel menu

• Dichotic listening (Ranjan et al., 2006)

• IVR on the phone

74

AcknowledgementDanielle Lottridge, Tomer Moscovich, Mike McGuffin,

Jingnan Yang, Heather Thorne, Alvin Chin,

Derek Nowrouzezahrai, Yongchang Zhao, Lisa Min,

Andrew Chignell, Paula Gilverson, Michelle Qu,

Andrew McGutman, Noah Lockwood, John Chattoe,

Annie Xu, Silvia Gonzaloz, Anna Malandrino

Anonymous reviewers

Mobile Usage Design

• Stability issues

76

earPod Novice to Expert Transition

2 61 7

3 5

0

4