Fishing or a Z?: Investigating the Effects of Error onMimetic and Alphabet Device-based Gesture Interaction"
Abdallah ‘Abdo’ El Ali Johan Kildal
Vuokko Lantz
Oct. 23, 2012
h6p://staff.science.uva.nl/~elali/
2
Outline"
I. Introduc5on
II. Methods
III. Results
IV. Discussion
V. Future Work
3
4
Introduction
Introduction"
Device-‐based gestures: Gesturing by moving a smartphone
device in 3D space Research seGngs, home
environments, everyday mobile interac5on
Alterna5ve to when users are encumbered (e.g., manual mul5tasking)
Natural
Promising alterna5ve to mobile touchscreen/keyboard input under situa5onal impairments
5
Motivation"
But errors are an inevitable part of interac5on with technology…
Many gesture classes are available (e.g., iconic, symbolic, deic5c) for use in smartphones, but which have minimum user frustra5on when recogni5on errors occur?
We inves5gate user error tolerance for two iconic gesture sets used in HCI: mime5c and alphabet gestures
6
Related Work"
Gesture-‐based Interac5on Gesture-‐Task mapping (Khnel et al., 2011; Ruiz et al., 2011) Social acceptability (Rico & Brewster, 2010) Gesture Taxonomies: Deic5c, symbolic, physical, mime5c/pantomimic, abstract (‘ \m/ ’)
(Rime & Schiaratura, 1991; …)
Recogni5on Errors Speech: Repeat (Suhm et al., 2001) and hyperar5culate (Oviad et al., 1998) Mul5modal: Modality-‐switching (“spiral depth” of 6) (Oviad & van Gent, 1996) Touch-‐less Vision-‐based Gesture Recogni5on: How many errors before switching to
keyboard input? 40% user error tolerance before modality switching (Karam & Schraefel, 2006)
Lidle work on which gesture sets are most robust to errors during gesture-‐based interac5on!
7
Iconic Gestures"
8
Mimetic / Pantomimic Gestures: natural, familiar, easy to learn, varied by activities e.g., Fishing, calling, …
Alphabet Gestures: familiar, easy to learn, varied by stroke e.g., letter “C”, letter “S”, …
Research Question"
What are the effects of unrecognized gestures on user experience, and what are the differences between mime5c and alphabet gestures (under varying error rates: 0-‐20%, 20-‐40%, 40-‐60%)?
Hypotheses:
Mime5c gestures → users less familiar with ideal shape → more gesture varia5on under high error rates → but lower subjec5ve workload due to higher degrees of freedom
Alphabet gestures → users more familiar with ideal shape → more rigid gestures under increasing error rates → but higher subjec5ve workload due to lower degrees of freedom
9
10
Methods
Mimetic Gesture Design"
11
Alphabet Gesture Design"
12
Study Design" Qualita5ve Study; Automated Wizard-‐of-‐Oz method (Fabbrizio et al., 2005)
24 subjects (16 male, 8 female) aged between 22-‐41 (M= 29.6, SD= 4.5)
Mixed between-‐ and within subject factorial design: 2 (gesture type: mime5c vs. alphabet) x 3 (error rate: low vs. med vs. high)
Experiment in Presenta5on®, Wii Remote® interac5on using GlovePie™
Random error distribu5on across trials (20 prac5ce, 180 test)
Tutorial & videos given of how to 'properly' perform each gesture
Data collected: 1. Modified NASA-‐TLX workload [0-‐20 range] ques5onnaire data (Hart & Wickens, 1990; Brewster,
1994) 2. Experiment logs 3. Video recordings of subjects’ gesture interac5on 4. Post-‐experiment interviews 13
14
!"#$%$&'()*$+,
!" !#
-./&%$0,
'1)23#4&5'$*67.%*6
!" !# !$ !% !& !'
8'()*$+'89':".#),'2&"'()*$+'
;*<'=""*"
-&7'=""*"
>.?3'=""*"'
@'()*$+,'2&"'=""*"'$*67.%*6'A&""*"'$*67.%*6'B'4)*$+'*"7&"'$*C65&"4#)#6$&7'D'"#67*/.E#%*6F
Setup & Procedure"
15
Fishing Gesture
Personal Information
Form
t
≈1 hr
Instructions& Tutorial
PracticeBlock
Test Block
NASA-TLXQuestionnaire
Exit Interview
Reward
x3
Test Block
Perform Fishing Gesture
Perform Fishing Gesture
[...]
Experiment Session
Automated Wizard-of-Oz
16
Low High
MENTAL DEMAND
Low High
PHYSICAL DEMAND
Low High
TIME PRESSURE
Low High
EFFORT EXPENDED
Poor Good
PERFORMANCE LEVEL ACHIEVED
Low High
FRUSTRATION EXPERIENCED
Low High
ANNOYANCE EXPERIENCED
Low High
OVERALL PREFERENCE RATING
#: ____
(Hart & Wickens, 1990)
(Brewster, 1994)
NASA-TLX
17
Results
!"#$%&
'()*+,%&
-../0$
'"0./01%#,"
203*$0%$+/#
'0"."0"#,"
4356",$+7"8/09&/%:
;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"
!!!
!"#$%"&'($)%*+$)
!!!!
!!!!
!!!!!
Workload"
18
!"#$%&
'()*+,%&
-../0$
'"0./01%#,"
203*$0%$+/#
'0"."0"#,"
4356",$+7"8/09&/%:
;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"
!"#$%&'()*'+(,-'+
!!!
!!
!! !!
!!!!!!!!!
!!!!!!
Workload"
19
Workload"
Mime5c gestures are beder tolerated up to error rates of 40% (cf., Karam & Schraefel, 2006), compared with error rates of up to only 20% for alphabet gestures
From a usability perspec5ve, mime5c gestures more robust to recogni5on failures than alphabet gestures
20
Observations"
Mime5c gestures evolve into real-‐world counterparts under error, alphabet gestures tend to become more rigid and well structured
Canonical Varia5ons via posi5ve reinforcement: Survival of the fidest gesture varia5ons E.g., S12 exhibited real-‐world varia5ons on both the Fishing and Trashing
gestures
Varia5ons develop as low as spiral depth of 2 (i.e., min. 2 recogni5on errors)
21
User Feedback" Perceived Canonical Varia5ons
S9: “The shaking, that was the hardest one because you couldn’t just shake freely [gestures in hand], it had to be more precise shaking [swing to the leA, swing to the right] so not just any sort of shaking [shakes hand in many dimensions]”
Cultural and Individual differences S10: “For the glass filling, there are many ways to do it. SomeFmes very fast, someFmes
slow like beer.”
Perceived Performance
Ad hoc explana5ons (e.g., fa5gue) given why there were more errors in some blocks
S18: “[Performance] between the first and second blocks [baseline and low error rate condiFons], it was the same... 10-‐15%.”
Social Acceptability Alphabet gestures less socially acceptable when they fail S16: “When it doesn’t take your C, you keep doing it, and it looks ridiculous.”
22
23
Discussion
Limitations"
Lab-‐study
Task Independence
Random error distribu5on
24
Implications for Gesture Recognition"
Mime5c gestures evolve into real-‐world counterparts under error, alphabet gestures tend to become more rigid and well structured
One shot recogni5on for mime5c gestures important!
Interes5ng explana5ons (e.g., canonical varia5ons) and causes (e.g., fa5gue) given why there were more errors in some blocks
Transparency in gesture recogni5on technology may beder support users in error handling strategies
25
Implications for Gesture-based Interaction"
40% error tolerance in line with previous work (Karam & Schraefel, 2006), which shows usability of gesture-‐based interac5on
Mime5c gestures overall have beder user experience, more use cases, and thus more suitable for device-‐based gesture interac5on (even under high recogni5on error!)
26
27
Future Work
Future Work"
Quan5ta5ve assessment of how many errors precisely before canonical variant?
Other classes of gestures (e.g., manipula5ve)
Influence of device form factor
28
29
Questions
h6p://staff.science.uva.nl/~elali/
References"Brewster, S. Providing a structured method for integra5ng non-‐speech audio into human-‐computer interfaces. PhD
thesis, University of York, 1994.
Fabbrizio, G. D., Tur, G., and Hakkani-‐Tur, D. Automated wizard-‐of-‐oz for spoken dialogue systems. In Proc. INTERSPEECH 2005 (2005), 1857–1860.
Karam, M., and Schraefel, M. C. Inves5ga5ng user tolerance for errors in vision-‐enabled gesture-‐based interac5ons. In Proc. AVI ’06, ACM (NY, USA, 2006), 225–232.
Khnel, C., Westermann, T., Hemmert, F., Kratz, S., Mller, A., and Muller, S. I’m home: Defining and evalua5ng a gesture set for smart-‐home control. Interna5onal Journal of Human-‐Computer Studies 69, 11 (2011), 693 – 704.
Hart, S., and Wickens, C. Manprint: an Approach to Systems Integra5on. Van Nostrand Reinhold, 1990, ch. Workload Assessment and Predic5on, 257–292.
Oviad, S., Maceachern, M., and Anne Levow, G. Predic5ng hyperar5culate speech during human-‐computer error resolu5on. Speech Communica5on 24 (1998), 87–110. 17.
Oviad, S., and Van Gent, R. Error resolu5on during mul5modal human-‐computer interac5on. In Proc. ICSLP ’96, vol. 1 (oct 1996), 204–207.
Rico, J., and Brewster, S. Usable gestures for mobile interfaces: evalua5ng social acceptability. In Proc. CHI ’10, ACM (NY, USA, 2010), 887–896.
Rime, B., and Schiaratura, L. Fundamentals of Nonverbal Behavior. Cambridge University Press, 1991, ch. Gesture and speech, 239–281.
Ruiz, J., Li, Y., and Lank, E. User-‐defined mo5on gestures for mobile interac5on. In Proc. CHI ’11, ACM (NY, USA, 2011), 197–206.
Suhm, B., Myers, B., and Waibel, A. Mul5modal error correc5on for speech user interfaces. ACM Trans. Comput.-‐Hum. Interact. 8 (2001), 60–98.
30
Top Related