Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the...
Transcript of Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the...
![Page 1: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/1.jpg)
Gestures without Libraries, Toolkits or Training: A $1 Recognize r for Use r Inte rface Prototypes
Authors: Jacob O. Wobbrock, Andrew D. Wilson, Yang LiPresente r: Spence r Ne lson
![Page 2: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/2.jpg)
Introduction: Background
Gestures becoming more re levant with the spread of touchscreen devices
Gesture recognition has gene rally been the fie ld of AI/Machine learning expe rts, rathe r than HCI/UX experts
This limits the adoption of gestures in consumer products
Previous solutions include librarie s and toolkits, as we ll as “ad-hoc” recognize rs tuned to a prede fined gesture se t
![Page 3: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/3.jpg)
Introduction: Objectives
Create and present a gesture recognition algorithm that is easy to implement in any language , without advanced knowledge
Compare the algorithm ($1) to more theore tically advanced recognize rs
Analyze types of gestures for use with this recognize r
![Page 4: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/4.jpg)
Related Work
Previous work used specialized algorithms to make classifications (Hidden Markov Mode ls, neural ne tworks)
Complex me thods gene rally fall to be ing difficult to work with (debugging, training time , computation cost, ove rfitting)
Othe rs (ad-hoc) have poor scalability and do not allow additional gestures
Toolkit based e fforts work we ll, but only in the language(s) in which they are available
![Page 5: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/5.jpg)
$1 Recognizer: Challenges
![Page 6: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/6.jpg)
$1 Recognizer: Algorithm
$1 is a four step algorithm, best summarized as:
1. Resampling2. Rotate by “indicative angle”3. Scale and Translate4. Optimize best score with rotation re finement
![Page 7: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/7.jpg)
Step 1: Resampling
Goal: e liminate variability due to movement speed during gesture
Takes original points and transforms it into a se t of N equidistant points
Calculate s the length of the path, then steps through each point until length/N-1 distance is cove red, then adds a new point (using linear inte rpolation for be tween point distances)
![Page 8: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/8.jpg)
![Page 9: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/9.jpg)
Step 2: Rotate on Indicative Angle
Goal: e liminate variability on angle of gesture
Calculate s the centroid
Rotate s angle from centroid to first point to 0 degrees
![Page 10: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/10.jpg)
![Page 11: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/11.jpg)
Step 3: Scale and Translate
Goal: Eliminate Variability in size of gesture
Se lects a re fe rence square for all gestures
Bounds the gesture by a rectangle , then scale s rectangle to match square
Translate s gesture to a re fe rence point - moves centroid to origin
![Page 12: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/12.jpg)
![Page 13: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/13.jpg)
Step 4: Optimize angle
Calculate s ave rage distance for C and T for each Ti (Euclidean distance ), converts closest value to a score be tween 0 and 1
![Page 14: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/14.jpg)
Step 4
To optimize the score , the candidate must be rotated to find the optimal angle
Attempted three me thods:
Brute -force with prede fined accuracy (.5 degrees)
Hill Climbing
Golden Section Search
![Page 15: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/15.jpg)
Optimization methods
Hill Climbing was be tte r for similar gestures, but worse for dissimilar gestures when compared to Golden Section Search
Hill Climbing - Maximum 90 ite rations, minimum 0
GSS - Maximum 10 ite rations, minimum 10
![Page 16: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/16.jpg)
Limitations
Rotation, Scale , and Position invariant - Use ful, but also means that two gestures that are represent any combination of these ope rations from each othe r are mutually exclusive
E.g square vs. rectangle , up and down arrows
1-D gestures also require additional e ffort, as any variation in the unused dimension will be scaled significantly
![Page 17: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/17.jpg)
Flexibility
$1 allows de fining of multiple template s by the same name , which allows for more variation in a gesture that all maps to the same gesture function
![Page 18: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/18.jpg)
Evaluation
Used 10 subjects, ente ring gestures on an HP iPAQ h4355 Pocke t PC
Performed 10 se ts of 16 gestures at each slow, medium, and fast speeds
![Page 19: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/19.jpg)
Evaluation
![Page 20: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/20.jpg)
Results
Accuracy of 99.02% (.98% recognition e rrors)
Beats Rubine , and is comparable to DTW
![Page 21: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/21.jpg)
![Page 22: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/22.jpg)
Strengths
Algorithm with ~100 lines of code to implement
Achieves 99+% recognition
Comparable to othe r successful, and more complex, classifie rs
Allows designe rs to add arbitrary gestures at any time
Highly flexible and extendable
![Page 23: Gestures without Libraries, Toolkits or€¦ · rather than HCI/UX experts. This limits the adoption of gestures in consumer products. ... Evaluation. Used 10 subjects, entering gestures](https://reader034.fdocuments.in/reader034/viewer/2022052020/6034d803e934d644bb7accad/html5/thumbnails/23.jpg)
Weaknesses
Limits the gesture space somewhat, but these limitations are re lative ly small
Claims that library/package based attempts are le ss usable