Gesture Recognition -...
Transcript of Gesture Recognition -...
![Page 2: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/2.jpg)
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
2
![Page 3: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/3.jpg)
Gesture-‐based interaction, why? The gestures are a natural way to interact with object, tools and other
people As substitution for other forms of communication when other
interactions are not possible ▪ Impaired people ▪ Special context
As complement to other types of interaction modalities
3
![Page 4: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/4.jpg)
A motion of the limbs or body made to express or help express thought or to emphasize speech.
The act of moving the limbs or body as an expression of thought or emphasis.
An act or a remark made as a formality or as a sign of intention or attitude.
A succession of postures.
Own definition (for this seminar): An intentional sign made with the body or limbs to communicate intention or information
4
![Page 5: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/5.jpg)
Gestures Vs Gesticulation Also the gesticulation provides information
Static Vs Dynamic Gestures Static gestures (aka postures, poses,…) Dynamic gestures: a sequence of postures/positions
Multi-‐dimensional gestures 2D gestures 3D gestures Pointing gesture
5
![Page 6: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/6.jpg)
2D gesture 3D gesture Pointing gesture
6
![Page 7: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/7.jpg)
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
7
![Page 8: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/8.jpg)
Dynamic gesture recognition (through computer vision) can be divided in the following main phases: Detection Tracking Gesture segmentation Gesture recognition
Features extraction
8
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 9: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/9.jpg)
Two sub-‐steps: Image acquisition Preprocessing
9
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 10: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/10.jpg)
Image acquisition Mono-‐camera, multi-‐camera, stereo-‐camera, or 3D camera
Camera resolution (low Vs high resolution)
Frames per second
10
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 11: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/11.jpg)
Preprocessing Pixel level segmentation ▪ Color segmentation ▪ Hand detection ▪ Color marker detection
Motion segmentation ▪ Background subtraction ▪ Works good on known background (static background)
▪ Cannot detect stationary hands or determine which moving object is the hand
11
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 12: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/12.jpg)
Preprocessing Contour detection ▪ Not directly depending on skin color and lighting conditions ▪ Can be a large number of objects (even in the background)
Correlation ▪ Problems when objects are rotated or scaled ▪ Problem can be avoided with continuously updating the template
12
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 13: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/13.jpg)
13
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
1
1
2
2
Frame 1
Frame 2
![Page 14: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/14.jpg)
Approaches Kalman filter ▪ Easily computable in real-‐time ▪ Basic form of Kalman filters cannot track objects
on unknown background
Condensation ▪ One of the most used technique for tracking ▪ Detect and track contour of moving objects in a
cluttered environment
CAMshift ▪ Fast, real-‐time ▪ It may be possible to improve accuracy by using
different color representation ▪ There are quite a few parameters
… System without tracking
In controlled environment with a special gesture vocabulary
14
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 15: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/15.jpg)
Gesture segmentation Initial (final) posture When hands are not moving -‐> end of gesture
Gesture decomposition ▪ Preparation, stroke and retraction
Statistical approach ▪ Hidden Markov Model
15
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 16: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/16.jpg)
Potential features: Position, acceleration, velocity Spatial – temporal width FFT of the position …
The features can be extracted in three steps of the process chain
Post-‐processing should be done before providing the features to the GR block
16
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 17: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/17.jpg)
Classification Algorithms: Hidden Markov Model (HMM) ▪ Ergodic HMM, Left-‐Right HMM, Left-‐Right
Banded Hierarchic HMM, Input-‐Output HMM, Parametric HMM, etc. etc. etc.
Conditional Random Fields (CRF) ▪ Hidden CRF, Latent-‐dynamic Discriminative
CRF, etc.
Neural Networks (NN), Decision Trees, Support Vector Machine (SVN), KNN, Dynamic time warping (DTW), etc.
Boosting Algorithms, etc. Hybrid algorithms
17
Featu
res E
xtra
ction
s
Detection
Tracking
Gesture Segmentation
Gesture Recognition
![Page 18: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/18.jpg)
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
18
![Page 19: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/19.jpg)
Design challenges Lighting conditions Provide a feedback to the user Gesture vocabulary (small – large, kind of gesture used, etc.)
Real-‐time interaction Wearable gesture interfaces Multimodality
19
Skinput project
![Page 20: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/20.jpg)
Advantages: Natural way of interaction “Space” effective interaction modality (compared with keyboard and mouse) ▪ Removes the user’s dependency on a surface ▪ Remote interaction
Drawbacks: Tiring (e.g. gorilla arm) User dependent gestures – few universal understandable gestures
Computationally expensive 20
![Page 21: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/21.jpg)
21
Sixthsense G-speak
Natal project
The Project Natal sensor device
![Page 22: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/22.jpg)
Project natal: http://www.xbox.com/en-‐US/live/projectnatal/
Oblong g-‐speak: http://www.oblong.com/ Sixth Sense:
http://www.pranavmistry.com/projects/sixthsense/ Touchless: http://www.codeplex.com/touchless Wiiremotes
(and soon the Play Station Move)
22
![Page 23: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/23.jpg)
Gesture-‐based interaction Characterization
Recognition Typical approach
Design challenges, advantages, drawbacks Applications
Conclusion
23
![Page 24: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/24.jpg)
Gesture based interaction Gesture as interface Gesture characterization ▪ Gesticulation & gesture ▪ Dynamic Vs static gesture ▪ Multi-‐dimensional gesture
Typical approach Challenges, advantages, and drawback
24
![Page 25: Gesture Recognition - Seminardiuf.unifr.ch/main/diva/sites/diuf.unifr.ch.main.diva/files/joomla... · Research Seminar Gesture](https://reader038.fdocuments.in/reader038/viewer/2022103107/5ab458967f8b9a7c5b8bacf0/html5/thumbnails/25.jpg)
25