A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu,...
-
Upload
wesley-howard -
Category
Documents
-
view
215 -
download
2
Transcript of A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu,...
![Page 1: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/1.jpg)
1
A Robust Method of Detecting Hand Gestures Using
Depth Sensors
Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang
Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on
![Page 2: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/2.jpg)
2
Outline
Introduction
Related Works
The Proposed Method
Experimental Results
Conclusion
![Page 3: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/4.jpg)
4
Introduction
In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant. Medical system, computer games, and human-robot
Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy.
Goal: detect hand gestures with color and depth information
![Page 5: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/5.jpg)
5
Related Works
![Page 6: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/6.jpg)
6
Related Works
Body [4][5] V.S. Hand
Hand Superiority: simple
Inferiority: small scale, low resolution
Strict condition: cluttered background, lighting variation
![Page 7: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/7.jpg)
7
Related Works
Hand gesture recognition Only color[12]
Data glove[7]
Training process[9][10]
Earth Mover’s Distance(EMD)[11]
![Page 8: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/8.jpg)
8
References [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy, vol. 7, pp.
203–205, 1993.
[9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand pose estimation using depth sensors,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, nov. 2011.
[10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7.
[11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096.
[12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincolored objects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379.
![Page 9: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/9.jpg)
9
The Proposed Method
![Page 10: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/10.jpg)
10
The Proposed Method
Gesture Representation
Finger RecognitionFind convex hull Detect fingertip and direction
Hand SegmentationFind hands through color
Separate hands
by k-meansFind palm center
![Page 11: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/11.jpg)
11
Hand Segmentation (I)
Find hands through color
Train skin-color[12], detect face[15], image filtering[16], color threshold
L*a*b color space b
Operate AND on the two images
Minimum depth (10cm)
![Page 12: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/12.jpg)
12
Hand Segmentation (I)--Find hands through color
RGB image Depth image L*a*b color space where b = 2
L*a*b color space where b = 3
Skin color images after ANDoperation
Binary image of hand segmentation
![Page 13: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/13.jpg)
13
Hand Segmentation (II)
Separate hands by k-means k=2
Assignment:
Update:
Threshold of distance between 2 clusters
![Page 14: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/14.jpg)
14
Hand Segmentation (II)--Separate Hands By K-means
![Page 15: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/15.jpg)
15
Hand Segmentation (III)
Find palm center Inscribed circle
Minimum inner distance
Maximum element of inner distances set
![Page 16: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/16.jpg)
16
Finger Recognition (I)
Find convex hull Graham’s scan algorithm
P: the lowest y-coordinate
Sort in increasing order of angle
Point to point is left/right turn
Left-turn: O ; right-turn: X
![Page 17: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/17.jpg)
17
Finger Recognition (II)
Detect fingertip and direction Fingers are long and narrow
Find an isosceles triangle with V V: Every vertex on the convex hull
Set a maximum threshold to the vertex angle
The direction vector is paralleled with the median length of an isosceles triangle
![Page 18: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/18.jpg)
18
Finger Recognition
Blue point : cluster centroid Green point : palm centerRed points : fingertipsYellow curves : hand contourLong lines : finger directions Structures around the hand : convex hull
![Page 19: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/19.jpg)
19
The Proposed Method
Gesture Representation
Finger RecognitionFind convex hull Detect fingertip and direction
Hand SegmentationFind hands through color
Separate hands
by k-meansFind palm center
![Page 20: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/20.jpg)
20
Gesture Representation
All information about hands Palm center location
Finger number
Fingertips location
Finger direction vectors
Gestures Rock-paper-scissors game
Drag images
Grasping, releasing
![Page 21: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/21.jpg)
21
Experimental Results
![Page 22: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/22.jpg)
22
Experimental Results
Use Kinect as input of depth and color images
The detection successful rate can reach 95%.
No matter the hand is horizontally or vertically placed.
![Page 23: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/23.jpg)
23
![Page 24: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/24.jpg)
24
Experimental Results
![Page 25: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/25.jpg)
25
Experimental Results--Shadow Puppetry
![Page 26: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/26.jpg)
26
Conclusion
![Page 27: A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649cf95503460f949ca019/html5/thumbnails/27.jpg)
27
Conclusion
Present a new method to detect hands’ positions and gestures
NO training, NO database
Future works Set a threshold to the distance between the palm center
and the fingers
Add additional sensor devices to overcome no palm detection
Shadow Puppetry project