A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor
description
Transcript of A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor
![Page 1: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/1.jpg)
A New Writing Experience :Finger Writing in the Air Using a Kinect SensorXin Zhang, Zhichao Ye, Lianwen Jin,Ziyong Feng, and Shaojie Xu
MultiMedia, IEEE, 2013
FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR
Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu
IEEE International Conference onMultimedia and Expo Workshops (ICMEW), 2013
![Page 2: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/2.jpg)
2
Outline• Introduction • Related Work• Proposed Method• Experimental Results• Conclusion
![Page 3: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/4.jpg)
4
Introduction• So far most of writing systems still rely on:
• Keyboard• Touch screen• …(Extra devices)
• Essential goal of HCI:
• Making interaction between user and computer more natural
![Page 5: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/5.jpg)
5
Introduction• In this paper:
• Propose a finger-writing-in-the-air system (based on Kinect):
• Using depth, color and motion information
• Real-time
• User-friendly and unconstrained
![Page 6: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/6.jpg)
6
Related Work
![Page 7: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/7.jpg)
Related work• Hand Segmentation
• Skin color:• Gaussian (mixture) model[2]
• Illumination and hand-face overlapping
• Depth:• noise
• Motion:• Motion Cue[3]
• The hand should be the most distinct moving object.
X
X
X
7
![Page 8: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/8.jpg)
Related work• Fingertip Detection
• Curvature[6]
• Template matching[1]
• Geodesic distance
[1] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of JCSC, 16(3):421–436, 2007.
[2] S. L. Phung, A. Bouzerdoum, and D. Chai. Skin segmentation using color pixel classification: Analysis and comparison. IEEE Trans. on PAMI, 27:148–154, 2005.
[3] Jonathan Alon, Vassilis Athitsos, Quan Yuan and Stan Sclaroff. A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation. IEEE Trans. on PAMI, 31:1685–1699, 2009.
[6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Journal of ETRI, 33(3):415–422, 2011.
8
![Page 9: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/9.jpg)
Related work• Writing-in-the-air system [10]:
Hand Segmentation
Data Conversion
Region Clustering
Fingertip Identification
Arm point
Fingertip
K-means
[10] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye and WeixinYang. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in the Air System. In Proc. of IEEE ICIMCS, 2012.
9
![Page 10: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/10.jpg)
10
ProposedMethod
![Page 11: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/11.jpg)
Flow Chart• Hand Segmentation Fingertip Detection
11
![Page 12: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/12.jpg)
12
• DSB-MM segmentation algorithm
Hand Segmentation
![Page 13: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/13.jpg)
13
• Depth Model• Solve the issues:
• lighting • hand-face overlapping• moving background
• Hand D:
Hand Segmentation
R(n) : hand region at frame nω : : growth factor
depth
↑
↑
![Page 14: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/14.jpg)
Hand Segmentation• Depth Model
A static hand A moving hand
14
![Page 15: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/15.jpg)
15
• Skin Model• YCbCr color space
• Quantify Y Component into three regions:• Bright
• Normal
• Dark
• Gaussian classifier[2]:
Hand Segmentation
Reduce the storage size
skin
Non-skin
: mean vector of the i-th skin class covariance of the i-th skin class mean vector of the i-th non-skin covariance of the i-th non-skin class
(Squared Mahalanobis distance)
![Page 16: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/16.jpg)
Hand Segmentation• Skin Model
Color Image Depth Model Skin Model Depth + Skin
16
![Page 17: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/17.jpg)
17
• Background Model• Codebook background model[8] •
Hand Segmentation[8] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.
![Page 18: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/18.jpg)
18
• Background Model• Codebook background model[8] •
Hand Segmentation
Color image A Color image BForeground result A Foreground result B
![Page 19: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/19.jpg)
19
• DSB-MM segmentation algorithm
Hand Segmentation
![Page 20: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/20.jpg)
20
• DSB-MM segmentation algorithm• Each model should have different reliabilities.• Adaptive voting system
• A pixel is kept as hand pixel by
Hand Segmentation
![Page 21: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/21.jpg)
21
• Artificial Neural Network (ANN) • (1) All the models contribute to the final result.• (2) None of them is absolutely reliable.
Hand Segmentation
“1 0 0”, “0 1 0” or “0 0 1” representing 1/3, 1/2 or 2/3
Training:resilient back propagation algorithm (RPROP)
![Page 22: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/22.jpg)
22
Hand Segmentation
Origin Depth Skin Background Mixture
![Page 23: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/23.jpg)
Flow Chart• Hand Segmentation Fingertip Detection
23
![Page 24: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/24.jpg)
24
• Side-mode & Frontal-mode
Fingertip Detection
-- (Red) : Side-modeㄧ (Blue) : Frontal-mode
![Page 25: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/25.jpg)
25
• Side-mode
• Fingertip : the farthest point from the arm point
• Palm point: • Ellipse fitting technique (center point)
• Arm point: • The center of the increased region
Fingertip Detection
![Page 26: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/26.jpg)
26
• Side-mode• The farthest distance to the arm point:
• Side-Mode Criterion:
Fingertip Detection
![Page 27: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/27.jpg)
27
• Frontal-mode• Fingertip : the point with the smallest depth value
Fingertip Detection
![Page 28: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/28.jpg)
28
ExperimentalResults
![Page 29: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/29.jpg)
29
Experimental Results• Intel Core i5-2400 CPU
• 3.10 GHz and 4 Gbytes of RAM
• 20 frames per second(fps)
• 375 videos(44522 frames)
• Recognition of the classifier:• 6763 frequently used Chinese character• 26 English letters (upper case & lower case)• 10 digits
![Page 30: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/30.jpg)
30
Experimental Results• Finger-writing character recognition
• Linking all detected fingertip positions + mean filter• Modified quadratic discriminant function (MQDF) character classifier[9]
[9] T. Long and L. Jin. Building Compact MQDF Classifier for Large Character Set Recognition by Subspace Distribution Sharing. Pattern Recognition, 41(9):2916-2926, 2008.
![Page 31: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/31.jpg)
31
Experimental Results• Error distance (Fingertip detection):
![Page 32: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/32.jpg)
32
Experimental Results
![Page 33: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/33.jpg)
33
Experimental Results
![Page 34: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/34.jpg)
34
Conclusion
![Page 35: A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor](https://reader036.fdocuments.in/reader036/viewer/2022062302/56816385550346895dd46dba/html5/thumbnails/35.jpg)
35
Conclusion• Propose a real-time finger-writing-in-the-air system
• Hand Segmentation:• Depth + Skin + Motion• Adaptive depth threshold of hand region
• Fingertip Detection:• Side-mode• Frontal-mode