WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based...
Transcript of WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based...
![Page 1: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/1.jpg)
WiGest: A Ubiquitous WiFi-based Gesture Recognition System
Heba Abdelnasser†, Moustafa Youssef‡, and Khaled A. Harras*Alexandria Univ.†, Egypt-Japan Univ. of Sc. and Tech.‡, Carnegie Mellon Univ.*
![Page 2: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/2.jpg)
Motivation
● Rendering touch input – user is wearing gloves.
● Wet/dirty hands.
![Page 3: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/3.jpg)
Motivation
● Rendering touch input – user is wearing gloves.
● Wet/dirty hands.
● Special sensors● E.g. Samsung Galaxy S5
![Page 4: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/4.jpg)
WiGest Approach
● Free the user from specialized sensors● Using the ubiquitous WiFi● Works with any phone
● Leveraging natural human movements to control devices
![Page 5: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/5.jpg)
Basic Idea
• Leveraging the impact of hand motion on the received WiFion the phone to control WiFi-enabled devices.
![Page 6: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/6.jpg)
WiGest Example
![Page 7: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/7.jpg)
Challenges
• WiFi is a noisy signal• Avoiding false positives – human
interference• Handling the variations of
gestures and their attributes
• Energy consumption• No training
– Handle different humans– Same human a different times
-36
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
-41
-40
-39
-38
-37
-36
-35
-34
0 1 2 3 4 5 6 7 8 9 10
RS
SI
dB
m
Time in Seconds
Far
Near
![Page 8: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/8.jpg)
Agenda
Introduction• WiGest System• Results• Conlusions
![Page 9: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/9.jpg)
Agenda
Introduction• WiGest System• Results• Conlusions
![Page 10: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/10.jpg)
WiGest System
![Page 11: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/11.jpg)
WiGest System
![Page 12: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/12.jpg)
Primitive Extraction
![Page 13: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/13.jpg)
Primitive Extraction
1. Denoising– Using Wavelet denoising
2. Edges detection,– Extract different primitives.
3. Parameters extraction,– Magnitude: near/far.– Speed: fast/slow.
-36
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
Input Output
![Page 14: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/14.jpg)
Noise Reduction
Time in sec
Scale
2.5 3 3.5 4 4.5 5 5.5 6 6.5 7
5
10
15
20
25
30
35
-36
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
![Page 15: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/15.jpg)
Noise Reduction
• Removing noise using wavelet denoising– It can preserve signal details while filtering out the noise
and variations in the signal– Consists of three stages:
• Decomposition the signal to approximation and detail coefficients.
• Thresholding detail coefficients• Reconstruct the denoised signal by adding approximation and
detail after thresholding
![Page 16: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/16.jpg)
Noise Reduction
Raw signal After denoising
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
-36
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
![Page 17: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/17.jpg)
Edges Extraction
• Detecting gesture edges using wavelet analysis– In detail (high-pass filter) coefficients
• Falling edge causes a local maxima• Rising edge causes a local minima
![Page 18: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/18.jpg)
Edges Extraction
Denoised signal
DWT detail coefficient
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
-6
-4
-2
0
2
4
6
1 2 3 4 5 6 7 8
Time in Seconds
![Page 19: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/19.jpg)
Edges Extraction
Denoised signal
DWT detail coefficient
-33
-30
-27
-24
1 2 3 4 5 6 7 8
RS
SI
dB
m
Time in Seconds
![Page 20: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/20.jpg)
Edges Extraction
Denoised signal
DWT detail coefficient
![Page 21: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/21.jpg)
Edges Extraction
Denoised signal
DWT detail coefficientExtracted primitives
![Page 22: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/22.jpg)
Parameters Extraction
Magnitude
-41
-40
-39
-38
-37
-36
-35
-34
0 1 2 3 4 5 6 7 8 9 10
RS
SI
dB
m
Time in Seconds
Far
Near
-39
-38
-37
-36
-35
0 2 4 6 8 10 12 14 16 18
RS
SI
dB
m
Time in Seconds
Slow Fast
Speed
![Page 23: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/23.jpg)
Parameters Extraction
Magnitude
-41
-40
-39
-38
-37
-36
-35
-34
0 1 2 3 4 5 6 7 8 9 10
RS
SI
dB
m
Time in Seconds
Far
Near
-39
-38
-37
-36
-35
0 2 4 6 8 10 12 14 16 18
RS
SI
dB
m
Time in Seconds
Slow Fast
Speed
![Page 24: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/24.jpg)
System Logic Flow
![Page 25: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/25.jpg)
Segmentation
• Silence period determines the start and end of a gesture
• Preamble (unique signature), to start the communication channel
Frequency =
3 /2 Hz,
Count = 2
Frequency =
5 /2 Hz,
Count = 3
-39
-36
-33
-30
0 1 2 3 4 5 6 7 8
RSSI
dBm
Time in SecondsSilence period Gesture delimiter
![Page 26: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/26.jpg)
Pattern Encoding
• The extracted gesture primitives are converted to string sequence:
– Rising edge : +– Falling edge : -– Pause : 0
• Gesture matching– String matching to identify the gesture
Extracted gesture pattern : - + - +Gesture : Infinity
![Page 27: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/27.jpg)
Gesture Identification
Extracted gesture pattern : - + - +Gesture : Infinity
Preamble • Energy efficiency• Human interference
and noise
Pattern encoding• Efficient matching• Error tolerance
![Page 28: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/28.jpg)
Gestures ParametersFrequency =
3 /2 Hz,
Count = 2
Frequency =
5 /2 Hz,
Count = 3
-39
-36
-33
-30
0 1 2 3 4 5 6 7 8
RSSI
dBm
Time in Seconds
Count• Number of gesture
repetitions
• E.g. double click and single click
Frequency• Number of gesture
repetitions per unit time
• E.g. speed of basketball
dribble
![Page 29: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/29.jpg)
System Logic Flow
![Page 30: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/30.jpg)
Action Mapping
• At the end, the extracted gesture is mapped to an
application action .
Gesture Action
Application
![Page 31: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/31.jpg)
Agenda
IntroductionWiGest System• Results• Conlusions
![Page 32: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/32.jpg)
Implementation and Evaluation
• Test environments• Typical apartment• Engineering building at our campus
• More than 1000 experiment• Off-the-shelf WiFi-equipped devices• Hardware:
• Cisco Linksys
• Android-based cell phone.
• HP laptop
39 ft
34 ft
115 ft
49 ft
![Page 33: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/33.jpg)
Experiments Performed
• Distance effect for different scenarios• Orientation• Number of APs• Human interference
• Whole home case study
• More results in the paper
![Page 34: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/34.jpg)
Distance Impact
0.5
0.6
0.7
0.8
0.9
1
3 7 10 13 16 20 23 26 39 52 62-70
-60
-50
-40
-30
Acc
ura
cy
Av
erag
e R
SS
I
Distance in feet
Accuracy Average RSSI
0.5
0.6
0.7
0.8
0.9
1
3 7 10 15 20 26 30 39 56-70
-60
-50
-40
-30
Acc
ura
cy
Av
erag
e R
SS
I
Distance in feet
Accuracy Average RSSI
0.5
0.6
0.7
0.8
0.9
1
7 10 13 16 20 30 36-70
-60
-50
-40
-30
Acc
ura
cy
Av
erag
e R
SS
I
Distance in feet
Accuracy Average RSSI
• 87% accuracy for distances up to 26ft
• Using a single AP
![Page 35: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/35.jpg)
Orientation Impact
• Overall accuracy of 90.5%• The highest accuracy is in West – body not blocking the
signal
0
0.2
0.4
0.6
0.8
1
North South East West Overall
Acc
ura
cy
![Page 36: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/36.jpg)
Number of APs Impact
0
0.2
0.4
0.6
0.8
1
1 APs 3 APs 5 APs 7 APs
Acc
ura
cy
Number of APs used in voting
• Accuracy 96% using 3 APs
• Reaches 100% with seven APs
![Page 37: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/37.jpg)
Human interference
• Accuracy 89% in the presence of four
interfering humans
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4
Det
ecti
on
Acc
ura
cy
Number of interfering humans
![Page 38: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/38.jpg)
Whole Home Media-player Case Study
• Accuracy of 96% using 2 APs
0 .9
0 .96
0 .93
0 .97
0 .99
1
0 .98 0.02
0.01
0.025
0.03
0.04
0.090.005 0.005
0.005
0.005
0.03 0.005
0
0 0 00
0 0000
0 0000
00 0000
0000
000
00
0
0
0
00
00
nu
ll
Ac
tu
al
Ac
tio
n P
erfo
rm
ed
Act ion Classif ied
39 ft
34 ft
![Page 39: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/39.jpg)
Agenda
IntroductionWiGest SystemResults• Conlusions
![Page 40: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/40.jpg)
Conclusion
• WiGest is a gesture recognition system• Robust to noise in the environment and interfering humans• Does not require any training• Energy-efficient• Works with any WiFi-enabled device
• Primitives detection accuracy is 87.5%– Using a single AP
• Accuracy increases to 96% using three overheard APs.
![Page 41: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/41.jpg)
Future Work
• Leveraging CSI for finer grained gestures• Other wireless technologies, e.g. cellular and
Bluetooth• Other applications
![Page 42: WiGest: A Ubiquitous WiFi-based Gesture Recognition System · WiGest: A Ubiquitous WiFi-based Gesture Recognition System Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras*](https://reader035.fdocuments.in/reader035/viewer/2022071113/5feac4c86e66044ca070593b/html5/thumbnails/42.jpg)
Thank You
Come and see our demo tomorrow10am to 1pm
wrc-ejust.org wrc_ejust wrc.ejust