Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the...
Transcript of Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the...
![Page 1: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/1.jpg)
CHI 2011 – May 9, 2011
Your Noise is My Command
Sensing Gestures
Using the Body as
an Antenna
Gabe Cohn1,2
Dan Morris1
Shwetak N. Patel1,2
Desney S. Tan1
1Microsoft Research 2University of Washington
ubicomp lab
![Page 2: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/2.jpg)
2
Beyond the Keyboard and Mouse
![Page 3: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/3.jpg)
3
Speech Recognition
![Page 4: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/4.jpg)
4
Capacitive Touch Screen
![Page 5: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/5.jpg)
5
Skinput Harrison et al., CHI 2010
![Page 6: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/6.jpg)
6
Muscle-Computer Interfaces Saponas et al., CHI 2010
Saponas et al., UIST 2009
Saponas et al., CHI 2008
![Page 7: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/7.jpg)
7
Computer Vision and Depth Cameras
![Page 8: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/8.jpg)
8
![Page 9: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/9.jpg)
9
![Page 10: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/10.jpg)
10
![Page 11: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/11.jpg)
11
Sensing Gestures
Using the Body as
an Antenna
no instrumentation to environment
touch gestures anywhere in the home
minimal instrumentation to user
![Page 12: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/12.jpg)
Typical
“bunny ears”
TV antenna
12
![Page 13: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/13.jpg)
Typical
“teenager”
human antenna
dielectric with
complex geometry
40 Hz – 400 MHz
13
“body antenna effect”
- body area networks (BAN)
- analyzing electrical activity on body
![Page 14: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/14.jpg)
human body antenna
< 200 kHz
electromagnetic (EM) noise
from powerlines and appliances
14
![Page 15: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/15.jpg)
15
Powerline Noise
Energy Monitoring and Activity
Recognition
Gupta et al., Ubicomp 2010
Patel at al., Ubicomp 2007
![Page 16: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/16.jpg)
16
Hypothesis
Powerline noise differs at different
- locations in the home
- positions on the wall
Using the human body antenna to
receive this noise, we can determine
- location in the home
- position on the wall
of a user’s touch gestures
![Page 17: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/17.jpg)
17
Apparatus measure voltage
sample with NI-DAQ
400 kS/s
16-bit resolution
store on laptop hard drive
![Page 18: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/18.jpg)
Core Experiment
10 Homes
• single-family and townhouses
• 1 – 3 floors
• Area: 120 – 290 sq m (µ = 215)
1300 – 3100 sq ft (µ = 2310)
• Built: 1948 – 2006 (µ = 1981)
18
10 Participants
• 5 male / 5 female
• Age: 28 – 61 (µ = 38)
• Weight: 52 – 82 kg (µ = 64)
115 – 180 lbs (µ = 141)
• Height: 150 – 188 cm (µ = 169)
59 – 74 in (µ = 67)
![Page 19: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/19.jpg)
19
Locations
6 locations in each home
5 light switches
2 in same room
1 blank wall
above an outlet
![Page 20: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/20.jpg)
Procedure
6 gestures per location
hold each for 6 seconds
guided by computer
commands
6 locations
6 gestures per location
4 “rounds” (repetitions)
144 total gestures per
participant 20
![Page 21: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/21.jpg)
21
Analysis 5 sec
Segmentation
![Page 22: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/22.jpg)
22
Analysis 5 sec
82 ms Segmentation Windowing
![Page 23: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/23.jpg)
23
Analysis Feature Extraction
Frequency (kHz)
Frequency (kHz)
Time (s)
Am
pli
tud
e (
V)
Po
wer
(dB
) P
ow
er
(dB
)
DC
RMS
Time Domain FFT 0 – 2 kHz
2 Features 332 Features
FFT 0 – 25 kHz FFT 0 – 200 kHz High Freq. Peaks
250 Features 400 Features 18 Features
1002 Total Features per 82 ms window
![Page 24: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/24.jpg)
24
Analysis Machine Learning
1002 features per 82 ms time
window
classification using the Weka
SVM (SMO implementation)
4-fold cross validation
fold by “round”
training/testing sets in different
“rounds”
avoids over-fitting
![Page 25: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/25.jpg)
25
Results Wall Touch
2-class classification
0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
Chance Accuracy
98.5%
![Page 26: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/26.jpg)
26
Results Touch Position on Wall
0% 20% 40% 60% 80% 100%
Chance Accuracy
87.4%
74.3%
All Walls
Blank Walls Only
5-position classification
![Page 27: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/27.jpg)
27
Results Location in Home
0% 20% 40% 60% 80% 100%
Chance Accuracy
Touch Walls
Stand Near Walls
99.1%
99.5%
6-location classification
2 walls in same room
![Page 28: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/28.jpg)
28
Results Location in Home and Position on Wall
30-class problem
0% 20% 40% 60% 80% 100%
Chance Accuracy
79.8%
un-optimized classification on each 82 ms window
![Page 29: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/29.jpg)
29
Summary Core Experiment:
• Location in home - near 100%
• Position on around switch – 87%
Additional Exploration:
• Differentiate right/left hand
• Differentiate appliance touched
• Estimate proximity to wall
• Estimate continuous position on wall
![Page 30: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/30.jpg)
30
Limitations
Improved feature set - relative features
- temporal features
Signal injection - on-body
- into power line
Real-time system
Smaller system (no laptop)
Contact point to body - contactless coupling to body
Explore gesture set
Explore application space
Generalizability of noise signals - Training procedure
- Locations with poor signals
- Strong signals overwhelm others
Changing state of home - Appliances switched on/off
Future Work
![Page 31: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/31.jpg)
31
Your Noise is My Command
Sensing Gestures
Using the Body as an
Antenna
Gabe Cohn
Dan Morris
Shwetak N. Patel
Desney S. Tan
ubicomp lab
Thank You!
www.gabeacohn.com
![Page 32: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/32.jpg)
32
Backup Slides
![Page 33: Your Noise is My CommandCHI 2011 – May 9, 2011 Your Noise is My Command Sensing Gestures Using the Body as an Antenna Gabe Cohn1,2 Dan Morris1 Shwetak N. Patel1,2 Desney 1S. Tan](https://reader034.fdocuments.in/reader034/viewer/2022050514/5f9e819daff80224413d346d/html5/thumbnails/33.jpg)
Applications
Long-term: • Home automation • Gaming
Short-term: • Impromptu interaction session • Meetings/presentations
33