Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †,...
-
Upload
monica-richards -
Category
Documents
-
view
221 -
download
1
Transcript of Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †,...
![Page 1: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/1.jpg)
Snooping Keystrokes with mm-level Audio Ranging on a Single Phone
Presenter: Jian Liu
Jian Liu†, Yan Wang†, Gorkem Kar #, Yingying Chen†, Jie Yang‡, Marco Gruteser#
†Dept. of ECE, Stevens Institute of Technology, USA# Winlab, Rutgers University, USA
‡ Dept. of CS, Florida State University, USA
DAISYData Analysis and Information SecuritY Lab
MobiCom 2015Paris, France
Sep. 9 – 11, 2015
![Page 2: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/2.jpg)
2
Mobile Device Hardware Advancements
Stereo recording
High definition audio capabilities targeted at audiophiles Microphone arrays (stereo recording & noise canceling) 4x improvement in audio sampling rates
Such advancements have security concerns
Audio chipset: 192kHz playback and recording
Mic-1
Mic-2 Mic-3
![Page 3: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/3.jpg)
3
The Results of the Advancements
Facilitating fine-grained localization based applications Tracking speakers in multiparty conversations Sensing touch interaction on surfaces around mobile devices
Eavesdropping keystrokes without suspicion Adding malware into the target user’s phone with microphone access Leaving a phone near a keyboard of the target user
Adding malware with Mics access Leaving a phone
![Page 4: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/4.jpg)
Be careful of these nearby phone!They can hear your typing!
4
![Page 5: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/5.jpg)
5
Related Work
typing has to satisfy English language pattern
require a-priori labeled training data
Label each key for training Multiple recording devices
Linguistic context Training with labeled dataMulti-phone to be placed
around
![Page 6: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/6.jpg)
6
Our Approach
No involvement of multiple phones
No linguistic model
No labeled training (e.g., without any cooperation of the target user)
![Page 7: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/7.jpg)
7
Available Audio Components in a Single Phone
Stereo recording of two microphones High sampling rate
Mic1
Mic2
Stereo recording
Mic3
Stereo 1
Stereo 2 Noise Cancellation
![Page 8: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/8.jpg)
8
What can we obtain from the dual-Mic in a phone to snoop keystrokes?
![Page 9: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/9.jpg)
9
`
Mic1Mic2t1=tt2=t+Δtt1=t’t2=t’+Δt’
Distance difference Δd1
Feature 1: Time Difference of Arrival (TDoA)
Most of the keys could be differentiated by the TDoAs
Theoretical TDoA
Measured TDoA
S L
Distance difference Δd2
![Page 10: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/10.jpg)
Limits of Measured TDoA Dual-Microphone TDoA can only identify a group of keystrokes
Mic1
Mic2
Mic1
Mic2
r1 r2
d
Half hyperbola of constant TDoA
TDoA = Δtr1 – r2 = Δt·v
10
Measured TDoA has the Resolution Limited by Sampling Rate
Sampling by ADCSpeed of sound: 343m/s
![Page 11: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/11.jpg)
Feature 2: Acoustic Signature Keystrokes of different keys sound different MFCCs (Mel-frequency Cepstral Coefficients) can be used to
discriminate sounds of different keys
11
MFCC of key ‘E’ MFCC of key ‘D’ MFCC of key ‘X’
![Page 12: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/12.jpg)
12
We can combine TDoA and acoustic signatures to identify each keystroke!
![Page 13: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/13.jpg)
System Overview
13
A Set of Keystrokes
Keystroke Detection & Segmentation
TDoA DerivationKey Groups Generation
Theoretical TDoA
Theoretical Key Groups
Grouping of Keystrokes
Acoustic Signature Extraction
MFCC-based Clustering with in a
Group
Cluster-based Letter Labeling
Identified Keystrokes
![Page 14: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/14.jpg)
Theoretical Key Groups
14
A theoretical key group – keys having similar theoretical TDoAs
One theoretical key group
Q WA
E R T Y U I O PS D F G H J K L
Z X C V B N M
SortingLink any pair of keys whose
theoretical TDoAs are too similar
![Page 15: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/15.jpg)
Keystroke Grouping
15
[sp − 5ms, sp + 100ms], where sp is starting point
A Set of Keystrokes
Keystroke Detection &
Segmentation
TDoA Derivation
Theoretical Key Groups
Grouping of Keystrokes
Cross-correlation approach
Theoretical key groups
g1 g2 g3 gn
Input keystrokes
![Page 16: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/16.jpg)
Clustering within Each Group & Labeling
16
MFCC features: same key shows higher correlation, while different keys
present lower correlation
Theoretical TDoA
Acoustic Signatur
e Extractio
n
MFCC-based Clustering with
in a Group
Cluster-based Letter Labeling
Identified Keystrokes
A theoretical key group:keystrokes of multiple
keys with similar TDoAs
clustering
Each cluster contains keystrokes of the
same key
Keystroke clusters
1t 2t 3tMean TDoAs
Finding Minimum Distance
Theoretical TDoA
E D X Labeling
![Page 17: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/17.jpg)
Evaluation
17
How robust is the system recovering keystrokes from different keyboards?
What is the performance with different sampling rates?
How does the placement of the phone influence the snooping accuracy?
![Page 18: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/18.jpg)
Experimental Setup
Phone/Recording Device Samsung Galaxy Note 3 (48kHz) External microphones (96/192kHz)
Keyboards Three keyboards with different keystroke sound intensity levels
18
15.3cm
Apple MC184LL/A Microsoft Surface Razer Black Widow Ultimate
![Page 19: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/19.jpg)
Experimental Setup
Data collection Randomly type the 26 keys a-z on keyboards In typical office environments with ambient noise (e.g., heater, air-
conditioner) 3,640 keystrokes are collected
Placements Three typical placements
Evaluation Metric Top-k Accuracy
- identify k candidate keys for each keystroke- whether the pressed keys are among identified key candidates
19
![Page 20: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/20.jpg)
Overall Performance
20
Average Accuracy Average Top-1 Accuracy: 86% Average Top-2 Accuracy: 95% Average Top-3 Accuracy: 98%
All three keyboards have comparable high accuracies
Apple Wire
less
Micr
osoft Surfa
ce
Razer B
lackwidow
00.20.40.60.8
1 k=1 k=2 k=3
Top-k
Accu
racy
![Page 21: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/21.jpg)
Impact of Sampling Rates
21
Top-1 Accuracies 48kHz: 85% 96kHz: 86% 192kHz: 94%
Higher sampling rate improves the recognition accuracy
48 96 1920.5
0.6
0.7
0.8
0.9
1k=1 k=2 k=3
Top-k
Accu
racy
Sampling Rate (kHz)
![Page 22: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/22.jpg)
22
ConclusionShow that a single phone can recover keystrokes by exploiting mm-level TDoA ranging and fine-grained acoustic features
Develop a training-free approach on a single phone that does not require a linguistic model to snoop keystrokes
Extensive experiments with different keyboards & microphones sampling rates demonstrate that our work could achieve sufficient accuracy for keystroke snooping
![Page 23: Snooping Keystrokes with mm-level Audio Ranging on a Single Phone Presenter: Jian Liu Jian Liu †, Yan Wang †, Gorkem Kar #, Yingying Chen †, Jie Yang ‡,](https://reader036.fdocuments.in/reader036/viewer/2022062314/56649ea45503460f94ba8e77/html5/thumbnails/23.jpg)
DAISYData Analysis and Information SecuritY Lab
23
Jian [email protected]
http://personal.stevens.edu/~jliu28/
Thank you!