Listening through a Vibration Motor

Nirupam Roy, Romit Roy Choudhury

University of Illinois at Urbana-Champaign

ABSTRACTThis paper demonstrates the feasibility of using the vibra-

tion motor in mobile devices as a sound sensor, almost like

a microphone. We show that the vibrating mass inside the

motor – designed to oscillate to changing magnetic fields –

also responds to air vibrations from nearby sounds. With

appropriate processing, the responses become intelligible,

to the extent that humans can understand the vibra-motor

recorded words with greater than 80% average accuracy.

Even off-the-shelf speech recognition softwares are able to

decode at 60% accuracy, without any training or machine

learning. While these findings are not fundamentally sur-

prising (given that any vibrating object should respond to

air vibrations), the fidelity to which this is possible has been

somewhat unexpected. We present our overall techniques

and results through a system called VibraPhone, and dis-

cuss implications to both sensing and security.

1. INTRODUCTIONVibration motors, also called “vibra-motors”, are smallactuators embedded in all types of phones and wear-ables. These actuators have been classically used toprovide tactile alerts to human users. This paper iden-tifies the possibility of using vibra-motors as a soundsensor, based on the observation that the same movablemass that causes the pulsation, should also respond tochanges in air pressure. Even though the vibra-motoris likely to be far less sensitive compared to the (muchlighter) diaphragm of an actual microphone, the ques-tion we ask is: to what fidelity can the sound be repro-duced?

Even modest reproduction could prompt new applica-tions and threats. On one hand, wearable devices likefitbits, that otherwise do not have a microphone, couldnow respond to voice commands. Further, in devicesthat already have microphones, perhaps better SNRcould be achieved by combining the uncorrelated (noise)properties of the vibra-motor and microphone. On theother hand, leaking sound through vibra-motors opensnew side channels – a malware that has default accessto a phone’s vibra-motor may now be able to eavesdropinto every phone conversation. Toys that have vibra-motors embedded could potentially listen into the am-

bience. This paper is an investigation into the vibra-motor’s e�cacy as a sound sensor, speech in particular.

Our work follows a recent line of work in which mo-tion sensors in smartphones have been shown to detectsound. Authors of Gyrophone [29] first demonstratedthe feasibility of detecting sound signals from the ro-tational motions of smartphone gyroscopes. A recentwork [45] reported how accelerometers may also be ableto detect sound, in fact, classify spoken keywords suchas “OK Google” or “Hello Siri”. Authors rightly iden-tified the applicability to continuous sound sensing –the energy-e�cient accelerometer could always stay ac-tive, and turn on the energy-hungry microphone onlyupon detecting a keyword. While certainly useful, weobserve that these systems run pattern recognition al-gorithms on the features of the signals. The vocabularyis naturally limited to less than 3 keywords, trained bya specific speaker. VibraPhone is attempting a di↵erentproblem altogether – instead of learning a motion sig-nature, it attempts to reconstruct the inherent speechcontent from the low bandwidth, highly distorted out-put of the vibra-motor. Hence, there are no vocabularyrestrictions, and the output of VibraPhone should bedecodable by speech-to-text softwares.

As a first step towards converting a vibra-motor intoa sound sensor, VibraPhone exploits the notion of re-verse electromotive force (back-EMF) in electronic cir-cuits. Briefly, the A/C current in the vibra-motor cre-ates a changing magnetic field around a coil, which inturn causes the vibra-motor mass to vibrate. However,when an external force impinges on the same mass –say due to the pressure of ambient sound – it causesadditional motion, translating into a current in the op-posite direction. This current, called back-EMF, can bedetected through an ADC after su�cient amplification.Of course, the signal extracted from the back-EMF isnoisy and at a lower bandwidth than human speech.However, given that human speech obeys an “acous-tic grammar”, we find an opportunity to recover thespoken words even from the back-EMF’s signal traces.VibraPhone focuses on exactly this problem, and de-velops a sequence of techniques, including spectral sub-traction, energy localization, formant extrapolation, and

1

harmonic reconstruction, to ultimately distill out legiblespeech.

Our experimentation platform is both a Samsungsmartphone and a custom circuit that uses vibra-motorchips purchased online (these chips are exactly the onesused in today’s phones and wearables). We characterizethe extent of signal reconstruction as a function of theloudness of the sound source. Performance metrics aredefined by the accuracy with which the reconstructedsignals are intelligible to humans and to (open-source)automatic speech recognition softwares. We use thesmartphone microphone as an upper bound, and forfairness, record the speech at the same sound pressurelevel (SPL) [23, 4, 40] across all the devices. We exper-iment across a range of scenarios within our universitybuilding, and observe that results are robust/usefulwhen the speaker is less than 2 meters from the vibra-motor.

Finally, we emphasize that smartphone vibra-motorscannot be used as microphones today, primarily becausethe actuator is simply not connected to an ADC. Tothis end, launching side-channel attacks is not immedi-ate. However, as discussed later, we find that enablingthe listening capability requires almost trivial rewiring(just soldering at 4 clearly visible junctions). This papersidesteps these immediacy questions and concentrateson the core nature of the information leakage. At theleast, we hope this work will draw attention to the per-mission policies on vibra-motors, which today are opento all apps by default. We have made various audiodemos of VibraPhone available on our website [5] – werequest the readers to listen to them to better experi-ence the audio e↵ects and reconstructions. In closing,the main contributions in this paper may be summa-rized as:

• Recognizing that ambient sound manifests itself asback-EMF inside vibra-motor chips. This leads toan actuator becoming a sound sensor with minimalchanges to the current mobile device hardware.

• Designing techniques that exploit constraints andstructure of human speech to decode words from anoisy, low bandwidth signal. Building the systemon a smartphone and custom hardware platform,and demonstrating decoding accuracy of up to 88%when a male user is speaking in normal voice nearhis phone.

The rest of the paper expands on these contributions.We begin with a brief introduction to vibra-motors andour hardware platform.

2. UNDERSTANDING VIBRA-MOTORSA vibra-motor is an electro-mechanical device thatmoves a magnetic mass rhythmically around a neutral

position to generate vibrations [34]. While there arevarious kinds of vibra-motors, a popular one is calledLinear Resonant Actuators (LRA) shown in Figure 1.With LRA, vibration is generated by the linear move-ment of the magnetic mass suspended near a coil, calledthe “voice coil”. Upon applying AC current to the mo-tor, the coil also behaves like a magnet (due to thegenerated electromagnetic field) and causes the massto be attracted or repelled, depending on the directionof the current. This generates vibration at the samefrequency as the input AC signal, while the amplitudeof vibration is dictated by the signal’s peak-to-peakvoltage. Thus LRAs o↵er control on both the mag-nitude and frequency of vibration. Most smartphonestoday use LRA based vibra-motors.

Figure 1: Basic sketch of an LRA vibra-motor.

2.1 Sound Sensing through back-EMFBack-EMF is an electro-magnetic e↵ect observed inmagnet-based motors when relative motion occurs be-tween the current carrying armature/coil and the mag-netic mass’s own field. According to Faraday’s law ofelectromagnetic induction [15], this changing magneticflux induces an electromotive force in the coil. Lenz’slaw [39] says this electromotive force acts in the re-verse direction of the driving voltage, called back-EMFof the motor. As the rate of change of the magneticflux is proportional to the speed of the magnetic mass,the back-EMF serves as an indicator of the extraneousvibration experienced by the mass.

Since sound is a source of external vibration, the mov-able mass in the vibra-motor is expected to exhibit a(subtle) response to it. Our experiments show that,when the vibra-motor is connected to an ADC, theback-EMF generated by the ambient sound can berecorded. This is possible even when the vibra-motoris passive (i.e., not pulsating to produce tactile alerts).We call this ADC output vibra-signal to distinguish itfrom the microphone signal that we will later use as abaseline for comparison. We now describe our platformto record and process the vibra-signal.

2.2 Experiment PlatformCustom hardware setup: Today’s smartphones o↵erlimited exposure/API to vibra-motor capabilities andother hardware components (e.g., amplifiers). To by-pass these restrictions, we have designed a custom hard-ware setup using o↵-the-shelf LRA vibra-motor chips

2

connected to our own ADC and amplifier. Figure 2shows our setup – we mount this vibration motor adja-cent to a standard microphone that serves as a compara-tive baseline. The vibra-signal is amplified and sampledat 16KHz. Test sounds include live speech from humansat varying distances, as well as sound playbacks throughspeakers at varying loudness levels.

Microphone

Vibra7onmotor

Amplifiercircuit

Figure 2: The custom hardware setup with col-located vibration motor and microphone.

Smartphones: While the custom hardware o↵ers bet-ter programmability, we also use a smartphone setup tounderstand the possibilities with today’s systems. Fig-ure 3 shows our prototype – terminals of the built-invibra-motor of a Samsung Galaxy S-III smartphone isconnected to the audio line-in input port with a sim-ple wire. The rewiring is trivial – for someone familiarwith the process, it can be completed in less than 10minutes. Once rewired, we collect the samples of thevibra-signal from the output channels of the earphonejack, using our custom Android application.

V.motorpowerport

Enameledwire

Audioport

V.motor

Figure 3: The smartphone setup with a simplewire connected between the vibra-motor’s out-put to the audio line-in port.

3. SOUNDS AND HUMAN SPEECHThis section is a high level introduction to speech pro-duction in humans, followed by a discussion on thestructure of speech signals.

3.1 Human Speech ProductionHuman speech can be viewed as periodic air waves pro-duced by the lungs, modulated through a sequence ofsteps in the throat, nose, and mouth. More specifically,the air from the lungs first passes through the vocalcords – a pair of membranous tissue – that constricts ordilates to block or allow the air flow (Figure 4). Whenthe vocal cords are constricted, the vibrations inducedin the air-flow are called voiced signals. The voiced sig-nals generate high energy pulses – in the frequency do-main, the signal contains a fundamental frequency andits harmonics. All vowels and some consonants like “b”and “g” are sourced in voiced signals.

Figure 4: The vocal cords constricted in (a) anddilated in (b), creating voiced and unvoiced airvibrations, that are then shaped by the glottisand epiglottis.

On the other hand, when the vocal cords dilate and al-low the air to flow through without heavy vibrations,the outcome is called unvoiced signals. This generatessounds similar to noise, and is the origin of certain con-sonants, such as “s”, “f”, “p”, “k”, “t”. Both voicedand unvoiced signals then pass through a flap of tissue,called glottis, which further pulsates to add power to thesignal as well as distinctiveness to an individual’s voice.These glottal pulses travel further and are finally mod-ulated by the oral/nasal cavities to produce fine-tunedspeech [11]. The overall speech production process isoften modeled as a “source-filter” in literature, essen-tially implying that the human trachea/mouth applies aseries of filters to the source sound signal. This source-filter model will later prove useful, when VibraPhoneattempts to reconstruct the original speech signal.

3.2 Structure in Speech SignalsWhile the above discussions present a biological/linguisticspoint of view, we now discuss how they relate to therecorded speech signals and their structures. Figure 5shows the spectrogram when a human user pronouncesthe alphabets “sa” – the signal was recorded through

3

Figure 5: The spectrogram of the spoken conso-nant ‘s’ followed by the vowel ‘a’ recorded withmicrophone.

a smartphone microphone (not a vibra-motor)1. Al-though a toy case, the spectrogram captures the keybuilding blocks of speech structure. We make a fewobservations that will underpin the challenges and thedesigns in the rest of the paper.

• The first visible signal (between 0.6 and 0.75 sec-onds) corresponds to the unvoiced component, theconsonant “s”. This signal is similar to noise withenergy spread out rather uniformly across the fre-quency band. The energy content in this signal islow to moderate.

• The second visible signal corresponds to the vowel“a” and is an example of the voiced component. Thesignal shows a low fundamental frequency and manyharmonics all the way to 4KHz. Fundamental fre-quencies are around 85–180Hz for males and 165–255Hz for females [41]. The energy content of thissignal is far stronger than the unvoiced counterpart.

• Within the voiced signal, the energy content is higherin the lower frequencies. These strong low frequencycomponents determine the intelligibility of the spo-ken phonemes (i.e. the perceptually distinct unitsof sound [42]), and are referred to as formants [27].The first two formants (say, F1, F2) remain between300–2500Hz and completely forms the sound of thevowels, while some consonants have another signif-icant formant, F3, at a higher frequency. Figure 6shows examples of 2 vowel formants – “i” and “a” –recorded by the microphone.

In extracting human speech from the vibra-motor’sback-EMF signal, VibraPhone will need to identify,construct, and bolster these formants through signalprocessing.1The Y axis shows up to 4KHz, since normal human con-versation in non-tonal languages like English is dominantlyconfined to this band.

Frequency (KHz) 1.3 2.7 4

Magn

itude

(dB)

20

25

30

35

40

45

F1 (330 Hz)

F2 (2.53 KHz)

1.3 2.7 4

F1 (800 Hz)F2 (1.46 KHz)

Figure 6: The locations of the first two formants(F1 and F2) for (a) the vowel sound ‘i’ and (b)the vowel sound ‘a’, both recorded with micro-phone.

4. CHALLENGESFigure 7(a,b) compares the spectrogram of the mi-crophone and the vibra-motor for the same spokenphoneme, “sa”. Figure 7(c,d) shows the same com-parison for a full word, namely, “entertainment”. Thereader is encouraged to listen to these sound clips atour project website [5]. Evidently, the vibra-motor’sresponse is weak and incomplete, and on careful analy-sis, exhibits various kinds of distortions even where thesignal is apparently strong. The goal in this paper isto reconstruct, to the extent possible, the left columnsof Figure 7 from the right columns. We face 4 keychallenges discussed next.

Figure 7: The spectrogram for “sa” as recordedby: (a) the microphone and (b) the vibra-motor.The spectrogram for the full word “entertain-ment” as recorded by: (c) the microphone and(d) the vibra-motor. The vibra-motor’s re-sponse is weak and partially missing.

(1) Over-Sensitivity at Resonance FrequencyAll rigid objects tend to oscillate at a fixed natural fre-quency when struck by an external force. When the

4

force is periodically repeated at a frequency close tothe object’s natural frequency, the object shows exag-gerated amplitude of oscillation – called resonance [33].Resonance is often an undesirable phenomenon, desta-bilizing the operation of an electro-mechanical device.Microphones, for example, carefully avoid resonance bydesigning its diaphragm at a specific material, tension,and sti↵ness – that way, the resonance frequencies lieoutside the operating region [18, 10]. In some cases, ad-ditional hardware is embedded to damp the vibrationat the resonant frequencies [10].

Unfortunately, vibra-motors used in today’s smart-phones exhibit sharp resonance between 216 to 232Hz,depending on the mounting structure. Some weak com-ponents of speech formants are often present in thesebands – these components get amplified, appearingas a pseudo-formant. The pseudo-formants manifestas unexpected sounds within uttered words, a↵ect-ing intelligibility. The impact is exacerbated whenthe fundamental frequency of the voiced signal is it-self close to the resonant band – in such cases, thesound itself gets garbled. Figure 8 shows the e↵ect ofresonance when the vibration motor is sounded withdi↵erent frequency tones in succession (called a SineSweep [13, 12]). Observe that for all tones in the SineSweep, the vibra-motor exhibited appreciable responsein the resonance band. This is because the tones havesome frequency tail around the 225Hz, and this alwaysgets magnified. The microphone exhibits no such phe-nomenon. VibraPhone will certainly need to cope withresonance.

Figure 8: The spectrogram of (a) the micro-phone and (b) the vibra-motor, in response to aSine Sweep (i.e., tones played at increasing nar-row band frequencies). The vibration motor sig-nal shows an over-sensitive resonance frequencyband near 220Hz.

(2) High Frequency DeafnessThe vibra-motor’s e↵ective diaphragm – the areaamenable to the impinging sound – is around 10mm,almost 20x larger than that of a typical MEMS mi-crophone (0.5mm). This makes the vibration motordirectional for the high frequency sounds, i.e., thehigh frequencies arriving from other directions are sup-pressed, somewhat like a directional antenna. Unfortu-

nately, human voices contain lesser energy at frequen-cies higher than 2KHz, thereby making the vibra-motoreven less e↵ective in “picking up” these sounds. Someconsonants and some vowels – such as “i” and “e” –have formants close to or higher than 2KHz, and areseverely a↵ected. Figure 9 compares the spectrogramwhen just the vowel “a” was spoken – evidently, thevibra-motor is almost “deaf” to higher frequencies.

Figure 9: The spectrogram of the spokenvowel ‘a’ recorded with (a) microphone and(b)vibration motor. The vibra-motor exhibitsnear-deafness for frequencies > 2KHz.

(3) Higher Energy ThresholdA microphone’s sensitivity, i.e., the voltage producedfor a given sound pressure level, heavily depends on theweight and sti↵ness of its diaphragm. The spring-massarrangement of the vibra-motor is considerably moresti↵, mainly due to the heavier mass and high springconstant. While this is desirable for a vibration actua-tor, it is unfavorable to sound sensing. Thus, using theactuator as a sensor yields low sensitivity in general,and particularly to certain kinds of low-energy conso-nants (like f, s, v, z), called fricatives [17]. The e↵ectis visible in Figure 7 (a,b) – the fricative consonant “s”goes almost undetected with vibra-motors.

(4) Low Signal-to-Noise Ratio (SNR)In any electrical circuit, thermal noise is an unavoidablephenomena arising from the Brownian motion of elec-trons in resistive components. Fortunately, the low 26Ohm terminal resistance in vibra-motors leads to 10dBlower thermal noise than the reference MEMS micro-phone. However, due to low sensitivity, the strength ofthe vibra-signal is significantly lower, resulting in poorSNR across most of the spectrum. Figure 10 comparesthe SNR at di↵erent sound pressure levels2 – exceptaround the resonance frequencies, the SNR of the vibra-signal is significantly less compared to the microphone.

5. SYSTEM DESIGNOur system design is modeled as a source-filter, i.e., wetreat the final output of the vibra-motor as a result of2Note that sound pressure level is a standard metric to mea-sure the e↵ective pressure caused by sound waves, and istypically expressed in dBSPL [4].

5

Frequency (KHz) 0 5 10 15 20

SNR

(dB)

-10

0

10

20

30

4070 dB SPL65 dB SPL55 dB SPL

Frequency (KHz) 0 5 10 15 20

SNR

(dB)

-5

0

5

10

15

2070 dB SPL65 dB SPL55 dB SPL

Figure 10: The SNR of (a) the microphone and(b) the vibra-motor at various frequencies forvarying sound pressure levels (dB SPL). Notethe unequal Y axis range.

many filters applied serially to the original air-flow fromthe lungs. Figure 11 illustrates this view, suggestingthat an ideal solution should perform two broad tasks:(1) “undo” the vibra-motor’s distortions for signal com-ponents that have been detected, and (2) reconstructthe undetected signals by leveraging the predictablespeech structure in conjunction with the slight “signalhints” picked up by the vibra-motor. VibraPhone re-alizes these tasks through two corresponding modules,namely, signal pre-processing and partial speech synthe-sis. We describe them next.

Voicedspeechsource

Unvoicedspeechsource

OR X X

Vocaltractresponse

Vibra7onmotorresponse

Originalspeech

Recordedspeech

Sourcesound

Figure 11: The source-filter model of the speechgeneration and recording.

5.1 Signal Pre-processingAll of our algorithms operate on the frequency domainrepresentation of the signal. Therefore, we first convertthe amplified signal to the time-frequency domain usingthe Short Time Fourier Transform (STFT), which basi-cally computes the complex FFT coe�cients from 100millisecond segments (80% overlapped, Hanning win-dowed) of the input time signal. The result is a 2D ma-trix that we call time-frequency signal and illustratedin Figure 12 – each column is a time slice and each rowis a positive frequency bin. We will refer to this matrixfor various explanations.

Frequency Domain EqualizationWhen a microphone is subject to a Sine Sweep test,the frequency response is typically flat, meaning thatthe microphone responds almost uniformly to each fre-quency component. The vibra-motor’s response, on theother hand, is considerably jagged, and thereby induces

Figure 12: 2D time-frequency matrix

distortions into the arriving signal. Figure 13 shows acase where the vowel “u” is recorded by both the micro-phone and vibra-motor. The vibra-motor distortions on“u” are quite dramatic, altering the original formants at266 and 600Hz to new formants at 300Hz and 1.06KHz.In fact, the altered formants bear resemblance to thevowel “aa” (as in “father”), and in reality, do soundlike it. More generally, the vibra-motor’s frequency re-sponse exhibits this rough shape, thereby biasing all thevowels to the sound of “aa” or “o”.

Frequency (KHz) 1.3 2.7 4

Magn

itude

(dB)

10

15

20

25

30

35

40

45F1 (266 Hz)

F2 (600 Hz)

1.3 2.7 4

F1 (330 Hz)

F2 (1.06 KHz)

Figure 13: Formants of vowel ‘u’ recordedthrough (a) microphone and (b) vibra-motor.The vibra-motor introduces a spurious formantnear 1KHz.

Fortunately, the frequency response of the vibra-motoris only a function of the device and does not changewith time (at least until there is wear and tear of thedevice). We tested this by computing the correlationof the Sine Sweep frequency response at various soundpressure levels – the correlation proved strong, exceptfor a slight dip at the resonant frequencies due to thenon-linearities. Knowing the frequency response, we ap-ply an equalization technique, similar to channel equal-ization in communication. We estimate the inverse gainby computing the ratio of the coe�cients from the mi-crophone and the vibra-motor, and multiply the inversegain with the frequency coe�cients of the output signal.

Background Noise RemovalDeafness in vibra-motors implies that the motor’s re-sponse to high frequency signals (i.e., > 2KHz) is in-distinguishable from noise. If this noise exhibits a sta-

6

tistical structure, a family of spectral subtraction algo-rithms can be employed to improve SNR. However, twoissues need attention. (1) The pure noise segments inthe signal needs to be recognized, so that its statisti-cal properties are modeled accurately. This means thatnoise segments must be discriminated from speech. (2)Within the speech segments, voiced and unvoiced seg-ments must also be separated so that spectral subtrac-tion is only applied on the voiced components. This isbecause unvoiced signals bear noise-like properties andspectral subtraction can be detrimental.

To reliably discriminate the presence of speech seg-ments, we exploit the exaggerated behavior in the res-onance frequency band. We consistently observed thatspeech brings out heavy resonance behavior in vibra-motors, while noise elicits a muted response. Thus,resonance proved to be an opportunity. Once speechis segregated from noise, the next step is to isolate thevoiced components in speech. For this, we leverage itswell-defined harmonic structure. Recall the 2D matrixin Figure 12. We consider a time window and slideit up/down to compute an autocorrelation coe�cientacross di↵erent frequencies. Due to the repetition ofthe harmonics, the autocorrelation spikes periodically,yielding robust detection accuracy. When autocorre-lation does not detect such periodic spikes, they aredeemed as the unvoiced segments.

The final task of spectral subtraction is performed onthe voiced signal alone. For a given voiced signal (i.e.,a set of columns in the matrix), the closest noise seg-ments in time are selected – these noise segments areaveraged over a modest time window. Put di↵erently,for every frequency bin, the mean noise floor is com-puted, and then subtracted from the corresponding binin the voiced signal. For zero mean Gaussian noise, thisdoes not o↵er any benefit, however, the noise is oftennot zero mean. In such cases, the SNR improves andalleviates the deafness. Figure 14 shows the beneficiale↵ect of spectral subtraction when “yes” is spoken.

Figure 14: The spectrogram of the spoken word“yes” (a) before and (b) after the spectral sub-traction.

Speech Energy LocalizationObserve that noise removal described above brings themean noise to zero, however, noise still exists and the

SNR is still not adequate. In other words, deafness isstill a problem. However, now that noise is zero meanand Gaussian, there is an opportunity to exploit its di-versity to further suppress it. Even localizing the speechsignal energy in the spectrogram would be valuable,even if the exact signal is not recovered in this step.

Our core idea is to average the signals from within afrequency window, and slide the frequency window allthe way to 10KHz. Referring to the 2D matrix, wecompute the average of W elements in each column (Wbeing the window size), and slide the window vertically;the same operation is performed for each column. Eachelement is a complex frequency coe�cient, containingboth the signal and the noise. With su�ciently largeW , the average converges to the average of the signalcontent in these elements since the (average) noise sumup to zero. Mathematically, if Ci denotes the signalat frequency fi, and Ci = Si + Ni, where Si is thespeech signal and Ni the noise, then the averaged C⇤

i iscomputed as:

C⇤i =

1

W

i+W2X

f=i�W2

Ci =1

W

i+W2X

f=i�W2

Si+1

W

i+W2X

f=i�W2

Ni (1)

Since the termP

Ni is zero mean Gaussian, it ap-proaches zero for larger W , while the 1

W

PSi term is

simple smoothing. For every frequency bin, we nor-malize the C⇤

i values over a time window so that theyrange between [0, 1]. The result is a 3D contour map,where the locations of higher elevations, i.e., hills, in-dicate the presence of speech signals. We identify thedominant hills and zero force all areas outside them.This is because speech signals always exhibit a largetime-frequency footprint, since human voice is not ca-pable of producing sounds that are narrow in frequencyand time. Figure 15 illustrates the e↵ect of this scheme– the dominant hills are demarcated as the location ofspeech energy. Evidently, the improvement is conspic-uous after this energy localization step.

5.2 Partial Speech SynthesisOnce the vibra-motor output has been pre-processed,the structure of speech can now be leveraged for signalrecovery – we describe our approach next.

Voice Source ExpansionAfter the localization step above, we know the locationof speech energy (in time-frequency domain), but wedo not know the speech signal. In attempting to re-cover this signal, we exploit the opportunity that thefundamental frequencies in speech actually manifest inhigher frequency harmonics. Therefore, knowledge ofthe lower fundamental frequencies can be expanded toreconstruct the higher frequencies. Unfortunately, the

7

0.1 0.3 0.6

Freque

ncy (K

Hz) 3

2.5

2

1.5

1

0.5

Time(s)0.1 0.3 0.6 0.1 0.3 0.6

Figure 15: Readers are requested to view thisfigure in color: (a) Raw vibra-motor signal,(b) The output of the speech energy localiza-

tion makes the signal energy visible through aheat-map like contour. (c) The correspondingmicrophone signal bearing good resemblance tothe energy locations.

actual fundamental frequency often gets distorted bythe resonant bands.

As a workaround, we use the relatively high SNR sig-nals in the range [250, 2000Hz] to synthesize the voicesource signal at higher frequencies. Synthesis is essen-tially achieved through careful replication. Specifically,the algorithm copies the coe�cient Ct,f , where t is thetime segment and f is the frequency bin of the time-frequency signal, and adds it to Ct,kf for all integer k,such that kf is less than the Nyquist frequency. Here in-teger k indicates the harmonic number for the frequencyf . Intuitively, we are copying the harmonics from thereliable range, and replicating them at the higher fre-quencies. As shown in Figure 16, this only synthesizesthe voiced components (recall the harmonics are onlypresent in the voiced signals). For unvoiced signals, weblindly fill in the deaf frequencies with copies of thelower frequency signals.

Time (Sec) 0.1 0.4 0.6 0.9 0.1 0.4 0.6

Freq

uenc

y (K

Hz)

3.53.02.52.01.51.00.5

0

Time (Sec) 0.1 0.4 0.6 0.9 0.1 0.4 0.6

Freq

uenc

y (K

Hz)

3.53.02.52.01.51.00.5

0

Figure 16: Result of source expansion for thevoiced signal components. (a) Raw vibra-signal,and (b) after harmonic replication. Readers arerequested to view this figure in color.

Speech ReconstructionRecall that the mouth and nasal cavities finally modu-late the air vibrations – this can be modeled as weightsmultiplied to the fundamental frequencies and their har-monics. While we do not know the values of theseweights, the location of the energies – computed from

the 3D contour hills – is indeed an estimate. We nowutilize this location estimate as an energy mask. As afirst step, we apply an exponential decay function alongthe frequency axis to model the low intensity of natu-ral speech at the higher frequencies. Then the energymask is multiplied with this modified signal, emulatingan adaptive gain filter. As this also improves the SNR ofthe unvoiced section of the speech, we apply a deferredspectral subtraction method on these segments to fur-ther remove the background noise. Finally, we convertthis resultant time-frequency signal to time domain us-ing inverse short time Fourier transform (ISTFT). Fig-ure 17 compares the output against the microphone andthe raw vibra-motor signal.

-1

0

1

Ampl

itude

-1

0

1

Time (Sec) 0.2 0.4 0.6 0.8

-1

0

1

Figure 17: Word “often” as manifested in the (a)raw vibra-motor signal, (b) after VibraPhone’sprocessing, and (c) microphone signal.

6. EVALUATIONSection 2.2 described the two experimentation plat-forms for our system, namely the custom hardwareand the Samsung Galaxy smartphone. In both cases,we evaluate VibraPhone’s speech intelligibility againstthe performance of the corresponding microphone. Inthe custom hardware, the microphone is positionedright next to the vibra-motor, while in the smartphone,their locations are modestly separated. We generatethe speech signals using a text-to-speech (TTS) utilityavailable in OS X 10.9, and play them at di↵erent vol-umes through a loudspeaker. The position/volume ofthe loudspeaker is adjusted such that the sound pres-sure levels at the vibra-motor and the microphone areequal. The accent and intonation of the TTS utility alsodoes not a↵ect the experiment since both VibraPhoneand the microphone hear the same TTS speech. Thecontent of the speech is drawn from Google’s TrillionWord Corpus [3] – we pick 2000 most frequent words,which is prescribed as a good benchmark [32].

8

6.1 Methodology and MetricsWe perform automatic and manual speech recognitionexperiments as follows.

(1) Automatic Speech Recognition (ASR)In ASR, a software programmatically converts the timedomain speech signal to text. ASR tools typically have3 distinct components: (a) an acoustic model, (b) a pro-nunciation dictionary, and (c) a language model. Theacoustic model is a trained statistical model (e.g., HMM,Neural Networks, etc. [14, 19]) that maps segments ofthe input waveform to a sequence of phonemes. Thesephonemes are then looked up in the pronunciation dic-tionary, which lists the candidate words (along withtheir possible pronunciations) based on the matchingphoneme sequence. Among these candidates, the mostlikely output is selected using a grammar or a languagemodel.

Our ASR tools is the open-source Sphinx4 (pre-alphaversion) library published by CMU [1, 20]. The acousticmodel is sensitive to the recording parameters, includ-ing the bandwidth and the features of the microphone.Such parameters do not apply to vibra-motors, so weused a generic acoustic model trained with standard mi-crophone data. This is not ideal for VibraPhone, andhence, the reported results are perhaps a slight under-estimate of VibraPhone’s capabilities.

(2) Manual Speech Recognition (MSR)We recruited a group of 6 volunteers from our depart-ment building – 1 native English speaker, 1 Indian fac-ulty with English as first language, 2 Indian PhD stu-dents, and 2 Chinese PhD students. We played thevibra-motor and microphone outputs to all the partici-pants simultaneously and collected their responses. Insome experiments, volunteers were asked to guess theword or phrase from the playback; in other experiments,the volunteers were given a list of phrases and asked topick the most likely one, including the option of “noneof the above”. All human responses were accompaniedby a subjective clarity score – every volunteer expressedhow intelligible the word was, even when he/she couldnot guess with confidence. Finally, in some experi-ments, volunteers were asked to guess first, and thenguess again based on a group discussion. Such discus-sions served as a “prior” for speech recognition, andoften the group consensus was di↵erent from the firstindividual guess.

MetricsAcross all experiments, 9 hours of sound was recordedand a total of 20,000 words were tested with ASR atvarious sound pressure levels (measured in dBSPL).For MSR, a total of 300 words and 40 phrases wereplayed, resulting in more than 2000 total human re-

sponses. We report “Accuracy” as the percentage ofwords/phrases that were correctly guessed, and showits variation across di↵erent loudness levels (measuredin dBSPL). We report “Perceived Clarity” as a subjec-tive score reported by individuals, even when they didnot decode the word with confidence. Finally, we report“Precision”, “Recall”, and “Fallout” for experiments inwhich the users were asked to select from a list. Re-call that precision intuitively refers to “what fraction ofyour guesses were correct”, and recall intuitively means“what fraction of the correct answers did you guess”.We now present the graphs, beginning with ASR.

6.2 Performance Results with ASR

Accuracy v/s LoudnessCustom Hardware: Figure 18(a) reports the accu-racy with ASR as a function of the sound pressure level(dbSPL), a standard metric proportionally related tothe loudness of the sound. VibraPhone’s accuracy isaround 88% at 80 dbSPL, which is equivalent to thesound pressure experienced by the smartphone’s mi-crophone during typical (against the ear) phone call.The microphone’s accuracy is obviously better at 95%,while the raw vibra-motor signal performs poorly at43% (almost half of VibraPhone). Importantly, thepre-processing and the synthesis gains are individuallysmall, but since intelligibility is defined as binary metrichere, the improvement jumps up when applied together.

Once the loudness decreases at 60 dbSPL – compara-ble to a normal conversation 1 meter away from themicrophone [2] – VibraPhone’s accuracy drops to ⇡60%. At lower sound pressure level, the accuracy dropsfaster since the vibra-motor’s sensitivity is inadequatefor “picking up” the air vibrations. However, the accu-racy can be improved with training the acoustic modelwith vibra-motors (recall that with ASR, the trainingis performed through microphones, which is unfavorableto VibraPhone).

Sound pressure level (dbSpl) 40 50 60 70 80

Accu

racy

(%)

0

20

40

60

80

100

MicrophoneProcessed vib.Raw vib.

Sound pressure level (dbSpl) 40 50 60 70 80

Accu

racy

(%)

0

20

40

60

80

100

MicrophoneProcessed vib.Raw vib.

Figure 18: Automatic recognition accuracy asa function of loudness for (a) the custom hard-ware, (b) the Samsung smartphone.

Samsung Smartphone: Figure 18(b) plots the accu-racy with ASR for the smartphone based platform. Vi-braPhone’s performance is weaker compared to the cus-tom hardware setup, although the di↵erence is marginal

9

– ASR output is still at 80% at 80 dbSPL. Admittedly,we are not exactly sure of the reason for this di↵erence– we conjecture that the smartphone signal processingpipeline may not be exactly tuned to the vibra-motorlike we have done in the custom case.

Rank of the WordsThe accuracy results above counts only perfect matchesbetween ASR’s output and the actual spoken word. Incertain applications, a list of possible words may alsobe useful, particularly when the quality of the speech ispoor. We record the list of all predictions from ASR foreach spoken word, played at 50 dbSPL. Figure 19 plotsthe CDF of the rank of the correct word in this list. Atthis relatively softer 50 dbSPL experiment, only ⇡ 20%of the words are ranked at 1, implying exact match. In41% of the cases, the words were within top-5 of thelist, and top-10 presents a 58% accuracy.

Word rank0 10 20 30 40

CDF

0

0.2

0.4

0.6

0.8

1

MicrophoneProcessed vib.Raw vib.

Figure 19: The CDF of word rank from ASR’sprediction at 50 dbSPL for custom hardware.

Phoneme SimilarityThe acoustic model we used with ASR is not idealfor VibraPhone – the impact is pronounced for dis-torted phonemes. Training ASR’s acoustic model withthe vibra-motor response is expected to o↵er improve-ments, but in the absence of that, we report a subjectiveoverview of the entropy in di↵erent phonemes recordedby VibraPhone. In other words, we ask whether au-tocorrelation between the same phonemes is high andcross correlation across phonemes are low. We extractthe STFT coe�cients of the 100 phonemes (28 vowelsand 72 consonants) from the International Phoneme Al-phabet [6, 7] and use these coe�cients as the features.We then calculate correlation coe�cient of all pairs ofphonemes in the list – Figure 20 presents the heat map.In case of raw vibra-signal in Figure 20(a), the (dis-torted) phonemes bear substantial similarity betweeneach other, indicated by the multiple dark o↵-diagonalblocks. The two large darker squares in the figure rep-resents the pulmonic (58 phonemes) and non-pulmonic(14 phonemes) consonant groups [26, 17]. However,with VibraPhone, Figure 20(b) shows substantial im-provements. The autocorrelation is strong across the

diagonal of the matrix, while the o↵-diagonal elementsare much less correlated. This extends hope that avibra-motor trained acoustic model could appreciablyboost VibraPhone’s performance.

Raw vib. phonemes

Raw

vib

. pho

nem

es

-0.2

0

0.2

0.4

0.6

0.8

1

Processed vib. phonemes

Proc

esse

d vi

b. p

hone

mes

-0.2

0

0.2

0.4

0.6

0.8

1

Figure 20: The heat-map shows the correlationof the frequency domain features of the phonemesounds, recorded with custom vibration motor:(a) before processing and (b) after processing.

6.3 Performance Results with MSR

Accuracy v/s loudnessFigure 21 shows the accuracy with manual speech recog-nition (MSR) in comparison to automatic (ASR). Un-surprisingly, the accuracy is around 20% more thanASR at higher loudness regimes (60 dbSPL or more) –the individuals guessed the words individually in theseexperiments. Using consensus from group discussion,the accuracy increases to 88% at 60 dbSPL. When theloudness is stronger, VibraPhone is comparable to mi-crophones, both for custom hardware and smartphones.

Sound pressure level (dbSpl) 40 50 60 70 80

Accu

racy (

%)

0

20

40

60

80

100

Human (group)Human (indiv.)ASR

Figure 21: This plot compares the accuracy ofhuman decoding with ASR. It shows the per-formance of the human decoders while workingindividually and as a collaborative team.

Hot-phrase detectionFigure 24 shows manual performance with “hot phrases”,i.e., where the volunteer was asked to pick a phrase fromthe list that best matched the spoken phrase (the volun-teer could also select none of the phrases). We provideda list of 10 written phrases before playing the positiveand negative samples in arbitrary sequence. Examplephrases were “turn left”, “happy birthday”, “start the

10

computer”, etc., and the negative samples were chosenwith comparable number of words and characters.

Figure 24(a) reports results from the custom hardware– volunteers almost perfectly identified the phrases andrejected the negative samples.However, when using the smartphone vibra-motor, Vi-braPhone failed to identify some positive samples – Fig-ure 24(b) shows the outcome in relatively higher falsenegative values. Of course, the degradation is relative– the absolute detection performance is still quite high,with accuracy and precision at 0.83 and 0.90, respec-tively, for the processed vibra-signal.

Metric Acc. Prec. Recall Fallout

Valu

e

0

0.2

0.4

0.6

0.8

1

Raw vibProcessed

Metric Acc. Prec. Recall Fallout

Valu

e

0

0.2

0.4

0.6

0.8

1Raw vibProcessed

Figure 22: The accuracy, precision, recall, andfallout values for manual hot-phrase detection.The recording device is (a) the custom hardwareand (b) the smartphone.

Perceived clarityHuman volunteers also assigned a “clarity score” on arange of [0, 10] to every word/phrase he/she listenedto (a score of 10 indicated a perfectly intelligible word).Figure 23 plots the average clarity score of the correctlydecoded samples and compares it between the vibra-tion motor and the microphone. The subjective per-ception of clarity does not change for the microphonefor sound pressure levels 50 dbSPL and above. WhileVibraPhone’s clarity is lower than microphone in gen-eral, the gap reduces at higher loudness levels. At 80dbSPL, the perceived clarity scores for microphones andVibraPhone are 9.1 and 7.6, respectively.

Sound pressure level (dbSpl) 40 50 60 70 80

Perce

ived c

larity

(0-10

)

0

2

4

6

8

10Vibra-signalMic-signal

Figure 23: The perceived clarity of the correctlydecoded speech recorded with microphone andvibration motor.

Kinds of WordsFigure 24 shows the top-10 and bottom-10 intelligiblewords from the ASR experiments. The font size isproportional to the decoding accuracy, indicating that“international” was decoded correctly most frequently,while prepositions like “a”, “and”, “or” were consis-tently missed. Unsurprisingly, longer words are decodedwith higher accuracy because of better interpolation be-tween the partially decoded phonemes. Figure 25 quan-tifies this with ASR and MSR, respectively – words with5+ characters are mostly multi-syllable, yielding im-proved recognition.

interna3onaldevelopmentna3onal

descrip3on

insurance decembercategoryright download

so9ware

januarypictures

childrenreviews

onlinethe

of

andaarehaveto

butso click

full

usboth

ortoo

Figure 24: Top 10 words that are (a)correctlyand (b)incorrectly decoded by ASR.

Sound pressure level (dbSpl) 40 50 60 70 80

Accu

racy

(%)

0

20

40

60

80

100

Long wordsAll wordsShort words

Sound pressure level (dbSpl) 40 50 60 70 80

Accu

racy

(%)

0

20

40

60

80

100

Long wordsAll wordsShort words

Figure 25: (a) ASR and (b) MSR accuracy forlong (> 6 chars) and short ( 6 chars) words, asa function of loudness.

7. POINTS OF DISCUSSIONWe discuss a few limitations of this paper, and a fewother kinds of applications using VibraPhone.

What is the Best Possible? We have not been ableto comment on the best possible performance possiblewith VibraPhone. Such an analysis will certainly needa deeper signal processing treatment, as well as detaileddomain knowledge from speech recognition. This workis more of a lower bound on feasibility, drawing on adiverse set of established techniques from literature,and modifying them to suit the needs of this specificproblem. We have initiated collaboration with signalprocessing researchers to push the envelope of this sidechannel leak.

Energy: We have sidestepped energy considerations inthis paper. However, we intuitively believe that Vibra-Phone is not likely to be energy hungry (even though thevibra-motor consumes considerable energy while pul-sating). This is because VibraPhone picks up the am-bient sounds while it is in the inactive/passive mode,

11

i.e., when it is not serving as an actuator. We plan tocharacterize the energy profile in future.

Applications: We observed that when vibra-motorsare pasted to walls and floors, and music is being playedin the adjacent rooms, VibraPhone is able to detectthese sounds better than the microphone. We also ob-served that by placing the vibra-motor on the throat,various speech components can be detected, and in somecases, compliments the response of the microphone. Fi-nally, we find that noise properties of vibra-motors andmicrophones are uncorrelated, enabling the possibilityof diversity combining (i.e., they could together behavelike a MIMO system, improving the capacity of acousticchannels). All these observations are preliminary, andhence, not reported in this paper – we plan to investi-gate them further as a continuation of VibraPhone.

8. RELATED WORKPast work on acoustic side channels and speech recov-ery are most relevant to this paper. Given both arereasonably mature areas, we sample a subset of them.

Passive speech recordingGyrophone and AccelWord [29, 45] are perhaps the clos-est to our work. In Gyrophone [29], authors identify theMEMS sensors’ capability to capture sound. The paperpresents a range of signal processing and machine learn-ing techniques to recover traces of ambient sounds fromthe gyroscope data [35]. AccelWord [45] takes a step for-ward and uses speech information from the accelerome-ter [22] to implemented a low energy voice control appli-cation for a limited vocabulary of commands. However,these techniques recover only a low bandwidth of thespectrum (< 200Hz), which does not even cover thefull range of fundamental frequencies in female speech(165� 255 Hz). Therefore, these techniques mainly fo-cus on extracting the reliable features of sound for con-sistent pattern classification. In contrast, VibraPhoneconcentrates on recovering a telephone-quality speech(bandwidth 4KHz [21, 30]) from the vibration motorsignal, making the output amenable to manual or au-tomatic decoding. Both Gyrophone and AccelWord areunable to produce (actually not designed for) machineunderstandable speech.

A family of techniques [31, 37, 43, 38] targets alight/LASER beam on an object exposed to the speechsignal and records its vibration by measuring the fluc-tuation of the reflected beam. Visual microphone [9]is also a similar technique that uses high speed videoof the target object to recover the vibration propor-tional to the speech signal. Camera based techniquesare devoid of the noisy data that pollute motion sen-sors/actuators, while they must tackle other di�cultchallenges in computer vision. A number of solutions

have monitored the change in received signal strength(RSS) and phase of the wireless radio signal reflectedo↵ the loudspeaker to capture the traces of sound. Theprojects [44] and [28] demonstrate successful soundrecovery using reflected radio signal even when the re-ceiver is not in the line-of-sight of the vibrating object.

Speech RecoveryWe borrowed building blocks from the vast literature ofspeech processing. A body of research [8, 25, 24] ex-plores artificial bandwidth expansion problems primar-ily to aid high quality voice transfer over band-limitedtelephonic channel. Some solutions attempt to identifythe phonemes from the low bandwidth signal and thenreplace them with high bandwidth phonemes from a li-brary. These solutions do not solve VibraPhone’s prob-lems as majority of them consider 4KHz signal as theinput providing enough diversity for correct phonemeidentification. VibraPhone attempts to extend the ef-fective bandwidth from 2KHz to 4Khz – a challenge be-cause the features up to 2KHz provide limited exposureto phonemes.

Data imputation techniques [36, 16] attempt to predicterasures in audio signals. When these signals exhibita consistent statistical model, the erasures can be pre-dicted well, enabling successful imputation. However,vibra-signals often lack such properties, and moreover,the location of erasures cannot be confidently demar-cated.

9. CONCLUSIONThis paper demonstrates that the vibration motor,present in almost all mobile devices today, can be usedas a listening sensor, similar to a microphone. Whilethis is not fundamentally surprising (since vibratingobjects should respond to ambient air vibrations), theease and extent to which the actuator has “picked up”sounds has been somewhat unexpected for us. Impor-tantly, the decoded sounds are not merely vibrationpatterns that correlates to some spoken words. Rather,they actually contain the phonemes and structure ofhuman voice, thereby requiring no machine learningor pattern recognition to extract them. We show thatwith basic signal processing techniques, combined withthe structure of human speech, the vibra-motor’s out-put can be quite intelligible to most human listeners.Even automatic speech recognizers were able to decodethe majority of the detected words and phrases, es-pecially at higher loudness. The application space ofsuch systems remains open, and could range from mal-ware eavesdropping into human phone conversation, tovoice controlled wearables, to better microphones thatuse the vibra-motor as a second MIMO-antenna. Ourongoing work is in pursuit of a few such applications.

12

10. REFERENCES[1] Cmu sphinx.

http://cmusphinx.sourceforge.net. Lastaccessed 6 December 2015.

[2] Sound pressure level chart.http://www.sengpielaudio.com/

TableOfSoundPressureLevels.htm. Lastaccessed 6 December 2015.

[3] Top 10000 words from google’s trillion wordcorpus. https://github.com/first20hours/google-10000-english. Last accessed 6December 2015.

[4] Unit of sound pressure level.http://trace.wisc.edu/docs/2004-About-dB/.Last accessed 9 December 2015.

[5] Vibraphone project webpage.http://synrg.csl.illinois.edu/vibraphone/.Last accessed 9 December 2015.

[6] Association, I. P. Handbook of theInternational Phonetic Association: A guide tothe use of the International Phonetic Alphabet.Cambridge University Press, 1999.

[7] Brown, A. International phonetic alphabet. TheEncyclopedia of Applied Linguistics (2013).

[8] Chennoukh, S., Gerrits, A., Miet, G., andSluijter, R. Speech enhancement via frequencybandwidth extension using line spectralfrequencies. In Acoustics, Speech, and SignalProcessing, 2001. Proceedings.(ICASSP’01). 2001IEEE International Conference on (2001), vol. 1,IEEE, pp. 665–668.

[9] Davis, A., Rubinstein, M., Wadhwa, N.,Mysore, G. J., Durand, F., and Freeman,W. T. The visual microphone: Passive recoveryof sound from video. ACM Trans. Graph 33, 4(2014), 79.

[10] Eargle, J. The Microphone Book: From monoto stereo to surround-a guide to microphonedesign and application. CRC Press, 2012.

[11] Fant, G. Acoustic theory of speech production:with calculations based on X-ray studies ofRussian articulations, vol. 2. Walter de Gruyter,1971.

[12] Farina, A. Simultaneous measurement ofimpulse response and distortion with a swept-sinetechnique. In Audio Engineering SocietyConvention 108 (2000), Audio EngineeringSociety.

[13] Fausti, P., and Farina, A. Acousticmeasurements in opera houses: comparisonbetween di↵erent techniques and equipment.Journal of Sound and Vibration 232, 1 (2000),213–229.

[14] Gales, M. J. Maximum likelihood lineartransformations for hmm-based speech

recognition. Computer speech & language 12, 2(1998), 75–98.

[15] Galili, I., Kaplan, D., and Lehavi, Y.Teaching faradays law of electromagneticinduction in an introductory physics course.American journal of physics 74, 4 (2006),337–343.

[16] Gemmeke, J. F., Virtanen, T., andHurmalainen, A. Exemplar-based sparserepresentations for noise robust automatic speechrecognition. Audio, Speech, and LanguageProcessing, IEEE Transactions on 19, 7 (2011),2067–2080.

[17] Hayes, B. Introductory phonology, vol. 32. JohnWiley & Sons, 2011.

[18] Hillenbrand, J., and Sessler, G. M.High-sensitivity piezoelectric microphones basedon stacked cellular polymer films (l). The Journalof the Acoustical Society of America 116, 6(2004), 3267–3270.

[19] Hinton, G., Deng, L., Yu, D., Dahl, G. E.,Mohamed, A.-r., Jaitly, N., Senior, A.,Vanhoucke, V., Nguyen, P., Sainath, T. N.,et al. Deep neural networks for acousticmodeling in speech recognition: The shared viewsof four research groups. Signal ProcessingMagazine, IEEE 29, 6 (2012), 82–97.

[20] Huggins-daines, D., Kumar, M., Chan, A.,Black, A. W., Ravishankar, M., andRudnicky, A. I. Pocketsphinx: A free, real-timecontinuous speech recognition system forhand-held devices. In in Proceedings of ICASSP(2006).

[21] Jax, P., and Vary, P. On artificial bandwidthextension of telephone speech. Signal Processing83, 8 (2003), 1707–1719.

[22] Johnson, C. D. Accelerometer principles.Process Control Instrumentation Technology(2009).

[23] Keele Jr, D. Low-frequency loudspeakerassessment by nearfield sound-pressuremeasurement. Journal of the audio engineeringsociety 22, 3 (1974), 154–162.

[24] Kontio, J., Laaksonen, L., and Alku, P.Neural network-based artificial bandwidthexpansion of speech. Audio, Speech, and LanguageProcessing, IEEE Transactions on 15, 3 (2007),873–881.

[25] Laaksonen, L., Kontio, J., and Alku, P.Artificial bandwidth expansion method toimprove intelligibility and quality of amr-codednarrowband speech. In ICASSP (1) (2005),pp. 809–812.

[26] Ladefoged, P. The revised internationalphonetic alphabet. Language (1990), 550–552.

13

[27] Lapteva, O. Speaker Perception andRecognition: An Integrative Framework forComputational Speech Processing. kasseluniversity press GmbH, 2011.

[28] McGrath, W. Technique and device forthrough-the-wall audio surveillance, Mar. 30 2005.US Patent App. 11/095,122.

[29] Michalevsky, Y., Boneh, D., and Nakibly,G. Gyrophone: Recognizing speech fromgyroscope signals. In Proc. 23rd USENIX SecuritySymposium (SEC?14), USENIX Association(2014).

[30] Moreno, P. J., and Stern, R. M. Sources ofdegradation of speech recognition in the telephonenetwork. In Acoustics, Speech, and SignalProcessing, 1994. ICASSP-94., 1994 IEEEInternational Conference on (1994), vol. 1, IEEE,pp. I–109.

[31] Muscatell, R. P. Laser microphone, Oct. 231984. US Patent 4,479,265.

[32] Nation, P., and Waring, R. Vocabulary size,text coverage and word lists. Vocabulary:Description, acquisition and pedagogy 14 (1997),6–19.

[33] Ogata, K. System dynamics, vol. 3. PrenticeHall New Jersey, 1998.

[34] Roy, N., Gowda, M., and Choudhury, R. R.Ripple: Communicating through physicalvibration.

[35] Scarborough, J. B. The Gyroscope: Theoryand Application. Interscience Pub., 1958.

[36] Smaragdis, P., Raj, B., and Shashanka, M.Missing data imputation for spectral audiosignals. In Machine Learning for SignalProcessing, 2009. MLSP 2009. IEEEInternational Workshop on (2009), IEEE, pp. 1–6.

[37] Smeets, G. Laser interference microphone forultrasonics and nonlinear acoustics. The Journalof the Acoustical Society of America 61, 3 (1977),872–875.

[38] Speciale, J. R. Pulsed laser microphone, Oct. 92001. US Patent 6,301,034.

[39] Tanner, P., Loebach, J., Cook, J., andHallen, H. A pulsed jumping ring apparatus fordemonstration of lenzs law. American Journal ofPhysics 69, 8 (2001), 911–916.

[40] Taylor, B. Guide for the Use of theInternational System of Units (SI): The MetricSystem. DIANE Publishing, 1995.

[41] Titze, I. R. Principles of voice production.National Center for Voice and Speech, 2000.

[42] Waibel, A., Hanazawa, T., Hinton, G.,Shikano, K., and Lang, K. J. Phonemerecognition using time-delay neural networks.Acoustics, Speech and Signal Processing, IEEETransactions on 37, 3 (1989), 328–339.

[43] Wang, C.-C., Trivedi, S., Jin, F.,Swaminathan, V., Rodriguez, P., andPrasad, N. S. High sensitivity pulsed laservibrometer and its application as a lasermicrophone. Applied Physics Letters 94, 5 (2009),051112.

[44] Wei, T., Wang, S., Zhou, A., and Zhang, X.Acoustic eavesdropping through wirelessvibrometry. In Proceedings of the 21st AnnualInternational Conference on Mobile Computingand Networking (2015), ACM, pp. 130–141.

[45] Zhang, L., Pathak, P. H., Wu, M., Zhao,Y., and Mohapatra, P. Accelword: Energye�cient hotword detection through accelerometer.In Proceedings of the 13th Annual InternationalConference on Mobile Systems, Applications, andServices (2015), ACM, pp. 301–315.

14