Lecture 1: Introduction & DSPm.mr-pc.org/t/csc83060/2016fa/lecture01.pdf · Introduction & DSP...

CSC 83060: Speech & Audio Understanding

Lecture 1:Introduction & DSP

Michael Mandel [email protected]

CUNY Graduate Center, Computer Science Programhttp://mr-pc.org/t/csc83060

With much content from Dan Ellis’ EE 6820 course

1 Sound and information

2 Course Structure

3 DSP review: Timescale modification

Michael Mandel (83060 SAU) Intro & DSP 1 / 32

[email protected]

http://mr-pc.org/t/csc83060

Outline


2 Course Structure



Sound and information

Sound is air pressure variation

Mechanical vibration

Pressure waves in air

Motion of sensor

Time-varying voltage

+ + + +

t

v(t)

Transducers convert air pressure ↔ voltage


What use is sound?

Footsteps examples:

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-0.5

0

0.5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-0.5

0

0.5

time / s

Hearing confers an evolutionary advantage

useful information, complements vision

. . . at a distance, in the dark, around corners

listeners are highly adapted to ‘natural sounds’ (includingspeech)


http://www.ee.columbia.edu/~dpwe/e6820/sounds/steps-m-d5.wav

http://www.ee.columbia.edu/~dpwe/e6820/sounds/steps-f-d5.wav

The scope of audio processing


The acoustic communication chain

message signal channel receiver decoder

!

synthesis audio processing recognition

Sound is an information bearer

Received sound reflects source(s)plus effect of environment (channel)


Levels of abstraction

Much processing concerns shifting between levels of abstraction

sound p(t)

representation (e.g. t-f energy)

‘information’

abstract

concrete

An

alys

is

Syn

thesis

Different representations serve different tasks

separating aspects, making things explicit, . . .


Outline


2 Course Structure



Course structure

GoalsI survey topics in sound analysis & processingI develop and intuition for sound signalsI learn some specific technologies

Course structureI weekly reading (20% + 10%)I midterm project proposal (20%)I final project (30% + 10%)

Recommended, but not required text

Speech and Audio Signal ProcessingGold, Morgan, & EllisWiley, 2011ISBN: 978-0-470-19536-9


Web-based

Course website:


for lecture notes, articles, examples, . . .

+ Blackboard for homework, etc.



Weekly assignments

Review papersI book chapters & journal papers reviewing a topic areaI discuss in classI written summaries & future directions due on Blackboard

Contribution papersI journal & conference papers presenting a novel contributionI student presentations in class

Project statusI written weekly status updates due on Blackboard


Final project

Most significant part of course (60%) of grade

Oral proposals mid-semester;Presentations in final class+ paper

ScopeI practical (Python or Matlab recommended)I identify a problem; try some solutionsI evaluation

TopicI few restrictions within world of audioI investigate other resourcesI develop in discussion with me

Copying


Examples of past projects

Automatic prosody classification Model-based note transcription


Outline


2 Course Structure



DSP review: digital signals

time

xd[n] = Q( xc(nT ) )

Discrete-time sampling limits bandwidth

Discrete-level quantization

limits dynamic range

T

ε

sampling interval T

sampling frequency ΩT = 2πT

quantizer Q(y) = ε⌊ yε

⌋Michael Mandel (83060 SAU) Intro & DSP 15 / 32

The speech signal: time domain

Speech is a sequence of different sound types

-0.2

-0.1

0

0.1

0.2

1.38 1.4 1.42.1

0

.1

1.52 1.54 1.56 1.58-0.1

0

0.1

1.86 1.88 1.921.9-0.05

0

0.05

2.42 2.44 2.46 2.4

-0.02

0

0.02

1.4 1.6 1.8 2 2.2 2.4 2.6 time/s

watch thin as a dimeahas

Vowel: periodic “has”

Fricative: aperiodic “watch”

Glide: smooth transition “watch”

Stop burst: transient “dime”


http://www.ee.columbia.edu/~dpwe/e6820/sounds/mpgr1_sx419-8k.wav

Timescale modification (TSM)Can we modify a sound to make it ‘slower’?i.e. speech pronounced more slowly

e.g. to help comprehension, analysis

or more quickly for ‘speed listening’?

Why not just slow it down?

xs(t) = xo( tr ), r = slowdown factor (> 1→ slower)

equivalent to playback at a different sampling rate

2.35 2.4 2.45 2.5 2.55 2.6-0.1

-0.05

0

0.05

0.1

2.35 2.4 2.45 2.5 2.55 2.6-0.1

-0.05

0

0.05

0.1

time/s

Original

2x slower

r = 2


http://www.ee.columbia.edu/~dpwe/e6820/sounds/mpgr-slowdown.wav

Time-domain TSM

Problem: want to preserve local time structurebut alter global time structure

Repeat segmentsI but: artifacts from abrupt edges

Cross-fade & overlap

ym[mL + n] = ym−1[mL + n] + w [n] · x[⌊m

r

⌋L + n

]

2.35 2.4 2.45 2.5 2.55 2.6-0.1

0

0.1

4.7 4.75 4.8 4.85 4.9 4.95-0.1

0

0.1

1

1

1 1 2 2 3 3 4 4 5 5 6

2

2

3

3

4

4

5 6

6

5

time / s

time / s


http://www.ee.columbia.edu/~dpwe/e6820/sounds/mpgr-fixwin.wav

Synchronous overlap-add (SOLA)

Idea: allow some leeway in placing window to optimize alignmentof waveforms

1

2

Km maximizes alignment of 1 and 2

Hence,

ym[mL + n] = ym−1[mL + n] + w [n] · x[⌊m

r

⌋L + n + Km

]Where Km chosen by cross-correlation:

Km = argmax0≤K≤Ku

∑Novn=0 y

m−1[mL + n] · x[⌊

mr

⌋L + n + K

]√∑(ym−1[mL + n])2

∑(x[⌊

mr

⌋L + n + K

])2


http://www.ee.columbia.edu/~dpwe/e6820/sounds/mpgr-solafs.wav

The Fourier domain

Fourier Series (periodic continuous x)

Ω0 =2π

T

x(t) =∑k

ckejkΩ0t

ck =1

2πT

∫ T/2

−T/2x(t)e−jkΩ0tdt k1 2 3 5 6 74

|ck|1.0

1.5 1 0.5 0 0.5 1 1.5 1

0.5

0

0.5

t

x(t)

Fourier Transform (aperiodic continuous x)

x(t) =1

2π

∫X (jΩ)e jΩtdΩ

X (jΩ) =

∫x(t)e−jΩtdt

0 0.002 0.004 0.006 0.008time / sec

level / dB

-0.01

0

0.01

0.02

x(t)

0 2000 4000 6000 8000freq / Hz

-80

-60

-40

-20 |X(jΩ)|


Discrete-time Fourier

DT Fourier Transform (aperiodic sampled x)

x [n] =1

2π

∫ π

−πX (e jω)e jωndω

X (e jω) =∑

x [n]e−jωn

n-1 1 2 3 4 5 6 7

0 π

|X(ejω)|

ω2π 3π 4π 5π

1

2

3

x [n]

Discrete Fourier Transform (N-point x)

x [n] =∑k

X [k]e j 2πknN

X [k] =∑n

x [n]e−j 2πknN

k

|X(ejω)||X[k]|

k=1...

n1 2 3 4 5 6 7

x [n]


Sampling and aliasingDiscrete-time signals equal the continuous time signal at discretesampling instants:

xd [n] = xc(nT )

Sampling cannot represent rapid fluctuations

0 1 2 3 4 5 6 7 8 9 10 1

0.5

0

0.5

1

sin

((ΩM +

2π

T

)Tn

)= sin(ΩMTn) ∀n ∈ Z

Nyquist limit (ΩT/2) from periodic spectrum:

ΩΩM−ΩT ΩT −ΩM ΩT - ΩM−ΩT + ΩM

Gp(jΩ)Ga(jΩ) “alias” of “baseband”

signal


Speech sounds in the Fourier domain

1.52 1.54 1.56 1.58-0.1

0

0.1

2.42 2.44 2.46 2.48

-0.02

0

0.02

0 1000 2000 3000 4000-100

-80

-60

-40

0 1000 2000 3000 4000-100

-80

-60

1.37 1.38 1.39 1.4 1.41 1.42-0.1

0

0.1

0 1000 2000 3000-100

-80

-60

-40

1.86 1.87 1.88 1.89 1.9 1.91-0.05

0

0.05

0 1000 2000 3000 4000-100

-80

-60

Vowel: periodic “has”

Fricative: aperiodic “watch”

Glide: transition “watch”

Stop: transient “dime”

time domain frequency domain

time / s freq / Hz

ener

gy

/ dB

dB = 20 log10(amplitude) = 10 log10(power)Voiced spectrum has pitch + formants


Short-time Fourier TransformWant to localize energy in time and frequency

break sound into short-time pieces

calculate DFT of each one

2.35 2.4 2.45

0

4000

3000

2000

1000

2.5 2.55 2.6-0.1

0

0.1

time / s

freq

/ H

z

k →

short-timewindow

DFT

m = 0 m = 1 m = 2 m = 3

L 2L 3L

Mathematically,

X [k ,m] =N−1∑n=0

x [n]w [n −mL] exp

(−j 2πk(n −mL)

N

)Michael Mandel (83060 SAU) Intro & DSP 24 / 32

The Spectrogram

Plot STFT X [k ,m] as a gray-scale image

time / s

time / s

freq

/ H

zintensity / dB

2.35 2.4 2.45 2.5 2.55 2.60

1000

2000

3000

4000

freq

/ H

z

0

1000

2000

3000

4000

0

0.1

-50

-40

-30

-20

-10

0

10

0 0.5 1 1.5 2 2.5


Time-frequency tradeoffLonger window w [n] gains frequency resolution at cost of timeresolution

1.4 1.6 1.8 2 2.2 2.4 2.6

freq

/ H

z

time / s

level/ dB

0

1000

2000

3000

4000

freq

/ H

z

0

1000

2000

3000

4000

0

0.2W

indo

w =

256

pt

ÒNar

row

ban

dÓ

Win

dow

= 4

8 pt

ÒWid

eban

dÓ

-50

-40

-30

-20

-10

0

10


Speech sounds on the Spectrogram

Most popular speech visualization

freq

/ H

z

0

1000

2000

3000

4000

1.4 1.6 1.8 2 2.2 2.4 2.6 time/s

watch thin as a dimeahas

Vow

el:

perio

dic

“has

”

Fric

've:

ap

erio

dic

“wat

ch”

Glid

e: tr

ansi

tion

“wat

ch”

Sto

p: tr

ansi

ent

“dim

e”

Wideband (short window) better than narrowband (long window)to see formants


TSM with the Spectrogram

Just stretch out the spectrogram?

Time

Fre

quen

cy

0 0.2 0.4 0.6 0.8 1 1.2 1.40

1000

2000

3000

4000

Time

Fre

quen

cy

0 0.2 0.4 0.6 0.8 1 1.2 1.40

1000

2000

3000

4000

how to resynthesize?spectrogram is only |Y [k,m]|


The Phase VocoderTimescale modification in the STFT domain

Magnitude from ‘stretched’ spectrogram:

|Y [k,m]| =∣∣∣X [k , m

r

]∣∣∣I e.g. by linear interpolation

But preserve phase increment between slices:

θY [k,m] = θX

[k,

m

r

]I e.g. by discrete differentiator

Does right thing for single sinusoidI keeps overlapped parts of sinusoid aligned

time

θ = ∆θ∆T

.

∆θ' = θ·2∆T .

∆T


http://www.ee.columbia.edu/~dpwe/e6820/sounds/mpgr-pvoc.wav

General issues in TSM

Time windowI stretching a narrowband spectrogram

Malleability of different soundsI vowels stretch well, stops lose nature

Not a well-formed problem?I want to alter time without frequency

. . . but time and frequency are not separate!I ‘satisfying’ result is a subjective judgment⇒ solution depends on auditory perception. . .


Summary

Information in soundI lots of it, multiple levels of abstraction

Course overviewI survey of audio processing topicsI practicals, readings, project

DSP reviewI digital signals, time domainI Fourier domain, STFT

Timescale modificationI properties of the speech signalI time-domainI phase vocoder


References

J. L. Flanagan and R. M. Golden. Phase vocoder. Bell System Technical Journal,pages 1493–1509, 1966.

M. Dolson. The Phase Vocoder: A Tutorial. Computer Music Journal, 10(4):14–27,1986.

M. Puckette. Phase-locked vocoder. In IEEE Workshop on Applications of SignalProcessing to Audio and Acoustics, pages 222–225, 1995.

A. T. Cemgil and S. J. Godsill. Probabilistic Phase Vocoder and its application toInterpolation of Missing Values in Audio Signals. In 13th European SignalProcessing Conference, Antalya, Turkey, 2005.


Lecture 1: Introduction & DSPm.mr-pc.org/t/csc83060/2016fa/lecture01.pdf · Introduction & DSP...

Documents

Transcript of Lecture 1: Introduction & DSPm.mr-pc.org/t/csc83060/2016fa/lecture01.pdf · Introduction & DSP...