Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline...

37
1 Physics-Based Sound Synthesis ELEC-E5620 - Audio Signal Processing, Lecture #8 Vesa Välimäki March 17, 2017 Sound check 0. General issues (Vesa & Fabian) 13.1.2017 1. History and future of audio DSP (Vesa) 20.1.2017 2. Digital filters in audio (Vesa) 27.1.2017 3. Audio filter design (Vesa) 3.2.2017 4. Analysis of audio signals (Vesa) 10.2.2017 No lecture (Evaluation week for Period III) 17.2.2017 5. Audio effects processing (Fabian) 24.2.2017 6. Synthesis of audio signals (Fabian) 3.3.2017 7. 3-D sound (Prof. Ville Pulkki) 10.3.2017 8. Physics-based sound synthesis (Vesa) 17.3.2017 9. Sampling rate conversion (Vesa) 24.3.2017 10. Audio coding (Vesa) 31.3.2017 Course Schedule in 2017 March 17, 2017 2 ©2003-2017 Vesa Välimäki

Transcript of Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline...

Page 1: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

1

Physics-Based Sound Synthesis

ELEC-E5620 - Audio Signal Processing, Lecture #8

Vesa Välimäki

March 17, 2017

Sound check

0. General issues (Vesa & Fabian) 13.1.2017

1. History and future of audio DSP (Vesa) 20.1.2017

2. Digital filters in audio (Vesa) 27.1.2017

3. Audio filter design (Vesa) 3.2.2017

4. Analysis of audio signals (Vesa) 10.2.2017

No lecture (Evaluation week for Period III) 17.2.2017

5. Audio effects processing (Fabian) 24.2.2017

6. Synthesis of audio signals (Fabian) 3.3.2017

7. 3-D sound (Prof. Ville Pulkki) 10.3.2017

8. Physics-based sound synthesis (Vesa) 17.3.2017

9. Sampling rate conversion (Vesa) 24.3.2017

10. Audio coding (Vesa) 31.3.2017

Course Schedule in 2017

March 17, 20172

©2003-2017 Vesa Välimäki

Page 2: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

2

©2003-2017 Vesa Välimäki 3

OutlinePhysical modeling of musical instruments

(Soitinmallinnus in Finnish)• Classification of methods and some examples• Digital waveguide modeling• Modeling of stringed musical instruments

– The acoustic guitar, the harpsichord, the piano, the Clavinet, the electric guitar, virtual air guitars, …

• Two demos: Virtual MS-20 Filter & Flutter Echo Modeling

March 17, 2017

DEMO

MS-20 Filter

Joonas & Mikko

4©2003-2017 Vesa Välimäki

March 17, 2017

Page 3: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

3

©2003-2017 Vesa Välimäki 5

Taxonomy of Sound Synthesis Methods(J. O. Smith, 1991)

(Source: http://www-ccrma.stanford.edu/~jos/kna/Taxonomy_Digital_Synthesis_Techniques.html)March 17, 2017

©2003-2017 Vesa Välimäki 6

Wavetable (Delay line)Noise

Out

Modifier

Karplus-Strong Algorithm

Control fundamental freq.

• Invented by Karplus and Strong (late 1970s; published 1983)

• A self-modifying wavetable– Wavetable is interpreted as a delay line, where the waveform propagates

– Modifier = average of 2 contiguous samples

• Dullness is reduced considerably– Amazing results with simple processing

March 17, 2017

Page 4: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

4

©2003-2017 Vesa Välimäki 7

Karplus-Strong: The Movie• Averaging of zero-mean random numbers

• Initial state (random)

• Two-point average of previous state

March 17, 2017

©2003-2017 Vesa Välimäki 8

Karplus-Strong Sound Examples• Four octaves of C-major scale (65 Hz ... 1046 Hz) played with

the Karplus-Strong algorithm

1) Sampling rate 8000 Hz

2) Sampling rate 11025 Hz

3) Sampling rate 22050 Hz

4) Sampling rate 44100 Hz

• High notes are out of tune!– f = fs / (L + 0.5), where the half-sample delay is caused by averaging

– Wavetable length L is rounded to the nearest integer

• Timbre and decay time depend on sampling rateMarch 17, 2017

Page 5: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

5

©2003-2017 Vesa Välimäki 9

• Desired vs. actual fundamental frequency of KS algorithm for a 50-kHz sampling rate (left)

• Corresponding approximation error in semitones (right)

Tuning Problem

25

20

15

10

5

00 15 20105Q

uan

tize

d F

req

uen

cy (

kHz)

Desired Frequency (kHz) Frequency (kHz)

Pit

ch E

rro

r (S

emit

on

es)

March 17, 2017

©2003-2017 Vesa Välimäki 10

• Simulate the sound production mechanism– In speech synthesis and coding, this is an old tradition

• High-quality synthesis of string and wind instruments

• More sophisticated signal processing than earlier– Compared to subtractive synthesis, sampling & wavetable synthesis, FM

synthesis, ...

– Combines linear & nonlinear methods, filtering, sampling, ...

– Considered computationally expensive

Sound Synthesis by Physical Modeling

March 17, 2017

Page 6: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

6

©2003-2017 Vesa Välimäki 11

Physical Modeling Techniques1) Source-Filter Modeling

– Physically informed subtractive synthesis

2) Finite Difference Modeling

– Discretized wave equation (difference equations)

3) Mass-Spring Networks

– “CORDIS-ANIMA” by ACROE (Grenoble, France)

4) Modal Synthesis

– “Modalys” software by IRCAM (Paris, France); formerly “Mosaic”

– The functional transform method by Trautmann and Rabenstein

5) Wave Digital Filters

– Method to convert models of continuous-time systems into digital filters

6) Digital Waveguide Synthesis

– Physical modeling using digital filters and delays; Main topic of this lecture

March 17, 2017

©2003-2017 Vesa Välimäki 12

Source-Filter Model• Source signal can be recorded or synthetic

• Filter can model a resonating structure – For example, the vocal tract, the soundbox of an instrument, …

Glottal waveform extracted from speech(Prof. Paavo Alku)

With formant filters:

March 17, 2017

Page 7: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

7

©2003-2017 Vesa Välimäki 13

Finite-Difference Models• Discretization of the partial differential equation

• Discretized approximation

and its recursive implementation

March 17, 2017

©2003-2017 Vesa Välimäki 14

Finite-Difference Models• The next state is computed based on 2 previous states

– For the basic wave equation

Space

Time

March 17, 2017

Page 8: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

8

©2003-2017 Vesa Välimäki 15

Mass-Spring Networks• A distributed mechanical system can be approximated by a

finite number of small mass elements

(Fig

ure

from

: T. D

. R

ossi

ng: T

he S

cien

ce o

f S

ound

.S

econ

dE

ditio

n. A

ddis

on-W

esle

y,19

90.)

March 17, 2017

©2003-2017 Vesa Välimäki 16

Mass-Spring Networks (2)• Mass elements are connected with spring elements

– Micro-dampers are used to simulate losses

• A mass-spring network can be simulated with a digital filter structure (Cadoz et al., 1983; Välimäki et al., 2006)

– Basic elements are discretized separately

– Apply Hooke’s law (F = -kx) and Newton’s second law (F = ma)

March 17, 2017

Page 9: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

9

©2003-2017 Vesa Välimäki 17

Mass-Spring Networks – The Movie

Movie link: http://www.acs.psu.edu/drussell/Demos/multi-dof/multi-dof.htmlMarch 17, 2017

• For example, 3 masses 3 transversal modes

©2003-2017 Vesa Välimäki 18

Modal Synthesis• Vibrating modes of a structure (instrument) are modeled

separately

• Filter bank of 2nd-order digital resonators Rk(z)– Type and location of excitation affects the input of each resonator

)(ny

)(2 zR

)(zRK

)(1 zR

Matrix

...

Input

March 17, 2017

Page 10: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

10

©2003-2017 Vesa Välimäki 19

Modalys• Modal synthesis software developed at IRCAM (Paris, France)

(Adrien, 1991; Morrison & Adrien, 1991)

• Contains many excitation and resonator models– Analyzed parameters for instruments bodies, bells, strings, bars, …

• Allows arbitrary connections– Blow a string, bow a tube, pluck a drum etc.

• Sound examples

1.Trumpet

2.Clarinet (very bad player)

3.Bowing a metal tube

4.Gong with time-varying properties

Sou

rce:

http

://w

ww

.irca

m.fr

/foru

mne

t/…

March 17, 2017

Wave Digital Filters• Developed by Fettweis (1970-) to simulate analog filters

• Similarly, a lumped acoustic model can be converted into a digital one using the bilinear transform

– Mass Inductor

– Damper Resistor

– Spring Capacitor

• Useful for the modeling of nonlinear excitation mechanisms – Hammer of the piano, reed in wind instruments

• Also linear resonator models– E.g. woodwind bore (tube)

• Currently used for modeling musical electronic circuits

March 17, 2017 ©2003-2017 Vesa Välimäki 20

Page 11: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

11

©2003-2017 Vesa Välimäki 21

History of Musical Instrument Modeling• Pythagoras studied string vibrations (about 500 B.C.)

• Basics of string vibration were understood around 1770

• Sound synthesis by mathematical modeling (Hiller & Ruiz, 1971)

• Mass-spring networks for physical modeling of musical instruments in Grenoble, France (1970’s)

• Karplus and Strong invented a modified wavetable synthesis method in 1979 (published in 1983)

• Jaffe and Smith extended the Karplus-Strong algorithm (1983)

• Julius Smith developed the theory of digital waveguides (1985-)

• Products since 1994 (Yamaha, Korg, Creative, Roland…)

March 17, 2017

©2003-2017 Vesa Välimäki 22(Source: http://www-ccrma.stanford.edu/~jos/kna/Taxonomy_Digital_Synthesis_Techniques.html)

Taxonomy of Sound Synthesis Methods(J. O. Smith, 1991)

March 17, 2017

Page 12: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

12

©2003-2017 Vesa Välimäki 23

FD Delay line Hl(z)

In Out

Loop filter

Plucked-String Synthesis Model

Fractional delay filter

Fundamental frequency Decay rate

• Originally developed at CCRMA, Stanford University (Jaffe & Smith, 1983)

• Further developments at TKK (Välimäki et al., 1996; Tolonen 1998, Karjalainen et al., 1998)

March 17, 2017

©2003-2017 Vesa Välimäki 24

Tuning Filter• Fractional delay filter F(z) fine-tunes the delay-line length so that

the pitch is right• Two alternatives

1) Allpass filter (Jaffe & Smith, 1983)– First-order allpass filter usually– Transient problems in time-varying cases (e.g., glissando)

2) Lagrange interpolation(Karjalainen & Laine, 1991; Välimäki et al., 1996)

– Linear interpolation (1st-order Lagrange) not good enough, because high frequencies will be damped

– Order 3 or 4 sufficient at 44.1 kHz

March 17, 2017

Page 13: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

13

©2003-2017 Vesa Välimäki 25

Waveguide Synth Tuning with Allpass filter• Phase delay of the first-order

allpass filter at low frequencies (dc):

D = (1 – a1) / (1 + a1)

• Phase delay: –()/ – Phase response divided by

angular frequency

– Note the minus sign

• Solve a1 for desired delay D:

z-1x(n)

a1

y(n)

a1

-

D

Da

1

11

March 17, 2017

©2003-2017 Vesa Välimäki 26

Tuning with Lagrange Interpolation

• Coefficient formulas for 3rd-order Lagrange FD filter (1 < D < 2):

• Use, for example, a 3rd-order Lagrange FD filter (N = 3), which is a four-tap FIR filter

March 17, 2017

Page 14: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

14

©2003-2017 Vesa Välimäki 27

Loop Filter

H z ga

a zl ( )

1

11

11

• Loop filter Hl(z) controls the decay rate of harmonics• One choice is a one-pole filter (“leaky integrator”)

(Välimäki et al., 1996)

– Parameter g determines overall decay rate (0 < g < 1)– Parameter a1 determines frequency-dependent decay (–1 < a1 < 0)

March 17, 2017

©2003-2017 Vesa Välimäki 28

Structure of Basic String Model• Implementation using a one-pole loop filter and a 3rd-order FIR

interpolation filter (Laurson et al., 2001))(nx

)3(h

)(ny

)](1)[( nang

)(na

1z 1z1z 1z

)2(h )1(h )0(h+

-Mz

)(1 ny

)(ny dL

Loop filterFIR fractional delay filter

Loop filter

Delay lineMarch 17, 2017

Page 15: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

15

©2003-2017 Vesa Välimäki 29

Commuted String Instrument Model• Instrument body is expensive to model

A high-order digital filter is needed

• Computational savings are obtained by commuting the pluck, string, and the body (Smith, 1993; Karjalainen et al., 1993)

E(z) S(z) B(z) y(n)(n)a)

b)y(n)E(z) S(z)b(n)

Excitation String Body

x(n)

e(n)

Commuted piano synthesis (Van Duyne

and Smith, 1995)

March 17, 2017

©2003-2017 Vesa Välimäki 30

Parameter Estimation• Anechoic recordings of guitar playing

• Signal analysis using short-time Fourier transform (Välimäki et al. 1996; Tolonen 1998)

• Parameter estimation using iterative methods

(Erkut et al., 2000)

• Excitation signals obtained by either by

1) Inverse filtering (Välimäki et al. 1996) OR2) by subtracting the harmonics from recorded tones & equalizing (Tolonen 1998; Välimäki & Tolonen, 1998)

March 17, 2017

Page 16: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

16

©2003-2017 Vesa Välimäki 31

Short-Time Fourier Analysis• Signal analysis using

short-time Fourier transform

(Välimäki et al. 1996; Tolonen 1998)

March 17, 2017

©2003-2017 Vesa Välimäki 32

Harmonics = Spectral Peaks• Peak picking from

the magnitude spectrum of each frame

(Välimäki et al. 1996; Tolonen 1998)

March 17, 2017

Page 17: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

17

©2003-2017 Vesa Välimäki 33

Decay Rate of Harmonics• Envelopes of

harmonics are created by connecting spectral peaks in time

• Decay rate estimate by linear regression

(Välimäki et al. 1996; Tolonen 1998)

March 17, 2017

©2003-2017 Vesa Välimäki 34

Loop Filter Design• One-pole loop-filter

design

(Välimäki et al. 1996)

• Target response is obtained from the decay rates

March 17, 2017

Page 18: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

18

©2003-2017 Vesa Välimäki 35

Extracting Input Signal• Excitation signals

obtained by either by

1) inverse filtering (Välimäki et al. 1996)

OR

2) subtracting harmonics from recorded tones & equalizing(Tolonen 1998; Välimäki & Tolonen, 1998)

March 17, 2017

©2003-2017 Vesa Välimäki 36

• Commuted waveguide synthesis (sampling + modeling)− Two string models in parallel (horizontal and vertical polarization)

− Excitations & special effects stored in a database (Laurson et al., 2001)

Plucking-point filter

Timbrecontrol

Database ofexcitation signals

From sympatheticcoupling matrix

To sympatheticcoupling matrix

Out

)(h zS

)(v zS

Specialeffects

Structure of Guitar Synthesizer

March 17, 2017

Page 19: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

19

©2003-2017 Vesa Välimäki 37

• Classical acoustic guitar

1. Guitar music from the Baroque era (J. S. Bach)

2. Modern guitar music (J. A. Muro)

Sound examples available on the web:

• http://www.acoustics.hut.fi/demos/

• http://www.siba.fi/soundingscore/guitar.html

Synthesis Examples

March 17, 2017

©2003-2017 Vesa Välimäki 38

How about other instruments?• Yes, we can do it! (… if they are stringed instruments)

March 17, 2017

Page 20: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

20

©2003-2017 Vesa Välimäki 39

TKK Clavichord Synthesizer• Modification of the TKK guitar synthesizer (Välimäki et al., 2003)• Commuted waveguide synthesis with more samples

– Beginning of soundbox response in excitation signal, the rest is sampled!

TimbrecontrolExcitation

signals

Sound

Soundboxresponses

Trigger at attack time

Trigger at release time

End knocksgsb

grelease

gc

S2

S1

gi1

gi2

go1

go2

March 17, 2017

©2003-2017 Vesa Välimäki 40

• Clavichord synthesizer played by the ENP control software(produced by Mikael Laurson and Jonte Knif, Sibelius Academy)

TKK Clavichord Synthesizer

J. J. Froberger, Gigue from Suite no. 30 in A minor

V. Välimäki et al., Computer Music J., Issue #1, 2003(Sound examples on CMJ DVD with issue #4, 2003)

March 17, 2017

Page 21: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

21

©2003-2017 Vesa Välimäki 41

Harpsichord• Ancient keyboard

instrument– Since the year 1400 …

• Based on plucked strings– Like the guitar

• More complicated than guitar or clavichord– Wide soundboard

– 1-2 keyboards

– 2-4 registers

March 17, 2017

©2003-2017 Vesa Välimäki 42

Anechoic Recordings

March 17, 2017

Page 22: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

22

©2003-2017 Vesa Välimäki 43

Plucking Machinery

(Picture from: T.D. Rossing, “The Science of Sound”, 2nd Ed. Addison-Wesley, 1990)

March 17, 2017

©2003-2017 Vesa Välimäki 44

TKK Harpsichord Synthesis Model• Modification of the clavichord synthesis model (Välimäki et al. 2004)

– Excitation and key release sound are samples– Vibrating string sound is synthesized algorithmically: S(z) ja R(z)– Tail of soundboard response is modeled with a reverb algorithm

March 17, 2017

Page 23: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

23

©2003-2017 Vesa Välimäki 45

Slight Change in String Model S(z)

• We inserted a feedforward comb filter– Can increase the difference in decay rates of neighboring harmonics

)(nx )(ny

b

a

1Lz 1z+

-

One-polefilter

Rz

)(zF r

Ripplefilter

+

-

March 17, 2017

©2003-2017 Vesa Välimäki 46

Before

• One-pole filter, delay line, FD filterin a feedback loop since 1991

)(nx )(ny

b

a

Lz 1z

One-polefilter

)(zF+

-

March 17, 2017

Page 24: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

24

©2003-2017 Vesa Välimäki 47

After

• One-pole filter, delay line, FD filter, and a feedforward comb filterin a feedback loop

)(nx )(ny

b

a

1Lz 1z+

-

One-polefilter

Rz

)(zF r

Ripplefilter

+

-

March 17, 2017

©2003-2017 Vesa Välimäki 48

Some Variation We’ve Always Wanted

Measured data

Approximations:

No variations

(1-pole filter only)

Variations

(one-pole filter

and comb filter)

March 17, 2017

Page 25: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

25

©2003-2017 Vesa Välimäki 49

But Does it Sound Better?

• If not, it doesn’t pay to do it• Informal listening test

1. Original recorded sound

2. Synthesis with old method

3. Synthesis with new method

4. Original recorded sound

5. Synthesis with new method

• Conclusion:Slight improvement in quality and slight increase in cost It paysMarch 17, 2017

©2003-2017 Vesa Välimäki 50

Soundbox Modeling

• Soundbox amplifies and colors the string sound

• Coloration is imitated with a digital filter– Soundboard resonances

– Short string parts behind the bridge

– Highest octave of 4-foot register

(without dampers)

– Higher modes of string

(not properly damped)

March 17, 2017

Page 26: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

26

©2003-2017 Vesa Välimäki 51

Soundboard Impulse Response

March 17, 2017

©2003-2017 Vesa Välimäki 52

Digital Model of the Soundboard• Use the source-filter principle

• Modeling with a reverb algorithm (Välimäki et al. 2004)

– Modification of the Feedback Delay Network

– Also 10th-order equalizer

DL1 H1(z) A1(z)

DLN AN(z)HN(z)

x(n) y(n)...

gfb

Delay line Lowpass Allpass

March 17, 2017

Page 27: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

27

©2003-2017 Vesa Välimäki 53

Soundboard Impulse Response

• Measured

March 17, 2017

©2003-2017 Vesa Välimäki 54

Soundboard Impulse Response

• Modeled

March 17, 2017

Page 28: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

28

©2003-2017 Vesa Välimäki 55

• Well suited to synthesis of string and wind instruments– String or tube is modeled with a delay-line loop

– Excitation is either a sample or a nonlinear system

• Bowed string and wind instrument models use a model of the excitation

– Nonlinear mapping of input pressure or velocity

– Similarities to waveshaping synthesis

• Commuted synthesis of bowed strings also possible (Smith, 1995)

Digital Waveguide Models

March 17, 2017

©2003-2017 Vesa Välimäki 56

Definition: Digital Waveguide• Digital waveguide = bi-directional delay line (Smith, 1985)

– A model for 1-D wave propagation, such as a string or a tube

• Every sampling instant contains a sample of a propagating wave

• Interpretation of samples can be chosen– Displacement for strings; sound pressure or volume velocity for tubes

z-1

y+(n,k)

z-1 z-1 z-1

z-1 z-1 z-1 z-1

y-(n,k)

y(n,k)

March 17, 2017

Page 29: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

29

©2003-2017 Vesa Välimäki 57

Output Signal from Digital Waveguide• Principle of superposition: Sum of signals propagating in

opposite directions at the same point

z-1

y+(n,k)

z-1 z-1 z-1

z-1 z-1 z-1 z-1

y-(n,k)

y(n,k)

March 17, 2017

©2003-2017 Vesa Välimäki 58

Scattering in a Digital Waveguide• If acoustic impedance Z changes, scattering occurs

– For example, tube radius changes, as in the vocal tract (Kelly & Lochbaum, 1962)

– Part of the wave is reflected while part gets through, when cross-sectional area A changes

March 17, 2017

Page 30: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

30

©2003-2017 Vesa Välimäki 59

Kelly–Lochbaum Tube Model (1962)

March 17, 2017

©2003-2017 Vesa Välimäki 60

Lossless String Model• A lossless, rigidly terminated string

– Reflection coefficient –1 at both ends

– For theoretical use only

(Figure from: http://www-ccrma.stanford.edu/~jos/)March 17, 2017

Page 31: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

31

©2003-2017 Vesa Välimäki 61

Lossy Waveguide Model• Lowpass filters at the ends of the digital waveguide

March 17, 2017

©2003-2017 Vesa Välimäki 62

Single-Delay-Loop Model• If there’s only 1 input and 1 output, all loss filters can be

combined! (Smith, 1992; Karjalainen et al., 1998)

• A single-delay-loop model results– Looks a lot like the Karplus–Strong algorithm, but the filter is different

– Excitation to be stored in the delay line must imitate a pluck

March 17, 2017

Page 32: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

32

©2003-2017 Vesa Välimäki 63

Waveguide Wind Instrument Models• A woodwind model consists of 2 parts

1) Linear tube and open end (digital waveguide and filters)

2) Nonlinear coupling of excitation

• For example, clarinet model (Smith, 1986), flute model (Välimäki et al., 1992), brass instrument models (Cook, 1992)

March 17, 2017

©2003-2017 Vesa Välimäki 64

Products Using Waveguide Synthesis• Physical modeling synthesizers

– Yamaha VL-1 & VL-7, 1994

– Korg Prophecy, 1995

• Sound cards– Creative Sound Blaster AWE64

– Creative Sound Blaster Live!

• Technology patented by Stanford University and Yamaha

Yamaha VL-1

VL-1 sound examples• Trumpet

• Tenor sax

• Shakuhachi

(Sound examples from: http://www-ccrma.stanford.edu/~jos/waveguide/Sound_Examples.html)March 17, 2017

Page 33: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

33

Waveguide Modeling of the Clavinet

©2003-2017 Vesa Välimäki

65

March 17, 2017

Modeling and Synthesis of the Clavinet

• A waveguide model with novel features (Gabrielli et al., 2013) – Excitation pulse model (velocity-dependent tangent hit) – Damping (yarn) model with different string length– Nonlinear pickup model– Equalization filters simulating the tone switches

• Collaboration between Aalto University, Università Politecnica delle Marche, Ancona, Italy & Univ. of Edinburgh, UK – Leonardo Gabrielli, Dr. Stefan Bilbao...

©2003-2017 Vesa Välimäki

66

March 17, 2017

Page 34: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

34

Waveguide Clavinet Synthesizer

©2003-2017 Vesa Välimäki

67

• Synthesis with good yarning Played from MIDI keyboard

• Synthesis with bad yarning

March 17, 2017

Electric Guitar Modeling and Synthesis

• A waveguide model with novel ideas (Lindroos et al., 2011)

– A new plucking model (pulse + filtered noise) – Time-varying loop gain (to account for two-stage decay) – Inharmonic comb filter model for the pickup position – Distortion helps much

©2003-2017 Vesa Välimäki

68

March 17, 2017

Page 35: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

35

©2003-2017 Vesa Välimäki 69

• Digital model of an electric guitar with hand movement recognition• Three alternative implementations:

1) Data gloves, 2) Colorful gloves and a web camera,3) Control sticks (acceleration sensors & ultra-sound radar)

http://airguitar.tml.hut.fi/

TKK Virtual Air Guitars

March 17, 2017

©2003-2017 Vesa Välimäki 70

Virtual Slide Guitar• An air guitar played with a bottle neck! (Pakarinen, Puputti &Välimäki, 2008)

Check our video: https://www.youtube.com/watch?v=eCPFYKq5zTk

March 17, 2017

Page 36: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

36

©2003-2017 Vesa Välimäki 71

Conclusions• Several methods available for physics-based sound synthesis

– Source-filter, modal, mass-spring, FDTD, wave digital, and digital waveguide models

• Digital waveguide synthesis used to be the most popular method

• With faster computers, modal synthesis and FDTD methods are becoming more attractive

March 17, 2017

©2003-2017 Vesa Välimäki 72

• J. M. Adrien, “The missing link: modal synthesis”, in: G. De Poli, A. Piccialli, and C. Roads, eds. Representations of Musical Signals. MIT Press, Cambridge, Massachusetts, 1991.

• B. Bank, “Physics-based sound synthesis of the piano,” Report no. 54, TKK Lab. of Acoustics and Audio Signal Processing, June 2000. Report and sound examples available online at: http://www.acoustics.hut.fi/~bbank/thesis.html

• C. Cadoz, A. Luciani, and J.-L. Florens, “Responsive input devices and sound synthesis by simulation of instrumental mechanisms: the CORDIS system,” Computer Music J., vol. 8 pp. 60–73, 1983.

• L. Gabrielli, V. Välimäki, H. Penttinen, S. Squartini & S. Bilbao, “A digital waveguide based approach for Clavinet modeling and synthesis,” EURASIP J. Applied Signal Processing, 2013. Audio examples at: http://a3lab.dii.univpm.it/projects/jasp-clavinet

• L. Hiller and P. Ruiz, “Synthesizing musical sounds by solving the wave equation for vibrating objects: parts I and II,” J. Audio Eng. Soc., vol. 19, no. 6, pp. 462–270 and no. 7, pp. 542–550, 1971

• D. Jaffe and J. O. Smith, “Extensions of the Karplus-Strong plucked string algorithm,” Computer Music Journal, vol. 7, no. 2, pp. 56–69, 1983.

References (1)

March 17, 2017

Page 37: Physics-Based Sound Synthesis - Aalto · 2017-03-17 · 2 ©2003-2017 Vesa Välimäki 3 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish)• Classification

37

©2003-2017 Vesa Välimäki 73

References (2)• M. Karjalainen, V. Välimäki, and T. Tolonen, “From the Karplus-Strong algorithm to digital

waveguide and beyond,” Computer Music J., vol. 22, no. 3, pp. 17–32, 1998. Available online at: http://www.acoustics.hut.fi/~vpv/cmj98.html

• K. Karplus and A. Strong, “Digital synthesis of plucked string and drum timbres,” Computer Music Journal, vol. 7, no. 2, pp. 43–55,1983.

• M. Laurson, C. Erkut, V. Välimäki, and M. Kuuskankare, “Methods for Modeling Realistic Playing in Acoustic Guitar Synthesis,” Computer Music J., vol. 25, no. 3, pp. 38–49, 2001.

• N. Lindroos, H. Penttinen, and V. Välimäki, “Parametric electric guitar synthesis,” Computer Music Journal, vol. 35, no. 3, pp. 18–27, Fall 2011.

• J. Morrison and J. M. Adrien, “Control mechanisms in the MOSAIC synthesis program,” in Proc. Int. Computer Music Conf., pp. 19–22, Montreal, 1991.

• J. Pakarinen, T. Puputti, and V. Välimäki, “Virtual slide guitar,” Computer Music Journal, vol. 32, no. 3, pp. 42–54, Fall 2008.

• Pakarinen, J., Tikander, M., and Karjalainen, M., ''Wave Digital Modeling of the Output Chain of a Vacuum-Tube Amplifier,'' in Proc. Int. Conf. Digital Audio Effects (DAFx'09), Como, Italy, Sept., 2009.

• H. Penttinen, J. Pakarinen, V. Välimäki, M. Laurson, H. Li, and M. Leman, “Model-based sound synthesis of the guqin,” Journal of the Acoustical Society of America, vol. 120, no. 6, pp. 4052–4063, Dec. 2006.

March 17, 2017

©2003-2017 Vesa Välimäki 74

References (3)• J. O. Smith, “Viewpoints on the history of digital synthesis,” in Proc. Int. Computer Music Conf.

(ICMC’91), pp. 1–10, Montreal, Canada, Oct. 1991. A revised version is available at: http://www-ccrma.stanford.edu/~jos/kna/

• J. O. Smith, “Physical modeling using digital waveguides,” Computer Music Journal, vol. 16, no. 4, pp. 74–91,1992. Available on-line at http://www-ccrma.stanford.edu/~jos/

• T. Tolonen, V. Välimäki, and M. Karjalainen, Evaluation of Modern Sound Synthesis Methods. Report no. 48, Lab. of Acoustics and Audio Signal Processing. Helsinki Univ. of Tech., Espoo, 1998. http://www.acoustics.hut.fi/~ttolonen/sound_synth_report.html

• V. Välimäki, J. Huopaniemi, M. Karjalainen, and Z. Jánosy, “Physical modeling of plucked string instruments with application to real-time sound synthesis,” J. Audio Eng. Soc., vol. 44, no. 5, pp. 331–353, May 1996.

• V. Välimäki, M. Laurson, and C. Erkut, “Commuted waveguide synthesis of the clavichord,” Computer Music Journal, vol. 27, no. 1, pp. 71–82, Spring 2003.

• V. Välimäki, J. Parker, and J. S. Abel, “Parametric spring reverberation effect,” Journal of the Audio Engineering Society, vol. 58, no. 7/8, pp. 547–562, July/August 2010.

• V. Välimäki, H. Penttinen, J. Knif, M. Laurson, and C. Erkut, “Sound synthesis of the harpsichord using a computationally efficient physical model,” EURASIP Journal on Applied Signal Processing, vol. 2004, no. 7, pp. 934–948, June 2004.

• V. Välimäki, J. Pakarinen, C. Erkut, and M. Karjalainen, “Discrete-time modelling of musical instruments,” Reports on Progress in Physics, vol. 69, no. 1, pp. 1-78, Jan. 2006.

March 17, 2017