Physically Based Sound

COMP259 Nikunj Raghuvanshi

Overview

Background

FEM Simulation

Modal Synthesis (FoleyAutomatic)

Comparison/Conclusions

Motivation

Sounds could in-principle be produced automatically, just like graphics: Sound Rendering

Sound Rendering has not received much research effort

Main Goal: Automatic generation of non-music, non-dialogue sound

Sound Production Today

Movies: Foley Artistshttp://www.marblehead.net/foley/index.html

Games: Anyone noticed the huge sound directory in Unreal Tournament?

PBS: Sound Production in Nature

Collisions/Other interactions lead to surface vibrations

Vibrations create pressure waves in airPressure waves sensed by ear

Surface Vibration Pressure Wave Ear

Vibration Propagation Perception

Main Aims of PBS

Physics simulator gives contact/collision information

Assign material properties for sound, Wood, concrete, metal etc.

Sound simulator generates sound using this data (in real time?)

Challenges

Sound must be produced at a minimum of ~44,000 Hz

Extremely High Temporal Resolution (timesteps in the range of 10-6-10-8 s)

Stiffness of underlying systems (eg. Metallic sounds. K/m~=108)

Stability may require even smaller timesteps

Two Approaches

FEM deformable simulationO'Brien, J. F. et. al., “Synthesizing Sounds from Physically Based Motion.” SIGGRAPH 2001.

FoleyAutomatic (Modal Synthesis)Kees van den Doel et. Al., “FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation.” SIGGRAPH 2001.

Main ideas

Deformable Simulation (arguably) much more “physically based”

Foley Automatic: Additive Synthesis

Component Sinusoids

Sound Signal

Overview

Background

FEM Simulation

Modal Synthesis (FoleyAutomatic)

Comparison/Conclusions

Simulation Requirements

Temporal ResolutionSimulate Vibration as well as PropagationVibration Modeling: Deformable Model for

ObjectsPropagation Modeling: Explicit Surface

RepresentationPhysical/Perceptual Realism

System Structure

Vibration Modelling

FEM with Tetrahedral Elements Linear Basis Functions, green’s strain Explicit Time Integration Typically #nodes = 500, #elements = 1500,

dt = 10-6-10-7 s

Sound Propagation Modelling

Fluid Dynamic FEM simulation of surrounding air? Very expensive. Instead…

Employ Huygen’s Principle: Pressure Wave may be seen as sum of pressure wavelets

ReceiverReceiver

Pressure Wave Pressure

“Wavelets”

n̂ v

ds

nvzp ˆ

msPacz /415 Acoustic Impedance of Air

Surface Vibrations and Sound

Pressure contribution of a patch,

Velocity

Density of Air

Sound Propagation Speed in Air

Unit Normal

Surface Vibrations and Sound

Approximate differential elements with surface triangles

Apply band pass filters: Low pass: windowed sinc filter High pass: DC blocking filter

Result: Pressure known for all surface triangles

Putting it all together

)cos(~

)( rx

apts rx

Pressure/Signal at Receiver

Filtered Average Pressure

Area of Triangle

Visibility Term

Approximation of Beam Pattern

Distance Falloff

n̂

Receiver

r

Vibrationx̂

Propagation Delay

Accumulation Buffer

c

dDelay

Receiver

d1

d2

Source

t=0

t1= d1/c

t2= d2/c

1

2

Receiver Distance from Source

Sound Propagation Speed

Results: Capabilities

General models

Generated sounds are accurate

Stereo Sound

Doppler’s Effect

Demo

Results: Accuracy

Results: Speed

Scene TimeStep(s) Nodes/Elems Time/Audio Time

Bowl 10-6 387/1081 91.3/4.01 mins

Clamped Bar 10-7 125/265 240.4/1.26 mins

Vibraphone 10-7 539/1484 1309.7/5.31 mins

(~1 day)

Timings on a 350MHz SGI Origin MIPS R12K processor

Overview

Background

FEM Simulation

Modal Synthesis (FoleyAutomatic)

Comparison/Conclusions

Features

Modal resonance model of solids Location dependent sounds Impact, slide, roll excitation models Real-time, low latency Easy integration with simulation/animation Practical Do not model propagation of sound from source

to receiver

Synthesis Method

ForceForceVibrationVibrationEmissionEmission

PropagationPropagation ListenerListener SpeakersSpeakers

Sound SamplesSound Samples

User

Vibration

),(),(]1

),([2

2

2txFtxu

tcx

xg iii

i

Surface u(x,t) of body responds to external contact force F(x,t)

u(x,t)F(x,t)

Strain Functional Speed of Sound

Under suitable boundary conditions, the solution to the PDE is a sum of sinusoids

Emission

Sound pressure s(t) linear functional L of surface vibration u(x,t)

)],([)( txuLts i

u(x,t)Ls(t)

nvzp ii ˆ~

Note that propagation is not modeled in above

The Modal Synthesis Model

u(x,t) F(p,t)Ls(t)

Impulse response/modal model

“The response u(x,t) of an arbitrary solid object to an external force can be described as a weighted sum of damped sinusoids”

Since L is linear, it implies at s(t) must be a sum of damped sinusoids too

Example: A 1D string

1st Mode 2nd Mode Frequency = f0

…Higher modes Frequency = f1= 2*f0 Frequency = fk= k*f0

)2sin( 000 tfea td )2sin( 11

1 tfea td )2sin( tfea ktd

kk

Main Idea: Sum contributions of all the modes

The point of impact decides the proportions in which the modes are to be mixed: ak. Therefore, ak is a function of p, the point of impact

The frequencies and damping parameters are a property of the object, and independent of how the object is hit

+ +...+

a0a1 ak

The Modal Synthesis Model

u(x,t) F(p,t)Ls(t)

)2sin()()(1

tfepats ktd

N

kk

k

Impulse response,

modal model

Parameters measured experimentally

Kth mode: Gain Factor Point Damping Vibration of impact Term Frequency

Force Modeling

ImpactSlidingRolling

Wavetable

Stochastic

At runtime: Find gain parameters given the location, strength and kind of force.

Synthesize sound from previous equation.

Impact Forces

•Duration: hardness (T)•Magnitude: energy transfer (w)•Multiple micro-collisions

TtTtwtF 0)),/2cos(1()( Example:

Sliding/Scraping

Micro-collisions lead to noisy audio-force

Sliding/Scraping

Wavetable approach Store force parameters Modulate amplitude with energy transfer Modulate rate with contact speed

Synthesis Approach Fractal noise represents roughness Filter through reson filter Resonance ~ contact speed Width ~ randomness of surface

Rolling

No relative surface motion

Differences with sliding:•Smoother: Use low pass•More damping•Harder to create•Less understood•Essential coupling?

Rolling: Smooth Surfaces

Polyhedral objects do not lead to smooth rolling forces

Instead use smooth surfaces directly

Rolling: Contact Evolution

Evolve the contact in Reduced coordinates

q = (u,v,s,t, )

q q q .. .

c(u,v)

d(s,t)

Rolling: Contact Evolution

Piecewise parametric surfaces, loop subdivision surfaces

Explicit integration, no stabilization Multiple contacts and conforming contacts

are not handled Used only when multiple contacts in close

spatio-temporal proximity

Demo

Dynamic Forces

Contact force

Rolling speed

Slipping speed

Impulses

…and locations

Pebble-in-Wok Demo

Results

0.1% CPU time per mode Graceful degradation of quality The bell demo is interactive Uses a PHANToM for interaction Authors do not report any real timings State that “sound quality” is perception-

based and has no metric as of now

Overview

Background

FEM Simulation

Modal Synthesis (FoleyAutomatic)

Comparison/Conclusions

Discussion

FEM: Physically Rigorous and GeneralToo slow for interactive applicationsDoesn’t scale wellInappropriate to apply a 30fps technique to

44000fps?Maybe too general for the problem

domain?

Discussion

Modal model exploits the vibrational nature

Higher EfficiencyBut, not rigorously physically basedFinding the parameters requires

experimentation and “earballing”No rigorous correlation between physical

and perceptual parameters

Discussion

For Realtime: Need for a technique to cover the middle ground

Extracting modal parameters in general requires solving PDEs

Not possible to do in an automated manner

Approximate modal parameters and then use modal synthesis?

Conclusion

PBS involves orders of magnitude smaller temporal and spatial scales

Research is sparse, problems are denseMain contributions of the two papers

besides vibration modeling: FEM: Efficient modeling of sound propagation FoleyAutomatic: Efficient, Approximate models

to handle surface properties and contact forces

References

O'Brien, J. F., Cook, P. R., Essl G., "Synthesizing Sounds from Physically Based Motion." The proceedings of ACM SIGGRAPH 2001, Los Angeles, California, August 11-17, pp. 529-536.

Kees van den Doel, Paul G. Kry and Dinesh K. Pai, “FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation” Computer Graphics (ACM SIGGRAPH 01 Conference Proceedings), pp. 537-544, 2001.

Acknowledgements

Some images were taken from the referred papers and the corresponding SIGGRAPH slides