Post on 24-May-2015
description
tugraz
Expressive Sound SynthesisFor Animation
Cécile Picard-Limpens
University of Nice/Sophia-AntipolisÉcole Doctorale STIC
REVES INRIA Sophia-Antipolis, FranceAdvisors: George Drettakis, INRIA Sophia Antipolis (Reves)
François Faure, INRIA Rhône-Alpes (Evasion)Nicolas Tsingos, DOLBY Laboratories, CA, USA
Defense for Ph.D. in Computer Science
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation1
tugraz
Outline
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation2
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Sound Renderingfor Virtual Reality and Games
Interactive Audio Rendering
(R. Vantielcke - WipeoutHD on Playstation 3)
Traditional ApproachPre-Recordings Triggered
+ : Easy to implement– : Repetitive audio, discrepancies, lack of flexibility
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation3
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Sound Renderingfor Virtual Reality and Games
Interactive Audio Rendering
(R. Vantielcke - WipeoutHD on Playstation 3)
Traditional ApproachPre-Recordings Triggered
+ : Easy to implement– : Repetitive audio, discrepancies, lack of flexibility
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation3
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
From Playback of Samplesto Synthesis
Digital Sound SynthesisSource modeling ←↩Sound propagation, Sound reception
(ArtiSynth)
TechniquesRigid body simulationFinite Element Method (FEM)
Physical Sound Simulation+ : Physical approach, easy parametrization,
Low memory usage– : Preprocess computation,
Interface between physics and sound system
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation4
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
From Playback of Samplesto Synthesis
Digital Sound SynthesisSource modeling ←↩Sound propagation, Sound reception
(ArtiSynth)
TechniquesRigid body simulationFinite Element Method (FEM)
Physical Sound Simulation+ : Physical approach, easy parametrization,
Low memory usage– : Preprocess computation,
Interface between physics and sound system
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation4
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Controlling the Sound SimulationChallenges
Sound Coherent With Visuals
Unpredictable character of soundsReal-time sound synthesis
Parametrization and Expressiveness
Control and interactivityAuthoring
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation5
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Our ContributionThree Research Axes
Physics-Based Sound synthesisContact modelingResonator modeling
Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation
Perspectives on a Hybrid Model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation6
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Our ContributionThree Research Axes
Physics-Based Sound synthesisContact modelingResonator modeling
Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation
Perspectives on a Hybrid Model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation6
Sound andVirtuality
General Background
Motivation
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Our ContributionThree Research Axes
Physics-Based Sound synthesisContact modelingResonator modeling
Example-Based Sound SynthesisAutomatic analysis of pre-recordingsFlexible synthesis for physics-driven animation
Perspectives on a Hybrid Model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation6
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Overview
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation7
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Sound from Contacts
DichotomyImpactsContinuous contacts
Two Schemes for Contact Force Modelling
Additive synthesis
Feed-forward scheme[van den Doel et al . ′01]
Bristle model
Direct computation of contact forces[Avanzini et al . ′02]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation8
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Contact Modeling
What Are The Current Limitationsfor Continuous Contacts?
Rate for physics engine reportNo geometric details when using visual texturesAuthoring and control are challenging
HOW Can We Solve Them?By extracting
Excitation profiles from visual textureswithAdaptive resolution[Picard et al ., VRIPHYS ′08]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation9
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Contact Modeling
What Are The Current Limitationsfor Continuous Contacts?
Rate for physics engine reportNo geometric details when using visual texturesAuthoring and control are challenging
HOW Can We Solve Them?By extracting
Excitation profiles from visual textureswithAdaptive resolution[Picard et al ., VRIPHYS ′08]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation9
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Method for Impact Sounds
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation10
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Method for Continuous Contact SoundsExtraction of Excitation Profiles
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation11
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Synthesis of Excitation ProfilesFor the Audio Force Modelling
Technique
Extraction from the visual texture imageRe-sampling along the trajectoryof the contact interaction (60Hz vs 44kHz)
Based on the Complexity of the Histogram
Simple texture image:Gradient of the image intensity
Complex texture image:Isocurves of constant brightness (isophotes)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation12
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Complex TexturesCoding the Excitation Profiles
Isophotes = Large amount of dataHow Can We Lighten the Info?
By Coding the Excitation Profiles= Main Features + Noise Part
= +Noise Part: Statistical approximation
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation13
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Real-Time Audio ManagementA Flexible Audio Pipeline
Simulations Driven by Ageia’s PhysX (now NVIDIA)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation14
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Audio Texture SynthesisA Solution for Interactive Simulations
A Sound in Coherence with Visuals
Flexible Resolution
Adapted to Procedural Generation
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation15
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Overview
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation16
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Vibration ModelsModal Analysis
Generating Sounds Based on Physics SimulationIn computer musics[Iovino et al . ′97, Cook ′02]
In computer graphics[Van Den Doel ′01, O′Brien et al . ′02]
Improvements for Interactive Sound RenderingModal parameter tracking[Maxwell et al . ′07]
Frequency content sparsity[Bonneel et al .′08]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation17
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Vibration ModelsModal Analysis
1 Get a Sounding Object and its Geometry
2 Construct the FEM (ex: Tetrahedral Mesh)3 Apply Newton Second Law to DOF
Md + Cd + Kd = f (1)
4 Eigendecomposition ⇒ Modal Parameters
M = LL−T ; L−1KL−T = V ΛV T (2)where V = matrix of eigenvectors
Λ = diagonal matrix of eigenvalues
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation18
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Vibration ModelsModal Analysis
In Real-time:Modal synthesis
s(t) =∑1
nai sin(wi t)e−di t (3)
Control for vibration models
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation19
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Vibration ModelsModal Analysis
What AreThe Current Limitations?
Meshing is difficultNo real control on the FEM resolutionNo clear interface between physics and audio
HOW Can We Solve Them?By proposing
A robust and multi-scale modal analysiswhich isCoherent with the physics simulation[Picard et al ., DAFx ′09]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation20
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Vibration ModelsModal Analysis
What AreThe Current Limitations?
Meshing is difficultNo real control on the FEM resolutionNo clear interface between physics and audio
HOW Can We Solve Them?By proposing
A robust and multi-scale modal analysiswhich isCoherent with the physics simulation[Picard et al ., DAFx ′09]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation20
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Our Deformation Model
Inspired from Work by Nesme et al.[Nesme et al .′06]
TechniqueMerged voxels used as Hexahedral Finite Elements
Implementation with the Sofa Framework
Validation of the ModelTests on a metal cube
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation21
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Robustness
Robust Even for Non-Manifold Geometries
Material: Aluminium
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation22
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Multi-Scale for Efficient Memory Usage
A Squirrel in Pine Wood
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation23
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Multi-Scale for Efficient Memory Usage
A Squirrel in Pine Wood: Different FE resolutions
3x3x3 4x4x4 8x8x8 9x9x9
Frequency Content = f (Hexahedral FE Resolution)
Higher resolution models
Frequency centroid shift
Convergence of the frequency content
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation24
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Comparison with Classical Approach
Sounding Bowl - Material: Aluminium
Classical Approach Our Approach(816 modes) (75 modes)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation25
Sound andVirtuality
Physics-BasedSynthesis
Contact Modeling
Audio TextureSynthesis For ComplexContacts
Resonator Modeling
A Robust andMulti-Scale ModalAnalysis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
A Robust and Multi-Scale Modal AnalysisA Solution for Sound Synthesis
Realistic
Adapted to Non-Manifold Geometries
Resources Flexibility
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation26
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Overview
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation27
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Implementation of Signal-Based Models
[Dobashi et al.′03
]
Concatenative Synthesis[Roads ′91, Schwarz ′06]
Sound Textures Based on Physics[Cook ′99][Dobashi et al . ′03, Zheng et al . ′09]
[Cook ′99
]Authoring and Interactive Control[Cook ′02]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation28
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Implementation of Signal-Based Models
What AreThe Current Limitations?
Processing is not genericParametrizing is difficult
HOW Can We Solve Them?By
Retargetting example soundsTo physics-driven animation[Picard et al ., AES ′09]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation29
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Implementation of Signal-Based Models
What AreThe Current Limitations?
Processing is not genericParametrizing is difficult
HOW Can We Solve Them?By
Retargetting example soundsTo physics-driven animation[Picard et al ., AES ′09]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation29
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Our Approach
SINUSOIDAL
1 DICTIONARY OF AUDIO GRAINS Impulsive / Continuous
2 CORRELATION PATTERNS
R E T A R G E T T I N G T O A N I M A T I O N
AUDIO RENDERER VIDEO RENDERER
PREPROCESSING
INTERACTIVE
OBJECT GEOMETRYVIRTUAL ENVIRONMENT
RIGID-BODYSIMULATION
BUILD COLLISIONSTRUCTURES
DEFINE PROCEDURES
ANIMATION WITH AUDIO
AUDIORECORDING
Our Contributions
TRANSIENT
Amplitude
Time
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation30
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Preprocess: A Generic Analysis
Impulsive and Continuous ContactsSpectral Modeling Synthesis (SMS) [Serra ′97]
Automatic Extraction of Audio GrainsDictionary: Impulsive/Continuous
Generation of Correlation Patternsbetween original recordings and audio grains
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation31
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
On-Line: Flexible Sound Synthesis
Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude
Interactive Physics-Driven AnimationsPhysics Info for Retargetting
Contact type: impulsive or continuous?Penetration force and relative velocity
Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes
Spectral domain adaptation/modification
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation32
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
On-Line: Flexible Sound Synthesis
Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude
Interactive Physics-Driven AnimationsPhysics Info for Retargetting
Contact type: impulsive or continuous?Penetration force and relative velocity
Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes
Spectral domain adaptation/modification
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation32
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
On-Line: Flexible Sound Synthesis
Resynthesis of the Original RecordingsCandidate grains: max. correlation amplitude
Interactive Physics-Driven AnimationsPhysics Info for Retargetting
Contact type: impulsive or continuous?Penetration force and relative velocity
Flexible Audio Shading ApproachAdditional, User-defined Resynthesis Schemes
Spectral domain adaptation/modification
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation32
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Resynthesis of the Original Recordings
94 recordings (14.6Mb)≈ 5000 grains + 94 Correlation Patterns (20% Gain)
Breaking Glass
Shooting Gun
Rolling
Additional Material:http://www-sop.inria.fr/members/Cecile.Picard/"‘Supplemental AES"’
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation33
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Flexible Audio Shading Approach
Easy Implementation of Time-Scaling
Faster RollingSlower Breaking
Synthesis of An Infinity Similar Audio Eventsby varying the audio content
Rythmic pattern from Breaking StoneNew material content: stone and gun
Rythmic pattern from Breaking GlassNew material content: ceramic
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation34
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Interactive Physics-Driven Animations
Simulations Driven by Sofa Framework
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation35
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Flexible SoundSynthesis
Retargetting ExampleSounds
Perspectives ona Hybrid Model
Conclusion andDiscussion
tugraz
Retargetting Example SoundsA Solution for Interactive Simulations
Variety
Adapted to Scenarios
Small Memory FootprintReal-Time Rendering
An attractive solution for industrial applications(Eden Games, an ATARI game studio)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation36
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Overview
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation37
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Sound ModelingWhen Nonlinearity Occurs
Problems of Single ModelsVibration models assume linearityExample-based sounds are hard to parametrize
Previous WorkModeling nonlinearities[O′Brien et al . ′01, Chadwick et al . ′09][Cook ′02]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation38
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Sound ModelingWhen Nonlinearity Occurs
Problems of Single ModelsVibration models assume linearityExample-based sounds are hard to parametrize
Previous WorkModeling nonlinearities[O′Brien et al . ′01, Chadwick et al . ′09][Cook ′02]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation38
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Fracture Events
BackgroundFrequently occur in virtual environments
Visual rendering[O′Brien et al . ′99, ′02][Parker and O′Brien. ′09]
Sound rendering: Little research[Warren et al . ′84] [Rath et al . ′03]
ChallengesEvent depends on the material involvedDifferents phases emerge from fracture event
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation39
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Fracture Events
BackgroundFrequently occur in virtual environments
Visual rendering[O′Brien et al . ′99, ′02][Parker and O′Brien. ′09]
Sound rendering: Little research[Warren et al . ′84] [Rath et al . ′03]
ChallengesEvent depends on the material involvedDifferents phases emerge from fracture event
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation39
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Parametrization of Our Hybrid Model
Selection CriteriaHybrid model applied when nonlinearity occurs
Techniques
FM synthesis
FM synthesisAudio grains
ParametrizationSmooth transition with vibration modelCoherence inside the hybrid model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation40
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Motivation
A Hybrid Model forFracture Events
Conclusion andDiscussion
tugraz
Discussion
Prospective model
Possible problem: report from the physics engine
Simplicity of the tools allows real-time rendering
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation41
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Overview
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation42
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Synthesis of Sounds for AnimationDifficulties
Audio-Visual Coherence
Extremely Dynamic Character
Precision of Synthesis
Large Variety of Objects
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation43
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
ContributionsAn Overview
Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms
Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio
Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation44
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
ContributionsAn Overview
Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms
Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio
Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation44
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
ContributionsAn Overview
Complex Contact Modeling2D visual textures used as roughness mapsAudible and position-dependent variationsDetail-layer mechanisms
Improved Modal Analysis for Resonator ModelingComplex non-manifold geometries can be handledMulti-scale resolutionCoherence between simulation and audio
Flexibility of Sound DesignAudio grains and correlation patternsDynamic retargetting to eventsExtended sound synthesis capabilities
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation44
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
ContributionsPerspectives
A Prospective Hybrid Modelfor Complex Physical Phenomena
Focus on NonlinearityCombination of physically basedand example-based methodsApplication Case: Fracture Events
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation45
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Overview
1 Sound and Virtuality
2 Physics-Based Sound SynthesisContact ModelingResonator Modeling
3 Example-Based SynthesisFlexible Sound Synthesis
4 Perspectives on a Hybrid ModelMotivation and Application
5 Conclusion and DiscussionContributionsExtensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation46
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Promising Directions for Future Work
Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments
Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene
Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation47
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Promising Directions for Future Work
Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments
Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene
Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation47
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Promising Directions for Future Work
Complex Contact ModelingTwo interacting texturesSurface-based interactionsAdequate perceptual experiments
Improved Modal Analysis for Resonator ModelingRecent work from [Nesme et al . Siggraph′09]Investigations with GPU for in-line computationComplete integration in a virtual scene
Example-Based TechniqueClustering of similar grainsStatistical analysis of correlation patternsPhysics engine design
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation47
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Promising Directions for Future Work
Hybrid Model for Fracture EventsFracture sound simulation frameworkTracking of relevant physical data
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation48
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Conclusion
New Physically Based Algorithmsfor Sound RenderingFlexibility of Sound ModelingIdeas on an Adequate Hybrid Sound Model
Additional info:http://www-sop.inria.fr/members/Cecile.Picard/
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation49
Sound andVirtuality
Physics-BasedSynthesis
Example-BasedSynthesis
Perspectives ona Hybrid Model
Conclusion andDiscussion
Contributions
Extensions andApplications
tugraz
Acknowledgements
George Drettakis, François Faure,and Nicolas Tsingos
REVES TeamMarie-Paule Cani and the Evasion TeamPaul G. Kry at the McGill University, Montréal
Eden Games, an ATARI game studio, Lyon
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation50