Concept of a Time Frequency Masking Model based onGabor filters and 2D convolution
Peter Balazs
Acoustics Research Institute (ARI)Austrian Academy of Sciences
in cooperation with:NuHAG Vienna
LATP, CMI & LMA, CNRS Marseille
FYMA, UCL Louvain-La-Neuve
Amade workshop05.10.2006
Peter Balazs (ARI) Multipliers & Masking 1 / 24
Overview:
1 IntroductionMP3
2 Simultaneous Masking
3 Gabor Filters
4 Time Frequency Masking
5 Current Research and Conclusion
Peter Balazs (ARI) Multipliers & Masking 2 / 24
Overview:
1 IntroductionMP3
2 Simultaneous Masking
3 Gabor Filters
4 Time Frequency Masking
5 Current Research and Conclusion
Peter Balazs (ARI) Multipliers & Masking 2 / 24
Overview:
1 IntroductionMP3
2 Simultaneous Masking
3 Gabor Filters
4 Time Frequency Masking
5 Current Research and Conclusion
Peter Balazs (ARI) Multipliers & Masking 2 / 24
Overview:
1 IntroductionMP3
2 Simultaneous Masking
3 Gabor Filters
4 Time Frequency Masking
5 Current Research and Conclusion
Peter Balazs (ARI) Multipliers & Masking 2 / 24
Overview:
1 IntroductionMP3
2 Simultaneous Masking
3 Gabor Filters
4 Time Frequency Masking
5 Current Research and Conclusion
Peter Balazs (ARI) Multipliers & Masking 2 / 24
Application : MP3-Player
• MP3• encoding / decoding scheme• MPEG1/MPEG2 (Layer 3)• signal processing• psychoacoustical model
• Signal Processing• applications: mobile phone,
UMTS, xDSL or digitaltelevision
• engineering field• mathematical field
• Psychoacoustic Masking
Peter Balazs (ARI) Multipliers & Masking 3 / 24
Application : MP3-Player
• MP3• encoding / decoding scheme• MPEG1/MPEG2 (Layer 3)• signal processing• psychoacoustical model
• Signal Processing• applications: mobile phone,
UMTS, xDSL or digitaltelevision
• engineering field• mathematical field
• Psychoacoustic Masking
Peter Balazs (ARI) Multipliers & Masking 3 / 24
Application : MP3-Player
• MP3• encoding / decoding scheme• MPEG1/MPEG2 (Layer 3)• signal processing• psychoacoustical model
• Signal Processing• applications: mobile phone,
UMTS, xDSL or digitaltelevision
• engineering field• mathematical field
• Psychoacoustic Masking
Peter Balazs (ARI) Multipliers & Masking 3 / 24
Psychoacoustic Masking: introduction
Masking:presence of one stimulus, the masker, decreases the response to anotherstimulus, the target.
Irrelevance Filter: searches (and deletes) perceptional irrelevant data (incomplex signals) using a masking model, supposing that one audiocomponent masks the others.
Typical Application:
1 Sound / Data Compression
2 Sound Design
3 Background - Foreground Separation
4 Improvement of Speech or Music Recognition
Peter Balazs (ARI) Multipliers & Masking 4 / 24
Psychoacoustic Masking: introduction
Masking:presence of one stimulus, the masker, decreases the response to anotherstimulus, the target.
Irrelevance Filter: searches (and deletes) perceptional irrelevant data (incomplex signals) using a masking model, supposing that one audiocomponent masks the others.
Typical Application:
1 Sound / Data Compression
2 Sound Design
3 Background - Foreground Separation
4 Improvement of Speech or Music Recognition
Peter Balazs (ARI) Multipliers & Masking 4 / 24
Psychoacoustic Masking: introduction
Masking:presence of one stimulus, the masker, decreases the response to anotherstimulus, the target.
Irrelevance Filter: searches (and deletes) perceptional irrelevant data (incomplex signals) using a masking model, supposing that one audiocomponent masks the others.
Typical Application:
1 Sound / Data Compression
2 Sound Design
3 Background - Foreground Separation
4 Improvement of Speech or Music Recognition
Peter Balazs (ARI) Multipliers & Masking 4 / 24
Psychoacoustic Masking : existing algorithm I
Existing algorithm in : simple model, but effective algorithm!Original audio file
”LossyCoding”
Peter Balazs (ARI) Multipliers & Masking 5 / 24
jumpref.wavMedia File (audio/wav)
Psychoacoustic Masking : existing algorithm II
Existing algorithm in : simple model, but effective algorithm!Irrelevance Filter
Original audio file
”Lossy Coding”
Peter Balazs (ARI) Multipliers & Masking 6 / 24
jumpmask.wavMedia File (audio/wav)
jumpdiff.wavMedia File (audio/wav)
Psychoacoustic Masking : existing algorithm III
Existing algorithm in : Original audio file (Spectrum)
Peter Balazs (ARI) Multipliers & Masking 7 / 24
Psychoacoustic Masking : existing algorithm IV
Existing algorithm in : Masked signal (Spectrum)
Peter Balazs (ARI) Multipliers & Masking 8 / 24
Psychoacoustic Masking : existing algorithm V
models simultaneousfrequency masking and usessingle spectra.
It calculates an adaptive threshold function for the spectra, but works witha time-frequency analysis. This is an adaptive Gabor Filter withcoefficients in {0, 1}.With Gabor theory some properties are explained:
• perfect reconstruction,• time frequency concentration,• smoothness and• numerical efficiency.
Peter Balazs (ARI) Multipliers & Masking 9 / 24
Psychoacoustic Masking : existing algorithm V
models simultaneousfrequency masking and usessingle spectra.
It calculates an adaptive threshold function for the spectra, but works witha time-frequency analysis. This is an adaptive Gabor Filter withcoefficients in {0, 1}.With Gabor theory some properties are explained:
• perfect reconstruction,• time frequency concentration,• smoothness and• numerical efficiency.
Peter Balazs (ARI) Multipliers & Masking 9 / 24
Psychoacoustic Masking : psychoacoustical experiments
Condition Correct Score1 68.6 %2 58.5 %3 50.5 %4 49.4 %
Peter Balazs (ARI) Multipliers & Masking 10 / 24
Psychoacoustic Masking : temporal masking
Also temporal masking exists:
Temporal and frequency masking have been investigated a lot. But thereare only very few studies for time frequency behaviour.
Goal: Investigate a true time-frequency masking model based on GaborFilters. Develop an efficient irrelevance algorithm!
Peter Balazs (ARI) Multipliers & Masking 11 / 24
Psychoacoustic Masking : temporal masking
Also temporal masking exists:
Temporal and frequency masking have been investigated a lot. But thereare only very few studies for time frequency behaviour.
Goal: Investigate a true time-frequency masking model based on GaborFilters. Develop an efficient irrelevance algorithm!
Peter Balazs (ARI) Multipliers & Masking 11 / 24
Psychoacoustic Masking : temporal masking
Also temporal masking exists:
Temporal and frequency masking have been investigated a lot. But thereare only very few studies for time frequency behaviour.
Goal: Investigate a true time-frequency masking model based on GaborFilters. Develop an efficient irrelevance algorithm!
Peter Balazs (ARI) Multipliers & Masking 11 / 24
What is a Gabor Filter?
Peter Balazs (ARI) Multipliers & Masking 12 / 24
Example for Gabor Filter = Gabor Multiplier
Original audio file:
Peter Balazs (ARI) Multipliers & Masking 13 / 24
jump1.wavMedia File (audio/wav)
Example for Gabor Filter = Gabor Multiplier
Symbol:
Peter Balazs (ARI) Multipliers & Masking 14 / 24
Example for Gabor Filter = Gabor Multiplier
Result of Gabor Multiplier.
Peter Balazs (ARI) Multipliers & Masking 15 / 24
result_jump1.wavMedia File (audio/wav)
Time Frequency Masking
Peter Balazs (ARI) Multipliers & Masking 16 / 24
Psychoacoustic Masking : simultaneous masking I
Existing Model, using bark scale
Peter Balazs (ARI) Multipliers & Masking 17 / 24
Psychoacoustic Masking : simultaneous masking I
Existing Model, using bark scale
Peter Balazs (ARI) Multipliers & Masking 17 / 24
Psychoacoustic Masking : simultaneous masking I
Existing Model, using bark scale
Peter Balazs (ARI) Multipliers & Masking 17 / 24
Time Frequency Masking : using Gabor multipliers
Simple concept for a time frequency masking model and algorithm:
Convolution with Gabor transform ⇒ Symbol for Gabor Filter.Other, more complex 2D convolution kernels are possible!
Peter Balazs (ARI) Multipliers & Masking 18 / 24
Time Frequency Masking : using Gabor multipliers
Simple concept for a time frequency masking model and algorithm:
Convolution with Gabor transform ⇒ Symbol for Gabor Filter.Other, more complex 2D convolution kernels are possible!
Peter Balazs (ARI) Multipliers & Masking 18 / 24
Time Frequency Masking : using Gabor multipliers
Simple concept for a time frequency masking model and algorithm:
Convolution with Gabor transform ⇒ Symbol for Gabor Filter.Other, more complex 2D convolution kernels are possible!
Peter Balazs (ARI) Multipliers & Masking 18 / 24
Time Frequency Masking : using Gabor multipliers
Simple concept for a time frequency masking model and algorithm:
Convolution with Gabor transform ⇒ Symbol for Gabor Filter.Other, more complex 2D convolution kernels are possible!
Peter Balazs (ARI) Multipliers & Masking 18 / 24
Time Frequency Masking : using Gabor multipliers
Simple concept for a time frequency masking model and algorithm:
Convolution with Gabor transform ⇒ Symbol for Gabor Filter.Other, more complex 2D convolution kernels are possible!
Peter Balazs (ARI) Multipliers & Masking 18 / 24
Current Research : psychoacoustical experiments
Masking effect of time frequency atoms is important =⇒ Current research.
=⇒ Sophie
-5
0
5
10
15
20
25
30
35
40
-5
0
5
10
15
20
25
30
35
40
-5
0
5
10
15
20
25
30
35
40
Qui
et T
hres
hold
s (d
B S
PL)
Signal Frequency (Hz)
S1 S4
S2
S3
S5
S6
2521 3181 4000 5015 6275 2521 3181 4000 5015 6275
300-msec Sinusoid3.3-msec Gaussian
Peter Balazs (ARI) Multipliers & Masking 19 / 24
Current Research : psychoacoustical experiments
Masking effect of time frequency atoms is important =⇒ Current research.=⇒ Sophie
-5
0
5
10
15
20
25
30
35
40
-5
0
5
10
15
20
25
30
35
40
-5
0
5
10
15
20
25
30
35
40
Qui
et T
hres
hold
s (d
B S
PL)
Signal Frequency (Hz)
S1 S4
S2
S3
S5
S6
2521 3181 4000 5015 6275 2521 3181 4000 5015 6275
300-msec Sinusoid3.3-msec Gaussian
Peter Balazs (ARI) Multipliers & Masking 19 / 24
Current Research
Peter Balazs (ARI) Multipliers & Masking 20 / 24
Current Research
• Investigate the mathematical and numerical background for a Gaborfilter further: (cooperation with NuHAG/LATP/UCL)
• Bessel and frame multipliers• irregular Gabor multipliers• numerics of discrete Gabor analysis
• Psychoacoustic experiment: (cooperation with LMA)• thresholds for Gaussian atoms• time frequency masking effects between two atoms• more complex sounds
• Algorithms: (cooperation with LMA)• efficient implementation (real time)• experimental testing of its validity for different applications
Peter Balazs (ARI) Multipliers & Masking 21 / 24
Current Research
• Investigate the mathematical and numerical background for a Gaborfilter further: (cooperation with NuHAG/LATP/UCL)
• Bessel and frame multipliers• irregular Gabor multipliers• numerics of discrete Gabor analysis
• Psychoacoustic experiment: (cooperation with LMA)• thresholds for Gaussian atoms• time frequency masking effects between two atoms• more complex sounds
• Algorithms: (cooperation with LMA)• efficient implementation (real time)• experimental testing of its validity for different applications
Peter Balazs (ARI) Multipliers & Masking 21 / 24
Current Research
• Investigate the mathematical and numerical background for a Gaborfilter further: (cooperation with NuHAG/LATP/UCL)
• Bessel and frame multipliers• irregular Gabor multipliers• numerics of discrete Gabor analysis
• Psychoacoustic experiment: (cooperation with LMA)• thresholds for Gaussian atoms• time frequency masking effects between two atoms• more complex sounds
• Algorithms: (cooperation with LMA)• efficient implementation (real time)• experimental testing of its validity for different applications
Peter Balazs (ARI) Multipliers & Masking 21 / 24
Conclusion
• Gabor Filter Theory gives an insight on time-variantfiltering.
• A simple Time Frequency Masking model exists.Further investigation starts at the time frequencyatom level .
• A lot of interesting research, both applied andtheoretical, still has to be done.
Peter Balazs (ARI) Multipliers & Masking 22 / 24
Conclusion
• Gabor Filter Theory gives an insight on time-variantfiltering.
• A simple Time Frequency Masking model exists.Further investigation starts at the time frequencyatom level .
• A lot of interesting research, both applied andtheoretical, still has to be done.
Peter Balazs (ARI) Multipliers & Masking 22 / 24
Conclusion
• Gabor Filter Theory gives an insight on time-variantfiltering.
• A simple Time Frequency Masking model exists.Further investigation starts at the time frequencyatom level .
• A lot of interesting research, both applied andtheoretical, still has to be done.
Peter Balazs (ARI) Multipliers & Masking 22 / 24
Personal References:
P. Balazs, Regular and Irregular Gabor Multipliers with Application toPsychoacoustic Masking, PhD Thesis, Universität Wien (2005)
P. Balazs, Basic Definition and Properties of Bessel Multipliers, Journal ofMathematical Analysis and Applications (in press, available online)
P. Balazs, B. Laback, G. Eckel and W. A. Deutsch, Perceptional Sparsity bySimultaneous Masking, preprint
P. Balazs, J.-P. Antoine, Weighted and controlled frames, submitted
Peter Balazs (ARI) Multipliers & Masking 23 / 24
Thank you for your attention!
Peter Balazs (ARI) Multipliers & Masking 24 / 24
IntroductionMP3
Simultaneous MaskingGabor FiltersTime Frequency MaskingCurrent Research and ConclusionReferences
Top Related