Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National...
-
Upload
ashley-marsh -
Category
Documents
-
view
212 -
download
0
Transcript of Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National...
![Page 1: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/1.jpg)
Onset Detection in Audio Music
J.-S Roger Jang (張智星 )
http://mirlab.org/jang
MIR Lab, CSIE Dept.
National Taiwan University
![Page 2: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/2.jpg)
-2-
What Are Note Onsets?
Energy profile of a percussive instrument is modeled as ADSR stages
Note onset is the time where the slope is the highest, during the attack time.
Soft onsets via violin, etc, are much harder to define and detect.
![Page 3: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/3.jpg)
-3-
Difficulty in Onset Detection
Music types Monophonic Easier
Polyphonic Harder
Instrument types Percussive instruments Easier
String instruments Harder (soft onsets)
![Page 4: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/4.jpg)
-4-
Why Onset Detection is Useful?
It is a basic step in music analysis Music transcription (from wave to midi) Music editing (Song segmentation) Tempo estimation Beat tracking Musical fingerprinting (the onset trace can serve as a robust id for fingerprinting)
![Page 5: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/5.jpg)
-5-
Onset Detection Function
ODF (onset detection function) creates a curve of onset strength, aka Onset strength curve
Novelty curve
Most ODFs are based on time-frequency representation (spectrogram) of Magnitude of STFT (Short-time Fourier transform)
Phase of STFT Mel-band of STFT Constant-Q transform
![Page 6: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/6.jpg)
-6-
ODF: Spectral Flux
Concept sum the positive change in each frequency bin
rectifier wave-half aka ,2
)(
size frame: index, freq: index, time:
)),1(),(()(1
xxxh
Nkn
knXknXhnsfN
k
),1( knX ),( knX
![Page 7: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/7.jpg)
-7-
Flowchart of OSC
Steps of OSC Spectrogram Mel-band spectrogram Spectral flux Smoothed OSC via Gaussian smoothing Trend of OSC via Gaussian smoothing Trend-subtracted OSC
Check out wave2osc.m to see these steps.
![Page 8: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/8.jpg)
-8-
Example of OSC
Try “wave2osc.m”
Time (sec)
Fre
q. b
in in
dex
Spectrogram
0.5 1 1.5 2 2.5 3 3.5 4 4.5
10
20
30
40
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
0.005
0.01
0.015
0.02
0.025
Time (sec)
Am
plitu
de
OSC (original and smoothed)
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
2
4
6
8x 10
-3
Time (sec)
Am
plitu
de
Smoothed OSC and its trend
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
1
2
3
4
5x 10
-3
Time (sec)
Am
plitu
de
Trend-subtracted OSC
![Page 9: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/9.jpg)
-9-
What Can You Do With OSC...
OSC onsets Pick peaks to have onsets
OSC tempo (BPM, beats per minute) Apply ACF (or other PDF) to find the BPM
OSC beat tracking Pick equal-spaced peaks to have beat positions
![Page 10: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/10.jpg)
-10-
Beat Tracking
Demos http://mirlab.org/demo/beatTracking
Try “beatTracking.m” in SAP toolbox
![Page 11: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/11.jpg)
-11-
Example of Beat Tracking
beatTracking.m
![Page 12: Onset Detection in Audio Music J.-S Roger Jang ( 張智星 ) MIR LabMIR Lab, CSIE Dept. National Taiwan University.](https://reader035.fdocuments.in/reader035/viewer/2022081007/56649cff5503460f949d0fc0/html5/thumbnails/12.jpg)
-12-
Performance Indices ofBeat Tracking
Many performance indices of BT Check out audio beat tracking task of MIREX
Mostly adopted ones Precision, recall, f-measure, accuracy
Try simSequence.m in SAP toolbox
0 1 2 3 4 5 6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
TP TP TPFP FP FP
FN FN
Computed
GT
Precision = tp/(tp+fp)=3/(3+3) = 0.5Recall = tp/(tp+fn)=3/(3+2) = 0.6F-measure = tp/(tp+(fn+fp)/2)=3/(3+(2+3)/2) = 0.545Accuracy = tp/(tp+fn+fp)=3/(3+2+3) = 0.375