Relation-wise Automatic Domain-Range Information Management for Knowledge Entries
Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain...
Transcript of Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain...
![Page 1: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/1.jpg)
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings
Stefan Balke and Meinard Mü[email protected]
![Page 2: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/2.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings2
Thanks to the Band!
ChristianDittmar
ThomasPrätzlich
JonathanDriedger
PatricioLópez-Serrano
ChristofWeiß
FrankZalkow
MeinardMüller
JakobAbeßer
![Page 3: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/3.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings3
Motivation
T T
![Page 4: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/4.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings4
Outline
§ Introduction
§ Definition of Structure
§ Principles of Automatic Structure Analysis
§ Structure Annotations in the WJD
§ Preliminary Results
![Page 5: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/5.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings5
Basic Structure
Head-In
Head-Out
Repetitions
T T
Instrument Comics by H. Grohganz: https://mir.sechsachtel.de/orchpics/
![Page 6: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/6.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings6
Solo-Centric Annotations
Transcription
Beats
| E7 A7 | D7 G7 | … Chords…
![Page 7: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/7.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings7
Song-Centric Annotations
T T
New annotations from the groups in Erlangen/Weimar!Payback Time
![Page 8: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/8.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings8
Annotation Example from the Weimar Jazz DBClifford Brown - Jordu
Time (s)
Song-Centric Annotations:§ Chorus boundaries§ Solo choruses§ Theme repetitions Thank you,
Moritz Berendes!
![Page 9: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/9.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings9
Automatic Structure Analysis (MIR)
§ Active research direction since ~20 years.
§ Main principles:
§ Repetition-based Structure Analysis
§ Homogeneity-based Structure Analysis
§ Novelty-based Structure Analysis
![Page 10: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/10.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings10
Self-Similarity Matrix1. Step: Extract Audio Features
Time (s)
MFC
C
§ Mel Frequency Cepstral Coefficients (MFCC) correlate to the timbre.
![Page 11: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/11.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings11
Self-Similarity Matrix2. Step: Calculate Pairwise Similarity
Time (s)
Tim
e (s
)
![Page 12: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/12.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings12
T T O
Self-Similarity MatrixAnalysis
TT
O
§ Repetitions:Path-like structures
§ Homogeneity:Block-like structures
§ Novelty:Corners
![Page 13: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/13.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings13
Self-Similarity MatrixNovelty Detection
TT
O Idea (Foote):
Use checkerboard-like kernelfunction to detect corner pointson main diagonal of SSM.
![Page 14: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/14.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings14
Self-Similarity MatrixChroma Features
Drum Solo
§ Chroma instead of MFCC§ Repetitions result in path-like
structures§ Head-In and Head-Out
T T O
TT
O
![Page 15: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/15.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings15
Self-Similarity MatrixRecap
TT
O
§ Repetitions:Path-like structures
§ Homogeneity:Block-like structures
§ Novelty:Corners
§ Features are important!
![Page 16: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/16.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings16
T T O
Preliminary ResultsClifford Brown – Jordu
Chroma MFCC
![Page 17: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/17.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings17
Preliminary ResultsJohn Coltrane – Blue Trane
Chroma MFCC
T T
![Page 18: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/18.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings18
Preliminary ResultsHerbie Hancock – Maiden Voyage
Chroma MFCC
I T T O
![Page 19: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/19.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings19
Retrieval Approach
§ Motivation: Lots of Recordings are available on YouTube.§ Use Solo Transcriptions for Version Identification§ Possible Sources: YouTube, Soundcloud, etc.
S1 S3T T O
WJD Transcription
Comparison
![Page 20: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/20.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings20
There's more than one way to bake a cake…Retrieval Approach
Transcription
Collection of PolyphonicMusic Recordings
T T
MatchingProcedure
vs.
![Page 21: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/21.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings21
Conclusions
§ WJD contains very cool annotations!
§ Additional song-centered annotations
§ Enrich listening experience
§ Annotations will be made available soon!
T T
![Page 22: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/22.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings22
Preliminary ResultsStan Getz – The Girl from Ipanema
Chroma MFCC
![Page 23: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/23.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings23
Implementation Details
§ Chroma:§ CENS-21-5 (2 Hz feature rate)
§ SSM:§ Forward-backward smoothing§ Tempo compensation (0.7 – 1.3, 5 steps)
§ MFCC:§ 10 coefficients (3 – 13)§ 41 frames smoothing, 5 downsampling (2 Hz feature rate)
§ Foote:§ Kernel size: 100 frames (30 s)
![Page 24: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/24.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings24
WJD – Modifications
§ Establish “song-centric” view in database§ Use time axis as absolute reference§ Annotate chorus boundaries§ Annotate missing solo boundaries
S1 S2 S3T T O
Chorus Boundary
![Page 25: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/25.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings25
T T O
![Page 26: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/26.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings26
T T
![Page 27: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/27.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings27
I T T O
![Page 28: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/28.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings28
Self-Similarity Matrix1. Step: Extract Audio Features
Time (s)
Chr
oma
C
F#
B
C#
……
§ Chroma Feature correlate to the harmonic and melodic progressions.
![Page 29: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/29.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings29
Self-Similarity Matrix2. Step: Calculate Pairwise Similarity
Time (s)
Tim
e (s
)
![Page 30: Using Domain Knowledge for Automatic Structure Analysis of … · 2017-10-20 · Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings 9 Automatic Structure Analysis](https://reader031.fdocuments.in/reader031/viewer/2022040820/5e68bfea827d325e6f401dc3/html5/thumbnails/30.jpg)
© AudioLabs, 2016Stefan Balke
Using Domain Knowledge for Automatic Structure Analysis of Jazz Recordings30
Self-Similarity MatrixAnalysis
T T
TT
Drum Solo
§ Repetitions result in path-like structures
§ Head-In and Head-Out