Intelligent Audio Systems: A review of the foundations and ... · Jay LeBoeuf Imagine Research...

Post on 22-Jul-2020

1 views 0 download

Transcript of Intelligent Audio Systems: A review of the foundations and ... · Jay LeBoeuf Imagine Research...

Jay LeBoeufImagine Research

jayatimagine-researchcom

July 2008

Intelligent Audio Systems A review of the foundations and applications of

semantic audio analysis and music information retrieval

WIKI REFERENCEShellip

These lecture notes contains hyperlinks to the CCRMA Wiki

On these pages you can find additional supplement the lecture material found in the class - providing extra tutorials support references for further reading or demonstration code snippets for those interested in a given topic

Click on the symbol on the lower-left corner of a slide to access additional resources

bull ccrmastanfordeduworkshops2008

bull Introductionsndash A little about yourselfndash Backgroundndash List of your region of interest and any specific items of

interest that yoursquod like to see covered

Administration

Final projects

Motivation by your personal interests or audio collections

Motivated by examining large audio collectionsExamples

httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type

ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc

Example Seedhellip

Queries and Evaluation

bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer

bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet

bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or

inferred)

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

WIKI REFERENCEShellip

These lecture notes contains hyperlinks to the CCRMA Wiki

On these pages you can find additional supplement the lecture material found in the class - providing extra tutorials support references for further reading or demonstration code snippets for those interested in a given topic

Click on the symbol on the lower-left corner of a slide to access additional resources

bull ccrmastanfordeduworkshops2008

bull Introductionsndash A little about yourselfndash Backgroundndash List of your region of interest and any specific items of

interest that yoursquod like to see covered

Administration

Final projects

Motivation by your personal interests or audio collections

Motivated by examining large audio collectionsExamples

httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type

ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc

Example Seedhellip

Queries and Evaluation

bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer

bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet

bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or

inferred)

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull ccrmastanfordeduworkshops2008

bull Introductionsndash A little about yourselfndash Backgroundndash List of your region of interest and any specific items of

interest that yoursquod like to see covered

Administration

Final projects

Motivation by your personal interests or audio collections

Motivated by examining large audio collectionsExamples

httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type

ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc

Example Seedhellip

Queries and Evaluation

bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer

bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet

bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or

inferred)

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Final projects

Motivation by your personal interests or audio collections

Motivated by examining large audio collectionsExamples

httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type

ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc

Example Seedhellip

Queries and Evaluation

bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer

bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet

bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or

inferred)

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Example Seedhellip

Queries and Evaluation

bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer

bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet

bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or

inferred)

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Queries and Evaluation

bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer

bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet

bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or

inferred)

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Opportunities

bull Genre ID (labels exist but even humans disagree)bull Artist classification

ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)

bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

FUTURE SOFTWARE CAPABILITIES

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

TempoKey 000000

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

TempoKey Analyzing tempohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey Calculating Bars Beatshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey Estimating Keyhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Instrument IDhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Instrument Groupinghellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Guitar Chord Analysishellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Transcribing Drumshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Transcribing Pianohellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Identifying Phraseshellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Segmenting Songhellip

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Tempo125 BPMKey A major Done

Track 1

Track 2

Track 3

Track 4

Track 5

Track 6

Track 7

Track 8

Track 9

Track 10

Kick

Snare

Drum OH L

Drum OH R

Track 1Dirty Guitar

Elec Bass

Male Vocals

Male Vocals

Clean Guitar

Piano

Kick

Snare

Drum OH L

Drum OH R

Male Vocals

Male Vocals

Dirty Guitar

Elec Bass

Clean Guitar

Piano

A C D G A C D G A C D G Em D Em D A C D G A

A A G A A G A A G A

Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus

Drums

Snare

Drum OH L

Drum OH R

Kick

Snare

Drum OH L

Drum OH R

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Why MIR

bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio

ndash Quaero ndash French ldquoGoogle killer with image and audio example queries

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull Retrieval based on similarity (IR and creative applications)

bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications

Commercial Applications

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Motivations Demos

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-

project-live-in-concertthe-methodology

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull Transcriptionist vs Descriptionist approach

ndash Music Transcription (restoration) ndash piano company from MIDI

bull httpzenphcomlistenhtml

- More transcription

Motivations Demos

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

BASIC SYSTEM OVERVIEW

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Basic system overview

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Segmentation

(Frames Onsets Beats Bars Chord

Changes etc)

Feature Extraction

(Time-based spectral energy

MFCC etc)

Analysis Decision Making

(Classification Clustering etc)

Basic system overview

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

TIMING AND SEGMENTATION

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

Timing and Segmentation

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

1 second1 second

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc

bull ldquoFramesrdquondash Different problems call for different frame lengths

bull Onset detectionbull Beat detection

ndash Beatndash Measure Bar Harmonic changes

bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue

Timing and Segmentation

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull What is an Onsetbull How to detect

ndash Envelope is not enoughndash Need to examine frequency bands

Onset detection

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Filterbank

Detect change in band 1

Detect change in band 2

hellip

Detect change in band N

Combine results

audio

Detection

output

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

1

2

3

4

5

6

7

8

octa

ve

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

threshold

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Adaptive

threshold

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull Beat detectionndash Tempo (eg 125 bpm)

bull Detecting periodicities from the onset detection curvendash Beat

bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic

modelndash Measure

bull Musical change rate bull Harmonic change rate

Beat and Tempo Detection

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Many many many approaches

Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is

translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied

Peak picking is applied to the autocorrelation function or to the spectrum representation

Beat and Tempo Detection

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

beats

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

FEATURE EXTRACTION

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Frame 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

FRAME 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Zero crossing rate = 9FRAME 1

ZERO CROSSING RATE

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

Frame 2

Zero crossing rate = 423

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

ANALYSIS AND DECISION MAKING

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition

bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares

bull Time for more sophistication

Heuristic Analysis

gt End of Lecture 1

gt End of Lecture 1