Melodic Features and Retrieval

ISMIR Graduate School, Barcelona 2004

Musicology 3-4

Frans Wiering, ICS, Utrecht University

Outline

yesterday’s assignment demo: MIR outside academia (7:20; 44:10) one-dimensional melody retrieval Gestalt view of melody advanced melody retrieval assignment

one-dimensional melody retrieval common assumption is (was?) pitch-only retrieval is

sufficient e.g. CCGGAAGGFFEEDDEC mechanisms for fuzzy matching

variants interval (distance between 2 pitches) pitch-contour

same/up/down (Parson’s Code) RURURDRDRDRDRUD

examples: www.musipedia.com (Rainer Typke) www.themefinder.org (CCARH)

Results from Musipedia

query is ranked 3 other hits are

very unlikely unfortunately no

notation/sound available

Haydn: evident false positive why?

Themefinder

Several 1-dimensional search options, e.g. pitch interval contour rhythm

wildcards each theme stored as a

number of strings matching by regular

expressions ca. 40.000 themes

Barlow and Morgenstern (1948)

ESAC encodings Lincoln, 16th Century Motet

(DARMS project)

results from Themefinder

Example from Byrd & Crawford (2001)

other hits not as far-fetched as

musipedia’s different rhythm different meter still not very similar

is this what people have in mind?

Query: +m2 +M2 P1 -M2 -m2 -M2

Nice one we’ve just discovered

www.tuneteller.com Pitch-only search of

MIDI on the internet many more MIR

systems in Rainer Typke’s survey. URL is in your mailbox

Why pitch-only retrieval is unsatisfactory information contribution of other 3 parameters

(estimate for Western music; Byrd & Crawford 2001) pitch: 50% rhythm: 40% timbre + dynamics: 10%

melodic confounds (Selfridge-Field 1998): rests repeated notes grace notes, ornamentation Mozart example

Why pitch-only retrieval is unsatisfactory information contribution of other 3 parameters

(estimate for Western music; Byrd & Crawford 2001) pitch: 50% rhythm: 40% timbre + dynamics: 10%

melodic confounds (Selfridge-Field 1998): rests repeated notes grace notes, ornamentation Mozart example

Gestalt and melody

melody: coherent succession of pitches from New Harvard Dictionary of Music

coherence important for similarity: creates musical meaning bottom-up (pitches and durations) top-down: segmenting, Gestalt

Gestalt theory of perception late 19th/early 20th century, Germany, later US perception of wholes rather than parts explanations: Gestalt principles of grouping application in visual and musical domain

Low-level Gestalt principles

Snyder mentions: proximity

rhythmic intervallic

similarity duration articulation

continuity melodic

these produce closure of wholes

Example: Beethoven 5th symphony: beginning 1st movement also illustrates high-level

principlesfrom Snyder (2001)

Low-level Gestalt principles

Snyder mentions: proximity

rhythmic intervallic

similarity duration articulation

continuity melodic

these produce closure of wholes

Example: Beethoven also illustrates high-level

principlesfrom Snyder (2001)

High-level Gestalt principles

parallellism very strong in Mozart, Ah

vous, second half of melody

intensification important organisational

principle in variations and improvisations

Mozart’s last variation

from Snyder (2001)

Application in analysis and retrieval Gestalt reduces memory

overload: we can ignore the details

Analytical: Schering (1911) 14th century Italian songs basic melodic shape might be nice for retrieval

Problem with Gestalt principles: many different formulations overlap; no rules for conflict intuitive, cannot be

successfully formalized

from New Grove, Music analysis

The cognitive interpretation: chunking what creates a boundary

interval leap long duration tonality (stable chords) etc

Example of quantification: Melucci & Orio (2004) using local boundary detection (Cambouropoulos 1997)

apply weight to intervals and durations boundary after maximum

chunks forther processed for indexing

Organising chunks

STM problem: max. 5-7 different elements very short span

solution: hierarchical grouping

melody schemas contours of melody

cf. Schering ex. examples: axial, arch, gap-

fill Mozart begins with gap-fill

next level: form A-B-A from Snyder (2001)

mental model of a songAh, vous dirai-je maman melody level

phrase level

chunk level

subchunk level

A ABanalysissy

nthe

sis

analysis: from ear to LTM (sub) chunks created by similarity and

continuity a lot of parallellism

boundaries by leaps and harmony chunks may have a harmonic aspect too

(I, V, V->I)

synthesis: from LTM to focus of attention recollection

using general characteristics of phrases and chunks

performance notes are reconstitued through some musical

grammar

Problems of melody retrieval

People remember high-level concepts, not notes often confused with poor performance abilities theme-intensive music (fugues) stimulate formation of such

concepts melodic variability and change

transposition augmentation/diminution ornamentation variation compositional processes: inversion, retrograde

other factors polyphony harmony

Set-based approaches to melody retrieval in polyphony General idea:

compare note sets: find supersets, calculate distance usually take rhythm and pitch into account hopefully more tolerant agains some of the problems of melodic variety

Clausen, Engelbrecht, Meyer, Schmidt (2000): PROMS matches onset times; wildcards elegant indexing

Lemström, Mäkinen, Ukkonen, Turkia (several articles, 2003-4) C-Brahms algorithms for matching line segments

P1: onsets P2: partial match onset times P3: common shared time

attention to time complexity Typke, Veltkamp, Wiering (2003-2004)

Orpheus system

Earth Mover’s Distance

The Earth Mover’s Distance (EMD) measures similarity by calculating a minimum flow that would match two set of weighted points. One set emits weight, the other one receives weight

Y. Rubner (1998); S. Cohen (1999)

Application to music

represent notes as weighted point sets in 2-dimensional space (pitch, time)

weight represents duration other possibilities

contour/metric position etc

other possible application:pitch event + acoustic feature(s)? here, the ‘earth’ is only moved along the temporal axis

Another example

interesting properties tolerant against

melodic confounds

suitable for polyphony

continuous partial matching

disadvantage triangle inequality

doesn’t hold less suitable for

indexing:

after alignment, the ‘earth’ is moved both along the temporal axis and along the pitch axis

Test on RISM A/II

Matching polyphony with the EMD

EMD’s partial matching property is essential MIDI example used as query for RISM database gross errors in playing are ironed out

Proportional Transportation Distance (PTD) Giannopoulos &

Veltkamp (2002) EMD, weigths of sets

normalised to 1 suitable for indexing

triangle inequality holds

no partial matching

Test on RISM A/II

only hits with approximately same length

need 4 queries to find all known items

False positive (EMD)

problems arise when length and/or number of notes differs considerably

Segmenting

overlapping segments of 6-9 consecutive notes

not musical units search results are combined better Recall-Precision

averages

Example of new searchhttp://teuge.labs.cs.uu.nl/Rntt.cgi/mir/mir.cgi

Concluding remarks about melodic retrieval lots of creativity go into melody; difficult to give rules

not a ‘basic musical structure’ (Temperley 2001) essential to use multiple features

pitch, rhythm harmony

segmentation finding perceptually relevant chunks is not easy finding complete melodies may be harder arbitrary segments may also work

indexing strategies for melody melodic change over time several projects have tentative results for polyphony

gut feeling: false positives are big issue notion of salience (Byrd and Crawford)