Symbolic Melodic Similarity (through Shape Similarity) Julián Urbano

Barcelona, November 12th 2013

Spam

• @julian_urbano

• PhD, Computer Science

– (Evaluation in) (Music) Information Retrieval

• Postdoctoral researcher

– Music Technology Group

– Universitat Pompeu Fabra

2

Outline

• Symbolic Melodic Similarity

• Representing Melodies

• Comparing Melodies

• Melodic Similarity through Shape Similarity

• Results

• Demo

3

SYMBOLIC MELODIC SIMILARITY

Symbolic Melodic Similarity

• Symbolic – No audio signals, just notes

• Melodic – No polyphony

– Usually no harmony

– Usually a single voice

• Similarity – Sound alike

– In terms of…good question!

– Very ill-defined: “whatever criteria you think makes two melodies sound alike”

5

Typical Use Case

• A user has a (large) collection of melodies and a query melody

• An SMS system ranks all melodies in the collection according to their estimated melodic similarity to the query melody

– Implicitly, similarity is considered non-binary

• Two melodies are similar to each other to some degree

• Similarity is not a binary yes-no relation

– Usually just the top-k most similar

6

Desired: Transposition Invariance

• Ignore “irrelevant” differences in pitch

7

Desired: Transposition Invariance

• Ignore “irrelevant” differences in pitch

– Same note, different octave

7

Desired: Transposition Invariance

• Ignore “irrelevant” differences in pitch

– Same note, different octave

– Same degree, different key

7

Desired: Transposition Invariance

• Ignore “irrelevant” differences in pitch

– Same note, different octave

– Same degree, different key

– Same pitch, different key

7

Desired: Time-scale Invariance

• Ignore “irrelevant” differences in rhythm

8

Desired: Time-scale Invariance

• Ignore “irrelevant” differences in rhythm

– Same duration, different time signature

8

Desired: Time-scale Invariance

• Ignore “irrelevant” differences in rhythm

– Same duration, different time signature

– Same duration, different tempo

8

Desired: No Metadata

• Try not to rely on metadata, because it is not always available

– Obvious: artist, genre, etc.

– Less-obvious: key, time signature, tempo, etc.

9

REPRESENTING AND COMPARING MELODIES

(Bad) Representations of Melodies

• Transposition Invariance

– Pitches: from 0 to 127, in semitones

• C4 A4 A#5 = 60 69 82

– Directed interval with tonic, in semitones

• Tonic = C4: D4 A4 E3 = +2 +9 -8

– Ignore octaves

• C4 D4 = 2 , C4 E4 = 4 , C4 E3 = 4

– Parsons Code

• C4 D4 D4 E3 = U R D

11

(Bad) Representations of Melodies

• Time-scale Invariance

– Actual duration, in milliseconds

• 250ms 187ms

– Metric duration

• Crotchet quaver = 1 1/2

12

(Good) Representation of Melodies

• Represent relation between successive notes

– Directed pitch interval

• C4 D4 E4 E3 = +2 +2 -12

– Duration ratio

• 250ms 500ms 190ms 570ms = 2 0.37 3

• Crotchet quaver quaver whole = 0.5 1 4

• Allows us to detect and keep changes locally

+3 +2 +3 -3 -2 +2 +7 -2 -3 -4 +2 -5 +3 +2 +3 -3 -2

1 1 1 1 1 6 2/3 1 1 1 1 1/2 1 1 1 1 1 13

Models of Similarity

• Given two melodies A and B… – Represent them following these criteria

– But how do we compute their similarity?

• Several types of models – Melody as text, classic retrieval

– Melody as text, editing distance

– Melody as graphs or trees, transition similarity

– Melody as n-grams, classic retrieval

– Melody as set of points, geometric similarities

– Melody as orthogonal chains, area between chains

– Melody as curves, alignment with geometric similarities

14

Melody as (sequence of) Text

• Melodies are just a string of text, much like sentences in a book

• Classic retrieval: compare frequencies of symbols

– C4 appears 5% of times, D6 does 0.38% of times, etc.

– Compare on a pitch-by-pitch basis

• Bag-of-words: loses sense of order in music

15

Melody as (sequence of) Text

• Melodies are just a string of text, much like sentences in a book

• Editing distance: how many transformations are needed to go from melody A to melody B?

– C7 E7 A6 C7

– C7 A6 D7 A6 C7

16

Melody as Graphs

• Melodies are a graph representing the probability of transitions from pitch to pitch

– Compare on a transition-by-transition basis

• Sort-of sense of order

17

D7

C7 E4

E7

G3

0.1

0.02

0.08

0.09

0.01

0.1 0.15

D7

C7 E4

E7

G3

0.1

0.1

0.09

0.09

0.01

0.1 0.02

Melody as (sequence of) n-grams

• Overlapping sequences of n successive notes – C7 E7 A6 C7 D7 • n=2 : (C7 E7) (E7 A6) (A6 C7) (C7 D7)

• n=3 : (C7 E7 A6) (E7 A6 C7) (A6 C7 D7)

• n=4 : (C7 E7 A6 C7) (E7 A6 C7 D7)

• Classic retrieval: compare frequencies of n-grams

• Motifs, though n is fixed

• Sort-of sense of order

18

Melody as (set of) Points

• Notes are points in the pitch-time plane, spaced according to the pitch and time representation

– Compare disposition of points (eg. EMD)

19

Melody as Orthogonal Chain

• Represent melodies in a stairway fashion

– Compare area between chains, looking for optimum

20

Alignment of Symbols

• Independent of representation

• Alignment: similar to editing distance, but weight differently insertions, deletions and mismatches

– C7 E7 A6 C7

– C7 A6 D7 A6 C7

(mismatch E7-D7 “costs” less than mismatch A6-E7)

• Problem: how to decide on weights

– Usually just compare numbers (pitch, octave, etc.)

– Some apply music theory 21

MELODIC SIMILARITY THROUGH SHAPE SIMILARITY

Melody as a Curve

• Distribute notes as points in the pitch-time plane, spaced according to their relative representation

• Calculate the curve interpolating those points

23

Intuition Behind

• Two melodies are as similar as their curves are

• First derivative measures how much change there is in pitch at any given point in time

– Nicely models the “change in music”

• Area between derivatives measures how much the two melodies differ in their change of pitch

24

Intuition Behind

• Naturally meets all transposition and time-scale invariance requirements

• Naturally keeps sense of order in music

25

Interpolation

• Many (boring) details regarding type of interpolation

– Lagrange polynomials

– Splines

• Type

• Degree

26

Interpolation

• We chose Uniform B-Splines of various degrees – Changes are kept local

– Easy to compute

– Easy to differentiate (polynomials) • Degrees 2 and 3 seemed to work better

– Computed in a span-by-span basis • Sense of motifs, like n-grams

– Easily adaptable to differences in length

– Easily incorporate several dimensions of music beyond pitch and time

– Easily compute similarity per dimension (partial derivative)

– Many other nice properties in terms of extreme behavior

27

Implementation

• Represent melodies A and B with relative distances between two successive notes

• Compute splines spans

– Same as compute n-grams and then smaller splines

• Align spline spans

– Weights computed from similarity in shape of spans

28

Two Approaches

• Based on geometric properties and characteristics of the collection

– Reward matches depending on average similarity between spline spans in the collection

– Penalize insertions and deletions based on geometric overall change with respect to x-axis

– Penalize mismatches based on geometric similarity of spline spans

• Again, many (not so boring) details…

– Normalization of dimensions and weights

29

Two Approaches

• How common spline spans are

– Reward matches and penalize insertions and removals based on how common the spline spans are

• The more common, the smaller the penalization and the higher the reward

– Penalize mismatches based on geometric similarity of spline spans

• Roughly compare the direction of the curves

• Compute the area between derivatives

• Pitch and time together or separately – Weight time differently, perceived as less important

30

Alignment

• Local: don’t penalize changes at the beginning and at the end of melodies

• Global: penalize changes everywhere

• Hybrid: penalize at the beginning, but don’t penalize at the end

– Listeners pay attention to the beginning the most

31

DOES ALL OF THIS EVEN WORK?

32

Evaluation of Systems

• (Kind of) yearly at MIREX

– Collection with >2000 melodies

– Select query melodies

– Run systems

– Humans listen to the retrieved melodies and judge how similar they are to the queries

– Systems are scored accordingly

• From 0 to 1 (perfect)

• And once again, many (not at all boring!) details… 33

2005

• Our algorithms didn’t participate, but anyway…

34

2005

• Our algorithms didn’t participate, but anyway…

34

best participant

2010

35

2011

36

2012

37

2013

38

Briefly…

• Alignment works better – Hybrid alignment works much better

• Geometric representations work better

• Incorporating rhythm in the comparison does not improve much, and it’s computationally more expensive – Slightly improves ranking though

• Compare curves very roughly, without computing areas between them – Additionally, it is computationally more efficient

39

HANDS-ON DEMO

MelodyShape

• A Java library and tool implementing all the algorithms submitted to MIREX https://github.com/julian-urbano/melodyshape

41

MelodyShape

• A Java library and tool implementing all the algorithms submitted to MIREX https://github.com/julian-urbano/melodyshape

41

MelodyShape

• A Java library and tool implementing all the algorithms submitted to MIREX https://github.com/julian-urbano/melodyshape

41

Two Libraries Needed

42

. . .

here

or here

To Run

• Just download everything to the same folder – melodyshape-1.1.jar

– commons-math3-3.2.jar

– commons-cli-1.2.jar

• And double click – melodyshape-1.1.jar

• Can also run from the command line

• All your melodies must be in MIDI format

43

References • Aloupis, G., Fevens, T., Langerman, S., Matsui, T., Mesa, A., Nuñez, Y., Rappaport, D., Toussaint, G.: Algorithms for Computing Geometric Measures of Melodic Similarity. Computer Music Journal

30(3), 67–76 (2006)

• Bainbridge, D., Dewsnip, M., Witten, I.H.: Searching Digital Music Libraries. Information Processing and Management 41(1), 41–56 (2005)

• de Boor, C.: A Practical guide to Splines. Springer, Heidelberg (2001)

• Byrd, D., Crawford, T.: Problems of Music Information Retrieval in the Real World. Information Processing and Management 38(2), 249–272 (2002)

• Casey, M.A., Veltkamp, R.C., Goto, M., Leman, M., Rhodes, C., Slaney, M.: Content-Based Music Information Retrieval: Current Directions and Future Challenges. Proceedings of the IEEE 96(4), 668–695 (2008)

• Clifford, R., Christodoulakis, M., Crawford, T., Meredith, D., Wiggins, G.: A Fast, Randomised, Maximal Subset Matching Algorithm for Document-Level Music Retrieval.In: International Conference on Music Information Retrieval, pp. 150–155 (2006)

• Doraisamy, S., Rüger, S.: Robust Polyphonic Music Retrieval with N-grams. Journal of Intelligent Systems 21(1), 53–70 (2003)

• Downie, J.S.: The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future. Computer Music Journal 28(2), 12–23 (2004)

• Downie, J. S., Ehmann, A. F., Bay, M., & Jones, M. C.: The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. In W. R. Zbigniew & A. A. Wieczorkowska (Eds.), Advances in Music Information Retrieval, pp. 93–115 (2010)

• Hanna, P., Ferraro, P., Robine, M.: On Optimizing the Editing Algorithms for Evaluating Similarity Between Monophonic Musical Sequences. Journal of New Music Research 36(4), 267–279 (2007)

• Hanna, P., Robine, M., Ferraro, P., Allali, J.: Improvements of Alignment Algorithms for Polyphonic Music Retrieval. In: International Symposium on Computer Music Modeling and Retrieval, pp. 244–251 (2008)

• Isaacson, E.U.: Music IR for Music Theory. In: The MIR/MDL Evaluation Project White paper Collection, 2nd edn., pp. 23–26 (2002)

• Kilian, J., Hoos, H.H.: Voice Separation — A Local Optimisation Approach. In: International Symposium on Music Information Retrieval, pp. 39–46 (2002)

• Lin, H.-J., Wu, H.-H.: Efficient Geometric Measure of Music Similarity. Information Processing Letters 109(2), 116–120 (2008)

• McAdams, S., Bregman, A.S.: Hearing Musical Streams. In: Roads, C., Strawn, J. (eds.) Foundations of Computer Music, pp. 658–598. The MIT Press, Cambridge (1985)

• Mongeau, M., Sankoff, D.: Comparison of Musical Sequences. Computers and the Humanities 24(3), 161–175 (1990)

• Rizo, D.: Symbolic Music Comparison with Tree Data Structures. Department of Computer Science, University of Alicante (2010).

• Selfridge-Field, E.: Conceptual and Representational Issues in Melodic Comparison. Computing in Musicology 11, 3–64 (1998)

• Smith, L.A., McNab, R.J., Witten, I.H.: Sequence-Based Melodic Comparison: A Dynamic Programming Approach. Computing in Musicology 11, 101–117 (1998)

• Smith, T.F., Waterman, M.S.: Identification of Common Molecular Subsequences. Journal of Molecular Biology 147(1), 195–197 (1981)

• Typke, R., Veltkamp, R.C., Wiering, F.: Searching Notated Polyphonic Music Using Transportation Distances. In: ACM International Conference on Multimedia, pp. 128–135 (2004)

• Typke, R., Wiering, F., Veltkamp, R.C.: A Survey of Music Information Retrieval Systems. In: International Conference on Music Information Retrieval, pp. 153–160 (2005)

• Uitdenbogerd, A., Zobel, J.: Melodic Matching Techniques for Large Music Databases. In: ACM International Conference on Multimedia, pp. 57–66 (1999)

• Ukkonen, E., Lemström, K., Mäkinen, V.: Geometric Algorithms for Transposition Invariant Content-Based Music Retrieval. In: International Conference on Music Information Retrieval, pp. 193–199 (2003)

• Urbano, J.: MIREX 2013 Symbolic Melodic Similarity: A Geometric Model supported with Hybrid Sequence Alignment. Music Information Retrieval Evaluation eXchange (2013)

• Urbano, J., Lloréns, J., Morato, J., & Sánchez-Cuadrado, S.: Melodic Similarity through Shape Similarity. In S. Ystad, M. Aramaki, R. Kronland-Martinet, & K. Jensen (Eds.), Exploring Music Contents, pp. 338–355 (2013)

• Urbano, J., Schedl, M. and Serra, X.: Evaluation in Music Information Retrieval. Journal of Intelligent Information Systems 41(3), 345-369 (2013)

• Ó Maidín, D.: A Geometrical Algorithm for Melodic Difference. Computing in Musicology 11, 65–72 (1998)

44