1

Other Topics in Music Organization/Representation

Donald Byrd

School of Informatics

Indiana University

Updated 29 April 2006

Copyright © 2003-06, Donald Byrd

30 Jan. 06 2

Classification: Surgeon General’s Warning

• Classification (ordinary hierarchic) is dangerous– Almost everything in the real world is messy– Absolute correlations between characteristics are rare– Example: some mammals lay eggs; some are “naked”– Example: musical instruments (piano as percussion,

etc.)• Nearly always, all you can say is “an X has

characteristic A, and usually also B, C, D…”• Leads to:

– People who know better claiming absolute correlations– Arguments among experts over which characteristic is

most fundamental– Don changing his mind

4 April 06 3

Comparison of Music-IR Task Classifications

• Typke, Rainer, Wiering, Frans, & Veltkamp, Remco C. (2005). A Survey of Music Information Retrieval Systems– Overview of 17 systems for content-based retrieval of

music in both audio & symbolic forms– Includes “map” of systems showing tasks & users for

which each is most appropriate– Horizontal axis (tasks) has similar idea to my Similarity

Scale for Content-Based Music IR– Main difference: Typke et al have “artist”– Doesn’t fit hierarchy, but useful--and dangerous!

5 April 06 4

Music Recommender Systems (1)

• Guest speaker: Justin Donaldson– PhD student, IU Computer Science– Intern, MusicStrands

• Pandora’s approach– Classification by experts with controlled vocabulary– “Music genome”

• MusicStrands approach– Co-occurrence, network analysis, with limited guidance by expert

• Examples: FOAFing the Music, Last.fm, MusicIP Mixer, Musicmatch Jukebox, MusicStrands, Pandora

5 April 06 5

Music Recommender Systems (2)

• All(?) existing systems try to find music similar to what you give them

• Instead, do the opposite– Tim Crawford to Don (2004): I don't want to find more music like

what I already know, I want music as different as possible from it!

• Jeremy Pickens' example: Eigenradio– http://eigenradio.media.mit.edu/christmas_2004.html

– NOT a good example!

• How to automate? Genre classification?

7 April 06 6

Music Recommender Systems (3)

• Systems: FOAFing the Music, Last.fm, MusicIP Mixer, Musicmatch Jukebox, MusicStrands, Pandora

• Pandora’s “music genome” idea– Assumes all music based on small number of “genes”– Content-based– Requires annotation by human experts

• Last.fm– Conventional collaborative approach(?)

• Others?

10 April 06 7

Maps, Visualizations, & Metrics (1)

• Example 1: Justin Donaldson’s 3-D visualization• Map requires metric (similarity measure) => positions in

n-dimensional space• n can be huge, except for visualization• Example 2: Pampalk et al, “Exploring Music Collections

by Browsing Different Views” (ISMIR 2003, CMJ 2004)– Organization by spectrum, periodicity, metadata– Uses self-organizing maps (SOMs)– SOMs can focus on audio analysis and/or metadata– Maps of same collection aligned => can move from one view to

another

10 April 06 8

Maps, Visualizations, & Metrics (2)

• With good similarity measure, easy to find similar stuff or different stuff!– Automatically (searching, filtering)– Do-it-yourself (browsing)

• What’s a good similarity measure for music?• Usual interpretation: what are good features to use?• Cf. Eigenradio: has objective features, but need subjective• Cf. Pampalk comment on main difficulty

10 April 06 9

Digital Music Libraries

• iTunes: no. 1 commercial system– Popular & simple but not always easy to find music

with– Not a real “music library”: does very little

• Variations2: research project => production system

13 April 06 10

What is a Digital Library?

• Not just library with computers & on-line catalog!• DL as collection/information system

• “collection of information that is both digitized and organized” -- Michael Lesk, NSF

• “networked collections of digital text, documents, images, sounds, scientific data, and software” -- PITAC report

• DL as organization• “organization that provides resources to select, structure, offer

intellectual access to, interpret, distribute, preserve integrity of, and ensure persistence over time of collections of digital works...”-- Digital Library Federation

• “Elephant in the Room” for all DLs: persistence over time = preservation

13 April 06 11

What is a Digital Music Library?

• Music has many special needs• Content formats

– Need audio, scores; want video, maybe MIDI, etc.

• Search capabilities– for content and metadata

• Intellectual Property Rights (IPR) => access control important

• Traditional library catalogs don’t handle music well– One reason: lack of music-specific metadata

13 April 06 12

Variations and Variations2

• Digital library of music sound recordings & scores• Original concept 1990, online since 1996• Variations2 started as pure research project• Now production system; replaced Variations in

2005– Accessible by all in Music Library; other locations

restricted for IPR reasons– Used daily by large student population– Currently: 11,500 titles, 15,000 hours of audio

• Over 6 TB uncompressed, 1.6 TB compressed (MP3, AAC)

– Opera, songs, instrumental music, jazz, rock, world music, etc.

18 April; rev. 25 April 13

Some Metadata and Digital Library Buzzwords

• MARC: metadata standard for library catalogs– From the Library of Congress– Old (1970’s): fixed format, etc.; “bibliographic”– Standard for maintaining & exchanging bibliographic information– Simple relationships, elaborate details

• Dublin Core (DC): general-purpose metadata standard– From Dublin Core Metadata Initiative (DCMI)– New (1990’s): XML, etc.; “metadata”– Simple, general, extensible– Terminology: http://dublincore.org/documents/dcmi-terms/

• Open Archives Initiative (OAI): metadata consumer• FRBR: metadata standard for library catalogs

– From IFLA, with support from Library of Congress, etc.– New (>2000)– Complex relationships, elaborate details

25 April 14

Functional Requirements for Bibliographic Records (FRBR)

• Represents much more complex relationships than MARC– MARC records refer explicitly to subject headings (LCSH), URLs– …and implicitly (via uniform names & titles) to other MARC

records– …but not consistently! – FRBR (like Variations2) records always refer to each other

• FRBR Entities– Group 1: Products of intellectual & artistic endeavor– Group 2: Those responsible for the intellectual & artistic content– Group 3: Subjects of works

• Much of following from by Barbara Tillett (2002), “The FRBR Model (Functional Requirements for Bibliographic Records)”

25 April 15

FRBR Entities

• Group 1: Products of intellectual & artistic endeavor1. Work (completely abstract)2. Expression3. Manifestation4. Item (completely concrete: you can touch one)– Almost heirarchic; “almost” since works can include other works

• Group 2: Those responsible for the intellectual & artistic content– Person– Corporate body

• Group 3: Subjects of works– Groups 1 & 2 plus– Concept– Object– Event– Place

25 April 16

Relationships of Group 1 Entities: Example

w1 J.S. Bach’s Goldberg Variations

e1 Performance by Glenn Gould in 1981

m1 Recording released on 33-1/3 rpm sound disc in 1982 by CBS Recordsi1a, 1b, 1c Copies of that 33-1/3 rpm disc acquired in 1984-87 by the Cook Music Library

m2 Recording re-released on compact disc in 1993 by Sonyi2a, i2b Copies of that compact disc acquired in 1996 by the Cook Music Library

m3 Digitization of the Sony re-released as MP3 in 2000

25 April 17

Relationships of Group 1 Entities (1)

25 April 18

Work

Expression

Manifestation

Item

is realized through

is embodied in

is exemplified by

Relationships of Group 1 Entities (2)

recursive

one

many

25 April; rev. 26 April 19

FRBR vs. Variations2 Data Models

FRBR• Group 1

– Work– Expression– Manifestation– Item

• Group 2– Person– Corporate body

(any named organization?)

Variations2

– Work

– Instantiation– Media object

– Contributor (person or organization)

– Container

Items in blue are close, though not exact, equivalents.

13 April 06 20

Elephant in the Room for Music DLs: Getting Catalog Information into FRBR or Variations2

• 2005 MLA discussion– Cataloging to current standards (MARC) is very expensive– FRBR and Variations2 both much more demanding– Michael Lesk/NSF: didn’t like funding metadata projects because

“they always said every other project should be more expensive”!– Libraries seem to be moving to FRBR anyway

• Idea 1: collaborative cataloging (ala OCLC)• Idea 2: take advantage of existing cataloging

– Variations2: simple “Import MARC” feature– VTLS: convert MARC => FRBR is much more ambitious

• Good ideas, but probably not enough• Idea 3: user-contributed metadata?

10 April 06 21

Variations2 Hands-on

• Possibilities– Search using Variations2 search window– Search using IUCAT– External (WWW or other) access via reserve lists, etc.– Create playlist– Add bookmarks– Create listening drill from playlist– Export to make a Web page– Use Opus window– Use Timeliner

10 April 06 22

Variations2 Data Types

• Work is realized through• Instantiation (recording or score) is embodied in• Container (CD, LP, edition of scores, etc.)

• Contributor (person or organization)– Contributor to work: composer, lyricist, etc.– Contributor to instantiation: performer, conductor,

engineer, etc.

12 April 06 23

Works & Work Relationships

• Work concept is new to Variations2/FRBR• Much more important to organize music than

(e.g.) books– Language of title says very little about content– Important relationships: song & album, aria & opera,

etc.

• Work relationships can be very complex– Part/whole– Arrangement– Version (improvisation, etc.)

10 April 06 24

Style Genres & Genre Classifications

• Genre Classifications are a mess• No consistency between classifications

– All-Music Guide: <=4 levels: 2 top-level (pop/classical), 34 second-level

– Amazon.com: ca. 23 main genres– GarageBand.com: 47 genres, flat– ID3 tags (in MP3's): 80 genres, flat; WinAmp version: 126 genres,

flat– Ishkur's EM Guide ("Electronic music" only): <=3 levels: 7 top-

level– iTunes: 37 genres, flat– MIREX 2005 : 9, 38 leaves

13 April 06 25

Style Genres & Genre Classifications

• No wonder: what makes a musical style is very subtle!• In many cases, "correct" genre can't be determined without

knowledge of the lyrics, even understanding• …or even intent of creators• Dave Datta (2005): automatic genre-classification

programs are finding something, & probably useful, but may not be genres as people understand them

• Turntablism is a separate genre--or is it? If it's done "mildly", what you'd hear is mostly the genre of the underlying music!

14 April 06 26

Transcription of Polyphonic Audio

• Cf. OMRAS experiments: an important research problem• Why is it so difficult?• Guest speaker: Ian Knopke, IU fellow in music informatics

14 April 06 27

Music Plus One (1)

• Chris Raphael’s accompaniment system• Goals of ``Music Plus One'' are similar to ``Music Minus

One'' (MMO)• But, with MMO, soloist must follow accompaniment• Goals: program must:

– Respond in real time to soloist's tempo changes & expression– Learn from past performances so it assimilates soloist's

interpretation in future– Bring sense of musicality to performance– Components: Listen and Play

14 April 06 28

Music Plus One (2)

• Listen– As soloist plays, signal analyzed to determine what notes have

played & exactly when– Greatly simplified because computer “knows” score (MIDI file)– But must be robust to inaccuracies & embellishments by soloist

while maintaining accuracy in matching signal to soloist's part– Uses Hidden Markov Model (HMM)

• Play: fuse knowledge sources– Output of Listen– Score (notes, rhythms, etc.)– To improve over successive rehearsals, use collection of past solo

performances by soloist– Performances of the accompaniment by a live player

20 April 06 29

Accompaniment System “Variations”

• Music written for system (beyond human capabilities)

• Erase the soloist => greatly enlarge repertoire

• Detecting beat in audio (alignment with score)...

• Much easier than detecting without the score (e.g., Digital Performer)

• For use by Variations2?– Example: Sacrificial Dance of Le Sacre du Printemps

17 April 06 30

“Music as Different as Possible”• Results are… interesting

– Two teams used Cage’s 4’ 33”– All lists good, none great

• Not much world music!– Team A: Ives’ The Unanswered Question– Team B: Prefuse 73’s B2 Living Life– Team C: Japanese gagaku Keibairaku No Kyu (Taishiki-Cho)– Team D: Balinese monkey chants

• Electronic/computer music can be much more extreme– Xenakis’s Bohor: few definite-pitched sounds– Dodge’s In Celebration: wild synthesized “voice”

• “You sit in a chair, touched by nothing, feeling the old self…”

• Consider language + basic parameters of sound– Pitch, duration/rhythm, dynamics, timbre

16 April 06 31

Expectation and Perception with Sponges, Dinosaurs, and Music

• Sponges– Contamination of kitchen surfaces before & after

cleaning with sponge surprised researchers

• Dinosaurs– 1922 audience fooled by test reel for The Lost World

• Music– Don’s experience with Kurzweil flute sound– Hammond organ Model A compared to pipe organ (ca.

1940)• In “blind” test, experts & students couldn’t tell them apart

17 April 06 32

Science, Scholarship, and Critical Thinking

• Good research is very hard– Electronic Musician article on analog summing

• The issue isn’t just science…– D. Huron on what he learned about music scholarship

• It isn’t just scholarship…– 1922 audience fooled by test reel for The Lost World

• It’s critical thinking– ALWAYS evaluate information sources– ALWAYS consider biases, including your own

• Darwin’s attitude about his biases

• “Most people would rather die than think…”

26 April 06 33

Intellectual Property Rights (IPR) (1)

• IPR is huge problem for music IT, including IR, both research & use– No one knows the answers! Different in different countries!

• For music, U.S. copyright is complex “bundle of rights”– mechanical right: use in commercial recordings, ROMs, online

delivery for private use– synchronization right: use in audio/visual works (movies, TV, etc.)– More complex than for text works because performing art

• U.S. Constitution: balance rights of creators and public– “To achieve these conflicting goals and serve the public interest

requires a delicate balance between the exclusive rights of authors and the long-term needs of a knowledgeable society.” —Mary Levering, U.S. Copyright Office

– After some time, work enters Public Domain

26 April 06 34

Intellectual Property Rights (IPR) (2)

• Law supposed to balance rights of creators & public, but…– Time till Public Domain getting longer & longer– “Joke”: When will old Disney movies be Public Domain?– Sonny Bono Copyright Extension Act: not till 70 years after death!– Digital Millenium Copyright Act (DMCA) restricts owner’s rights– Rep. Smith’s bill (in Congress soon?) even worse

• “Fair Use”: U.S. limit on exclusive rights of copyright owners– Traditionally used for excerpts for reviews, etc.– Not well-defined. Four tests:– 1. Purpose and character of use, including if commercial or nonprofit– 2. Nature of copyrighted work– 3. Amount and substantiality of portion used relative to work as a whole– 4. Effect of use on potential market for or value of copyrighted work

• Law also has educational exemptions

26 March, rev. 15 April 35

Intellectual Property Rights (IPR) (3)

• NB: I’m not a lawyer!• IPR in practice

– Mp3.com sued & shut down– Peer-to-Peer networks: Napster, Gnutella, FreeNet– Church choir director arranged work, did free performance;

donated to publisher => sued• Example: Student wants to quote brief excerpts from

Beethoven piano sonatas in class paper, in notation• Do they need permission from owner?

– Beethoven dead for more than 70 years => in Public Domain– …but not all editions– Still, don’t need permission: Fair Use applies– For recording, probably not P.D., but Fair Use applies

14 April 06 36

Music IR as Music Understanding

• Dannenberg (ISMIR 2001 invited paper) argued central problem of music IR is music understanding

• …also basis for much of computer music (composition & sound synthesis) and music perception and cognition– “A key problem in many fields is the understanding and

application of human musical thought and processing”

• Don: No understanding yet => sidestep intractable problems!

• Cf. “how people find information” vs. “how computers find information”

28 April 06 37

Detecting Beats/Tempo in Audio without a score (1)

• Related tasks: tempo detection & beat detection/slicing

• What can you do with them?– Create loops– Change tempo radically with no artifacts– Ask Will Pierce

• State of the art in commercial products– Digital Performer Beat Detection Engine™– “Employing sophisticated transient detection

technology…”• Likely to work only with very simple texture

28 April 06 38

Detecting Beats/Tempo in Audio without a score (2)

• What else can you do? More advanced stuff:– Change swing feel to straight 8ths (Digital Performer)

• State of the art in research systems– MIREX 2005 Audio Tempo Extraction contest

• www.music-ir.org/mirex2005/index.php/Audio_Tempo_Extraction

– Looking for notated & perceived tempo

– …and phase (= upbeat)

– Music w/stable tempo, wide variety of styles, many non-Western

– Texture?

• With beat slicing and audio similarity:– Violate IPR laws (Scrambled Hackz!)

• Interview: www.wired.com/news/columns/0,70664-0.html

• Video: www.popmodernism.org/scrambledhackz/?c=4

27 April 06 39

Intellectual Property Rights in The Real World

• NB: I’m not a lawyer!• A common way for people to decide what’s OK

– Consider ethics: any problem?– Consider practical effects: any problem?– If no and no, go ahead

• Example: Member of this class wants to share copyrighted music with others in the class– Ethics: it depends– Practical effects: ordinarily none

• Thorny issue: at what point is sampling a problem?• Deeper, thornier issue: does IPR make sense?

– Joey Morwick: maybe not– Ian Clarke (Freenet), Sven Koenig (Scrambled Hackz), promoters

of XOR circumvention: absolutely not!

28 April 06 40

Conclusion; Thank You

• Please, please think for yourself– ALWAYS evaluate information sources– ALWAYS consider biases, including your own

• Schoenberg: “This book I have learned from my students”

• Don Byrd: “This course I have learned from my students”– I’ve learned a lot about music and

technology• I’m available!