1 Content-based Music Information Retrieval Student: DENG Jie. Supervisor: Prof. LEUNG, Clement...
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of 1 Content-based Music Information Retrieval Student: DENG Jie. Supervisor: Prof. LEUNG, Clement...
1
Content-based Content-based Music Information Music Information
RetrievalRetrieval
Student: DENG Jie. Supervisor: Prof. LEUNG, Clement
Department of Computer ScienceHong Kong Baptist University
March, 2010
2
Outline Outline
What will be Covered: Introduction A Brief Review of Music MIR in the Real World and Challenges Current Content-based MIR Key
Techniques Evaluation of MIR Conclusion and Future Work
3
IntroductionIntroduction
Music Information Retrieval (MIR) is the interdisciplinary science of retrieving information from music.
– Mainly based on three-filed subjects: traditional information retrieval, musicology and digital audio.
Content-based MIR is the science of extracting features from musical content, such as melody, rhythm and tempo and so on to facilitate tasks such as analysis and music retrieval.
Aim:– To better understand “music” in the music work– To really search music by “music”
4
Music Channels
Chart Shows
Record store
Mate’s recommendation
Motivation - Music Motivation - Music DiscoveryDiscovery
+Gigs
Radio
5
Music IR Application Music IR Application Digital Music Libraries / Sound Archives
– Seeking for content-based access music libraries– Combined with metadata search of existing
catalogues Music Education
– Voice or instrumental teaching Music Related Legal and Copyright
– Is the creative content of this music work based on something for which others hold the rights?
Musicology– Is this piece of music work similar to any other
works?– Dose any part of this piece closely resemble any
part of any other works?– Is this piece of music work is based on others?
6
A Brief Review of MusicA Brief Review of Music
Music Concepts Three basic features of a musical sound
– Pitch– Intensity / Dynamics– Timbre / Tone color
There are many other terms describing music– Tempo– Tonality– Time Signature– Key Signature
7
A Brief Review of MusicA Brief Review of Music
Music Characteristics Music can be defined as the art of disposing and producing sounds and silences in time
– Has horizontal and vertical dimensions The main dimensions of music can be used for music retrieval (Reference Nicola Orio)
– Timbre : Quality of the produced sound– Orchestration : Sources of sound production– Acoustics : Quality of the recorded sound– Rhythm : Patterns of sound onsets– Melody : Sequence of notes– Harmony : Sequence of chords– Structure : Organization of the musical work
8
A Brief Review of MusicA Brief Review of Music Music Representation
– Visual (musical scores, manuscripts)– Aural (digital music)– Text– Hybrid (visual representation of an audio music file )
__________________________________________________________________________________
E---------------0---------3-3------3--1-1-------------3-1-1---------------- B---1-------------3-2-----2-2------2--3-3-----3-2---2---3-3------------1--- G---0--0--0h1-------0-----0-0------0--2-2---2---0-0-----2-2------------2--- D---2--2------------------------------0---0-------------0----0-1-2-3-3---3- A-3-------------------0-0-----0--0--------------0-------------------------- E-------------0------------------------------------------------------------
Common Music Notatio
n
Tablature
Example: Visual Representation
9
MIR in the Real World MIR in the Real World
There are mainly three-category USERS in MIR
“Professional”“Amateur” “Academic”
Just about anyone!
Librarians
Publishers
Producers
Performers
Composers
Lawyers ...
Vast numbers Very many
Musicologists
Educators
Significant numbers
10
MIR in the Real World MIR in the Real World Common Music Data and Format Audio recordings
– Sampled sound– Wave, MP3, AAC, etc.
Symbolic recordings– Abstract musical instructions, MusicXML– Scores, MIDI, Humdrum, etc.
11
MIR in the Real World MIR in the Real World Overview of some existing music search systems
Search by music related metadata: (artists, albums, tracks, music reviews, new release, etc.) Yahoo! Music and Allmusic are the examples of this search type
Search by music lyrics: Lyrics.com and SongLyrics.com
Music Media Management and Track Identification: Identify metadata for music tracks, for example Gracenote and MusicIP
Recommend similar music: by mining some music feature elements (melody, rhythm, tone color, etc) to recommend user some similar music
Recommend personalized music: by mining some users’ information to recommend them some their favorite music
12
Challenges in MIRChallenges in MIR Began in the 1950’s, still an emerging discipline Subjectivity and Versioning Many levels of music knowledge Lack of bibliographic control and data quality________________________________________________________________________
Signal Processing Machine Learning Human Computer Interaction
HearingRepresentation
UnderstandingAnalysis
ReactingInteraction
MIR Pipeline
13
A Simplified MIR MapA Simplified MIR Map
integration of audio visual, symbolic and textual data
This very schematic diagram highlights trends
Extracted or produced information
Actions
External data
14
Basic Steps of Content-based MIRBasic Steps of Content-based MIR
Representation of music contents– Features: melody, rhythms, etc.
Feature extraction from music data Feature indexing Query interface Matching query features against the
feature index
15
Representation of Music Content – Most music features used to represent
music are always melody.– Rhythm feature only consider the rhythm
omitting the melody.– Melody contour method uses three
characters to express the contour of melody.
Content-based MIRContent-based MIR
16
Content-based MIRContent-based MIR
Feature extraction from music data– There are two category algorithms: time
domain (Autocorrelation function, Average magnitude difference function and Simple inverse filter tracking) and frequency domain models (Spectrum and Cepstrum)
– Common extracted tools in the following:• Short-term Fourier Transform features (FFT)• Mel-Frequency Cepstral Coefficients (MFCC)• Daubechies Wavelet Coefficient Histogram (DWCH)
– Pitch is the main feature extracted in practice
17
Content-based MIRContent-based MIR
Feature Indexing – Index terms: play a similar role of words in
textual documents.– Sequence matching techniques: consider both
the query and the documents as sequences of symbols and model the possible difference between them.
– Geometric methods: cope with polyphonic scores and also exploit the properties of continuous distance measures.
– Based on the above methods, there are mainly three category music search on the melody feature.
• Melodic retrieval based on index terms (N-grams)• Melodic retrieval based on sequence matching • Melodic retrieval based on geometric methods
18
Content-based MIRContent-based MIR
Query Interface – Query by text (keywords: album, artist, track,
etc.)– Query by aural (singing or humming)
• Wave input (sing the whole or part of the songs)• Music notes segmentation• Thematic melodies are extracted, translated into
text representations of intervals, pith, and harmony
• Comparison procedure– Query by tapping
• Wave input by tapping• Compute the duration of each note• Similarity comparison
19
Content-based MIRContent-based MIR Matching Query Features against the
Feature Index – Approximate/Partial matching– Similarity measure (MFCC, GMM, KNN)– Precision: how many of the answers are in fact correct– Recall: how many of the correct answers are in fact retrieved– Relevance feedback
Vector space model– Documents and queries are presented by vectors– Each element in a vector is determined by an indexing
scheme (N-grams or others)– The value of each element is determined by a weight scheme– The similarity between document di and query qj :
20
Evaluation of MIREvaluation of MIR
The community has established an array of software tools to support this work – see http://music-ir.org/evaluation
In 2004, Audio Description Contest first attempted to build comparative benchmark of MIR algorithms.
Downie has already given us the foundations and future of the scientific evaluation of MIR systems.
Traditional information retrieval evaluation can also be adopted in MIR, for example precision and recall measures.
21
ConclusionConclusion Music is a complicated art form of information and requires special retrieval systems MIR technology is improving, but the real application is still lacking Basic music concepts and characteristics Basic steps and models of MIR Current Content-based MIR Key Techniques Scientific evaluation of MIR___________________________________________________________
Mining the semantic information in multimedia, especially in digital audio music, and then propose a comprehensive and adaptive method to automatically analysis and retrieve the high level semantic information of music, for example, emotion, mood, and style, etc.
Future WorkFuture Work
22
Related Related Researches and ProjectsResearches and Projects
ISMIR since 2000– International Symposium on Music Information
Retrieval WOCMAT since 2005
– Workshop On Computer Music and Audio Technology
Digital archive application– Data mining in digital music archive
Free music audio, sound processing tools and music-related visualization and mining tools
– http://www.music-ir.org/evaluation/tools.html Music IR evaluation since 2005
– http://www.music-ir.org/mirexwiki/index.php/Main_Page– Test collection: music documents, query sets, and
judgment– Major handle: copyright issue
23
ReferenceReference
• Michael S. LEW, Nicu Sebe. Content-Based Music Information Retrieval: Current Directions and Feature Challenges. Proceedings of the IEEE, April 2008
• Nicola Orio. Music Retrieval: A Tutorial and Review. Foundations and Trends in Information Retrieval, Volume1, Issue 1, Pages 1-96, 2006.
• J. T. Foote, "An Overview of Audio Information Retrieval." In ACM-Springer Multimedia Systems, vol. 7 no. 1, pp. 2-11, ACM Press/Springer-Verlag, January 1999
• Remco C. Veltkamp, Frans Wiering, Rainer Typke. Content Based Music Retrieval. In B. Furht (Ed.), Encyclopedia of Multimedia. Springer, 2006.
• Giovanna Neve, Nicola Orio: A Comparison of Melodic Segmentation Techniques for Music Information Retrieval. ECDL 2005: 49-56.
• Hwei-Jen Lin, Hung-Hsuan Wu. Efficient geometric measure of music similarity. Information Processing Letters, Volume 109, Issue2, Page 116-120, 2008.
• Iman S. H. Suyoto, Alexandra L. Uitdenbogerd, and Falk Scholer. Searching Musical Audio Using Symbolic Queries.IEEE. Transactions onAudio, Speech and Language Processing, 16(2):372–381, 2008.
• The Scientific Evaluation of Music Information. Retrieval Systems: Foundations and Future. Computer Music Journal, Computer Music Journal, 28:2, pp. 12–23, Summer 2004.
• Michael Fingerhut. Real music libraries in the virtual future: for an integrated view of music and music information. Digitale bibliotheken voor muziek, 2005.
24
Thank you!
Questions?