Representation and Segmentation of Melodies in Indian Art …...S. Gulati, J. Serrà, K. K. Ganguli,...
Transcript of Representation and Segmentation of Melodies in Indian Art …...S. Gulati, J. Serrà, K. K. Ganguli,...
Representation and Segmentation of Melodies in Indian Art Music
3rd CompMusic Workshop, IIT-Madras, Chennai, India 13th Dec, 2013
Sankalp Gulati
Supervisor: Prof. Xavier Serra PhD Candidate, MTG-UPF, Barcelona, Spain
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology for melodic motif discovery
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology for melodic motif discovery
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology for melodic motif discovery
Tonic Identification: JNMR article
Sengupta, R., Dey, N., Nag, D., DaĶa, A. K., & Mukerjee, A. (2005). Automatic tonic ( SA ) detection algorithm in Indian classical vocal music. In National Symposium on Acoustics (pp. 1–5).
Ranjani, H. G., Arthi, S., & Sreenivas, T. V. (2011). Carnatic music analysis: Shadja, swara identification and raga verification in Alapana using stochastic models. Applications of Signal Processing to Audio and Acoustics (WASPAA), IEEE Workshop , 29–32.
Salamon, J., Gulati, S., & Serra, X. (2012). A multipitch approach to tonic identification in Indian classical music. In Proc. of Int. Conf. on Music Information Retrieval (ISMIR) (pp. 499–504). Gulati, S., Salamon, J., & Serra, X. (2012). A two-stage approach for tonic identification in Indian art music. In 2nd CompMusic Workshop (pp. 119–127).
Bellur, A., Ishwar, V., Serra, X., & Murthy, H. (2012). A knowledge based signal processing approach to tonic identification in Indian classical music. In 2nd CompMusic Workshop (pp. 113–118).
Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).
Tonic Identification: Datasets
Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).
Tonic Identification: Results
Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).
Tonic Identification: Analysis
Performance as a function of length Performance as a function of category
Error type as a function of dataset
Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology
Melody Extraction and Representation
q Melody (lead artist/voice) § Pitch (Fundamental Frequency, F0) § Timbre § Loudness
Salamon, Justin, and Emilia Gómez. "Melody extraction from polyphonic music signals using pitch contour characteristics." Audio, Speech, and Language Processing, IEEE Transactions on 20.6 (2012): 1759-1770.
Rao, Vishweshwara, and Preeti Rao. "Vocal melody extraction in the presence of pitched accompaniment in polyphonic music." Audio, Speech, and Language Processing, IEEE Transactions on 18.8 (2010): 2145-2154.
De Cheveigné, A., & Kawahara, H. (2002). YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111, 1917.
Original Audio
Predominant Voice
F0 of the Predominant Voice
Loudness and Timbral Facets
Loudness and Timbre
q Zwicker, E. (1977). Procedure for calculating loudness of temporally variable sounds. The Journal of the Acoustical Society of America, 62(3), 675–682.
q Röbel, A., & Rodet, X. (2005). Efficient spectral envelope estimation and its application to pitch shifting and envelope preservation. In Proc. dafx.
Predominant F0 Frequency estimation
Synthesize predominant melodic source
Loudness feature extraction
Timbre feature extraction
Audio
Evaluation: Essentia Pitch Extraction q Predominant melody extraction (F0)
§ 6 Hindustani music pieces ~45 mins
Bogdanov, D., Wack, N., Gómez, E., Gulati, S., Herrera, P., Mayor, O., … Serra, X. (2013). Essentia: an audio analysis library for music information retrieval. In Proc. of int. society for music information retrieval conf. (ISMIR) (pp. 493–498).
Salamon, Justin, and Emilia Gómez. "Melody extraction from polyphonic music signals using pitch contour characteristics." Audio, Speech, and Language Processing, IEEE Transactions on 20.6 (2012): 1759-1770.
Rao, Vishweshwara, and Preeti Rao. "Vocal melody extraction in the presence of pitched accompaniment in polyphonic music." Audio, Speech, and Language Processing, IEEE Transactions on 18.8 (2010): 2145-2154.
Loudness and Timbre: Motif Detection q Motif similarity: DTW
§ mnDP (#72 instances) § DnDP (#56 instances)
q # Combinations: § Postives: 4096 § Negative: 4032
q Dataset: IIT Bombay § Raga: Alhaiya Bilawal § # Performances: 5 § # Artists: 4
Ross, J. C., & Rao, P. (2012). Detection Of Raga-Characteristic Phrases From Hindustani Classical Music Audio. In 2nd CompMusic Workshop (pp. 133–138).
ROC: Pitch
ROC: Timbre
ROC: Loudness
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology
Melody Segmentation q Task, context and music style/tradition
dependent q Do we need it? (motif discovery)
§ At what stage of processing?
q Melodic motifs <> nyas svar
q Melodic segmentation: estimating boundaries of nyas svars
Ross, J. C., & Rao, P. (2012). Detection Of Raga-Characteristic Phrases From Hindustani Classical Music Audio. In 2nd CompMusic Workshop (pp. 133–138).
Nyas Svar Segmentation
Nyas Svar Segmentation
Nyas Svar Segmentation
Nyas Svar Segmentation
S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013
1 2 3 4 5 6 7 8 9 10
850
900
950
1000
1050
1100
1150
T1
Sn
Sn-1
Sn+1
ε ρ1 T2 T3
ρ2
T4 T7 T8 T9
T5 T6
Nyās segment
Time (s)
Fund
amen
tal F
requ
ency
(cen
ts)
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology
Nyas Identification
S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013
Predominant melody extraction Tonic identification
Audio
Nyās svars
Melody representation
Histogram computation
Melody segmentation
Svar identification Segmentation
Local Feature extraction Contextual Local + Contextual
Segment classification Segment fusion Segment classification and fusion
Nyas Identification q Local Features (#9)
§ Segment length
§ Mean, variance of pitch values
§ Mean, variance of the differences in adjacent peak locations in pitch sequence
§ Mean, variance of peak amplitudes of pitch sequence
§ Temporal centroid
§ Flatness measure (output of segmentation method)
Nyas Identification q Local Features (#3)
§ Segment length
§ Mean, variance of pitch values
§ Mean, variance of the differences in adjacent peak locations in pitch sequence
§ Mean, variance of peak amplitudes of pitch sequence
§ Temporal centroid
§ Flatness measure (output of segmentation method)
Nyas Identification q Contextual Features (#24)
§ Segment length / (longest segment length in breath phrase)
§ Segment length / (length of the breath phrase)
§ Segment length / (length of the previous segment)
§ Segment length / (length of the following segment)
§ Duration between the ending and succeeding silence
§ Duration between the starting and preceding silence
§ All local features of adjacent segments
Nyas Identification q Contextual Features (#15)
§ Segment length / (longest segment length in breath phrase)
§ Segment length / (length of the breath phrase)
§ Segment length / (length of the previous segment)
§ Segment length / (length of the following segment)
§ Duration between the ending and succeeding silence
§ Duration between the starting and preceding silence
§ All local features of preceding segments
Nyas Identification: Baselines q Segmentation
§ Piece-wise linear segmentation
q Classification § DTW + kNN classification
q Several random baselines
Nyas Identification: Dataset q 20 recordings of Hindustani music
§ 15 Polyphonic: CompMusic collection § 5 Monophonic: Kaustuv’s recordings
q Unique artists: 8 q Unique Rāgs: 16 q Number of nyās segments: 1257 q Duration of nyās segments
§ Range: 150 ms – 16.7 s § Mean: 2.4 s § Median: 1.4 s
Nyas Identification: Results
Local features with
Proposed Segmentation
S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013
Nyas Identification: Results
Local features with
Proposed Segmentation
S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology for melodic motif discovery
Melodic Motif Discovery: Brute-force
Tonic Identification
Rhythm Analysis
Melodic Feature Extraction
Melody Representation
Melody Segmentation
Melodic Similarity
R�g characterization
& Music similarity
measures
Data extracted knowledge Expert knowledge
Redundancy reduction
Pattern Extraction
Listening tests
Dunya integration
Audio Metadata
Annotations
Melodic motivic discovery Evaluation
Melodic motives representation
Melodic motives similarity graph
• Rhythmic Variations (overall
timing, non-linear time variations)
• Extent of pitch variations • Melodic segmentation
information
• Nyas swaras • Motif transpositions • Salient melodic movements
Block diagram of the proposed semi-supervised methodology for melodic motif discovery