CJNLP-2008 1

Automatic Singing System of Japanese Songs in Chinese

6th Nov. 2008CJNLP-2008

EHARA TerumasaTokyo University of Science, Suwa

CJNLP-2008 2

Motivation

• To enhance the cultural exchange between Japan and China

• To make Chinese automatic singing system

• To improve the quality of singing software to it of the human singing

CJNLP-2008 3

Procedure of automatic singingJapanese song

Lyrics Musical score

Machine-translation/ post-editing

Speaking in Chinese

Converting to singing voice

Singing in Chinese

Tone / duration

CJNLP-2008 4

Converting to singing voiceSpeaking voice

STRAIGHT* (analyzing part)

* Kawahara, H. et al., “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds”, Speech Communication, Vol. 27, pp. 187–207, 1999.

F0 contourSpectrum envelope

Musical score

STRAIGHT (synthesizing part)

Singing voice

Duration and F0 control

CJNLP-2008 5

Duration control

• piecewise linear function

inp - out000-000360-200540-400630-650830-750

CJNLP-2008 6

F0 control

• Model 0F0(t) = Y0

• Model 1F0(t) = Y0 + (a1＊t + a0)

• Model 2F0(t) = Y0 + (a2＊t 2 + a1＊t + a0)

CJNLP-2008 7

Controlled F0 (model 0)

Blue line ： original F0Red line ： controlled F0

CJNLP-2008 8

Preliminary study

Original妈 麻 马 骂

Model 0

Model 1

Model 2

CJNLP-2008 9

Experiment

CJNLP-2008 10

Original Japanese song

(Spring has come)

CJNLP-2008 11

Machine-translation / post-editing

春が来た春が来たどこに来た山に来た里に来た野にも来た

春天来临了春天来临了来了哪里来了到山来了村落也来了野地

春天已来了春天已来了来到了哪里来到了山里来到了村落也来到野地

Machine-translation Post-editing

CJNLP-2008 12

Convert to a singing voice

Speaking Singing (F0 model 0)

来 到 了 哪 里 来 到 了 哪 里～～

CJNLP-2008 13

Duration control

来 到 了 哪 里～～～～

来到

了哪

里

CJNLP-2008 14

Speaking voice

CJNLP-2008 15

Singing voice

CJNLP-2008 16

Conclusion

• To develop an automatic singing system of Japanese songs in Chinese.

• To translate Japanese lyrics into Chinese using machine-translation and post-editing.

• To convert a speaking voice to a singing voice in Chinese.

CJNLP-2008 17

Future works

• To control Chinese syllabic tones appropriately.

• To use more sophisticated F0 and duration control technique*

• To extend our method to more complicated Japanese songs other than children’s songs

* Takeshi Saitou et. al : Vocal conversion from speaking voice to singing voice using STRAIGHT, Interspeech 2007, Aug. 2007.

CJNLP-2008 18

Acknowledgement• Providing STRAIGHT in research use

Prof. KAWAHARA Hideki (Wakayama univ.)• Collaborators

Assistant prof.OTA Kenko

Former graduate studentKUBO Yasushi

Under graduate studentKOBAYASHI Minoru, YAMAKOSHI Yuuta

Research studentQigeqi

CJNLP-2008 19

Thank you for your attention

MotivationProcedure of automatic singingConverting to singing voiceDuration controlF0 controlControlled F0 (model 0)Preliminary studyExperimentOriginal Japanese songMachine-translation / post-editingConvert to a singing voiceDuration controlSpeaking voiceSinging voiceConclusionFuture worksAcknowledgementThank you for your attention