KING’S SPEECH Foreign language: pronounce with...
Transcript of KING’S SPEECH Foreign language: pronounce with...
![Page 1: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/1.jpg)
KING’SSPEECHForeignlanguage:pronouncewithstyle!
“Towardsautomaticevaluationofpronunciation”
July5th 2017
![Page 2: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/2.jpg)
KING’S SPEECH:SYSTEM OVERVIEW
Speech&videoacquisitionandvisualization
Pronunciationanalysis
Gamescheduler
Avatar-basedrendering
Empowermentanalysis
Speechsynthesis/database
![Page 3: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/3.jpg)
STARTING POINT:TEXT AND SPEECH
Student
Teacher
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
IPA:Elva etydje
Howwellhasthestudentrepeatedtheteacher’sphrase?
Whattoevaluate?Vowels
Consonants
Intonation
Accentuation
Rhythmicgroups
Liaison
…andmore…
![Page 4: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/4.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
IS THE INPUT SPEECH GOOD ENOUGH?
Checktheinputsignal!Correctsamplingfrequency?
Sufficientsignalenergy?
Reasonableduration?
Hasthesignalclipped?
Doesitcontainnoise?
…?
ApplyVoiceActivityDetection(VAD)
![Page 5: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/5.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
SPEECH ALIGNMENT:WHAT’S BEHIND THE SCENES?
PhoneticalignmentUsesspeechrecognitiontechnology
TrainaHiddenMarkovModel(HMM)
Inputs:HMMmodel,speechfile&transcription
Output:timeintervalsofphonemesinspeechfile
Speechalignment
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
![Page 6: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/6.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
SPEECH ALIGNMENT:HOW TO VALIDATE IT?
Remember:wedon’tperformrecognition!CalculatetheLowFrequencyModulatedEnergy
Peakscorrespondtovowels(well,mostlikely!)
Compare:peaklocationsvs.alignmentresults
Speechalignment
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
LFMEAutomaticalignmentvalidation
![Page 7: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/7.jpg)
PRONUNCIATION ANALYSIS
Casestudy:Vowels
![Page 8: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/8.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
VOWELS ANALYSIS:WHAT FEATURES TO LOOK AT?
Formants:whatarethey?
Howtocalculatethem?LinearPredictionCoding(LPC)
Vocaltractlengthnormalization(VTLN)
Speechalignment
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
LFMEAutomaticalignmentvalidation
Formantsestimation LPC
Optimalorder
Warpingfunction(VTLN)
![Page 9: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/9.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
VOWELS ANALYSIS:WHAT ABOUT NASALS?
Nasals:whatarethey?
Howtodistinguish?Computesmoothedpowerspectrum
Findpeaksinregionsrelatedtoformants
Usepeakinformationasacousticparameters
Speechalignment
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
LFMEAutomaticalignmentvalidation
Formantsestimation LPC
Optimalorder
Warpingfunction(VTLN)
Nasalizationevaluation /ɛ/ /ɛ/̃
![Page 10: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/10.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
VOWELS ANALYSIS:COMPARISON AND FEEDBACK
Speechalignment
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
LFMEAutomaticalignmentvalidation
Formantsestimation LPC
Optimalorder
Warpingfunction(VTLN)
Nasalizationevaluation
Statisticalanalysis
Howtocomparevowels?Distancebetweenformantvectors(student/teacher)
Contextisimportant!
Checkdistancefromallothervowels
Oralswithorals,nasalswithnasals
![Page 11: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/11.jpg)
StudentAudiovalidation&pre-processing VAD
Teacher
VOWELS ANALYSIS:THE FULL PICTURE
Speechalignment
PhoneticTranscription(fənɛtɪk trænskrɪpʃən)
LFMEAutomaticalignmentvalidation
Formantsestimation LPC
Optimalorder
Warpingfunction(VTLN)
Nasalizationevaluation
Statisticalanalysis
![Page 12: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/12.jpg)
KING’S SPEECH:COMING NEXT
Speech&videoacquisitionandvisualization
Pronunciationanalysis
Gamescheduler
Avatar-basedrendering
Empowermentanalysis
Speechsynthesis/database
19th ofJuly:“Modular architectureforimmersivelearning applications”
LanguageKaraokescenario
Deeplearningphonemeevaluation
Student-TeacherFacialandSpeechsynchronization
![Page 13: KING’S SPEECH Foreign language: pronounce with style!artes.ucp.pt/enterface17/2017.07.05/Celine_Kings Speech... · 2017. 7. 5. · KING’S SPEECH Foreign language: pronounce with](https://reader036.fdocuments.in/reader036/viewer/2022071112/5fe89b5d2d29351e6919c878/html5/thumbnails/13.jpg)
THANK YOU FOR YOUR ATTENTION!