Chinese-Speaking 3D Talking Head
description
Transcript of Chinese-Speaking 3D Talking Head
![Page 1: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/1.jpg)
1
Chinese-Chinese-SpeakingSpeaking
3D 3D Talking HeadTalking Head
Project No: H08040Project No: H08040
Sang Siew HoonSang Siew Hoon
Supervisor: Dr Ng Teck KhimSupervisor: Dr Ng Teck Khim
![Page 2: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/2.jpg)
2
System ObjectivesSystem Objectives create a realistic Chinese-create a realistic Chinese-
speaking 3D talking model speaking 3D talking model
-- automaticautomatic
-- accurate and realisticaccurate and realistic
-- portableportable
![Page 3: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/3.jpg)
3
ProblemsProblems Lack of references on Lack of references on
ChineseChinese Involves various fields Involves various fields
-- Face modelingFace modeling
-- Text-to-speechText-to-speech
-- Face Animation (Focus)Face Animation (Focus)
![Page 4: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/4.jpg)
4
System OverviewSystem Overview5 modules5 modules
1.1. Face Model PreparationFace Model Preparation
2.2. Chinese VisemesChinese Visemes
3.3. Text-to-Speech (TTS)Text-to-Speech (TTS)
4.4. AnimationAnimation
5.5. RenderingRendering
![Page 5: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/5.jpg)
5
![Page 6: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/6.jpg)
6
Face Model Face Model PreparationPreparation
![Page 7: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/7.jpg)
7
Face Model Face Model PreparationPreparation
Face Modeling 3DS MaxFace Modeling 3DS Max
![Page 8: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/8.jpg)
8
Face Model Face Model PreparationPreparation
Defining MPEG-4 feature pointsDefining MPEG-4 feature points
![Page 9: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/9.jpg)
9
Face Model Face Model PreparationPreparation
Assigning lip verticesAssigning lip vertices
![Page 10: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/10.jpg)
10
Face Model Face Model PreparationPreparation
Cloning morph targetsCloning morph targets
FAP#3 open_jaw
FAP#4 lower_t_midlip
![Page 11: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/11.jpg)
11
Chinese VisemesChinese VisemesVisemesVisemes- the visual equivalent of a the visual equivalent of a
phonemephoneme
1.1. Classification of Chinese Classification of Chinese SyllablesSyllables
2.2. Definition of Chinese VisemesDefinition of Chinese Visemes
3.3. Refining Dynamic VisemesRefining Dynamic Visemes
![Page 12: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/12.jpg)
12
Chinese VisemesChinese Visemes Definition of Chinese VisemesDefinition of Chinese Visemes
![Page 13: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/13.jpg)
13
Chinese VisemesChinese Visemes Definition of Chinese VisemesDefinition of Chinese Visemes
![Page 14: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/14.jpg)
14
Chinese VisemesChinese Visemes Definition of Chinese VisemesDefinition of Chinese Visemes
![Page 15: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/15.jpg)
15
Chinese VisemesChinese Visemes Refining Dynamic VisemesRefining Dynamic Visemes1.1. <i> +<i> + e only e only replace <f> erreplace <f> er2.2. zh, z, y + I zh, z, y + I drop i drop i3.3. d, l, g, j, zh, z, y, w + u | ü d, l, g, j, zh, z, y, w + u | ü drop <i> drop <i>4.4. j, y + <f> ending with an j, y + <f> ending with an replace the an by en replace the an by en5.5. j + complex finals headed by i and followed by more than one <f> j + complex finals headed by i and followed by more than one <f> drop i in complex finals drop i in complex finals
![Page 16: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/16.jpg)
16
Text-to-SpeechText-to-Speech
![Page 17: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/17.jpg)
17
Text-to-SpeechText-to-Speech
![Page 18: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/18.jpg)
18
AnimationAnimation1.1. CoarticulationCoarticulation
2.2. Automatic Generation of Automatic Generation of Face AnimationFace Animation
3.3. Enhancing RealismEnhancing Realism
![Page 19: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/19.jpg)
19
AnimationAnimation Enhancing realismEnhancing realism
- Eye blinks- Eye blinks
- Eyebrow raising- Eyebrow raising
- Gaze- Gaze
- Head Rotation- Head Rotation
![Page 20: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/20.jpg)
20
ResultsResults Accurate and realisticAccurate and realistic No discrepancies for new No discrepancies for new
face modelsface models Automated results need Automated results need
slight manual interventionslight manual intervention
http://www.slamstudio.com/HYP/videoshttp://www.slamstudio.com/HYP/videos
![Page 21: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/21.jpg)
21
ConclusionConclusion ContributionsContributions1.1. own system to define Chinese visemes own system to define Chinese visemes
- adds to the current research works of- adds to the current research works of
Chinese talking heads (currently Chinese talking heads (currently limited). limited).
2.2. an automatic Chinese lip-an automatic Chinese lip-synchronization systemsynchronization system
- saves an animator much time and - saves an animator much time and efforteffort
![Page 22: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/22.jpg)
22
ConclusionConclusion System LimitationsSystem Limitations
1.1. TTS limitationTTS limitation
2.2. FAPs conflictsFAPs conflicts
3.3. Not integrated Not integrated
– – inconvenient for untrained inconvenient for untrained usersusers
![Page 23: Chinese-Speaking 3D Talking Head](https://reader036.fdocuments.in/reader036/viewer/2022062422/56814065550346895dabe153/html5/thumbnails/23.jpg)
23
ConclusionConclusion Future WorksFuture Works1.1. Automatic audio signal processing Automatic audio signal processing
2.2. Resolve FAPs values conflicts Resolve FAPs values conflicts
3.3. Automatic feature points Automatic feature points assignment system assignment system
4.4. Integration of systemIntegration of system