3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2,...

37
1 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008

Transcript of 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2,...

Page 1: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

1

3D BODY & FACE MODELING & ANIMATION

Dr. Tanju Erdem

May 2, 2008

Page 2: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

2

Outline

• 3D Human Modeling

• Facial Expressions

• Lip Synchronization

• Advanced Rendering

• Applications

Page 3: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

3

3D Human Modeling

• Body modeling– Skeleton and skin

• Face modeling – Texture mapping of multiple photographs

– Blend-shapes for facial expressions

• Hair modeling

Page 4: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

4

Body Modeling

• Skeleton fitted to the body mesh

• Hierarchical structure of 76 bones

Page 5: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

5

Body Modeling

• Shaded version of the skin

• ~14500 polygons

Page 6: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

6

Body Modeling

• Matrix palette skinning

• “Paint” weights for each joint

• Four influences per vertex

Page 7: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

7

Skeletal Animation

Page 8: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

8

Face Modeling• Momentum’s patented semi-automatic face

modeling method– A few photographs taken from specific angles– Accurately represent the geometry and the texture of

the face in every direction.

• Method is insensitive to:– Position, scale and illumination differences among

photographs– Rotation inaccuracies– Perspective deformations due to camera lens.

Page 9: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

9

2D Feature Points

Page 10: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

10

3D Reconstruction

nS

ynf

xnf pp ,, ,

Find

given

ff JI ˆ,ˆ fy

fx

f cc λ,,

fn

f

fnx

f

xnf

KESIScp ˆˆ

,•+•+

fn

f

fny

f

ynf

KESJScp ˆˆ

,•+•+

for each point (14x3)and for each view (5x6)

for each pt. & view (58x2)

Page 11: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

11

3D Face Adaptation•

Page 12: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

12

Texture Blending

Page 13: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

13

Facial Animation• Animation via weighted average of

blend shapes– Expression blendshapes

• Used to synthesize the six basic emotions– Happiness, Anger, Fear, Boredom, Surprise,

Sadness

– Viseme blendshapes• Used to synthesize sixteen visemes

– A, F, O, P, W, ...

Page 14: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

14

Expression Blendshapes

Page 15: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

15

Viseme Blendshapes

Page 16: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

16

Expressions

Page 17: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

17

Visemes

Page 18: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

18

Audio-Driven Face Animation• Speech/music for animation control• Auto-generate speaker & language

independent facial animation from speech:– Lip synchronization (lip-sync)– Expression – Head Motion

Page 19: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

19

Audio-driven AnimationAnalysis & Synthesis

AnimationAudio

AudioFeature

Extraction

Audio Database

AudioFeature

ExtractionAnimation

StatisticalTraining

Synthesis

Ana

lysi

sSy

nthe

sis

•lip-sync•expression•head motion•dance figuretiming info

Correlation model

Page 20: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

20

Lip Synchronization• Determine mouth shapes during

speech• Viseme (mouth shape)

– Basic unit for lip-sync, – Animation parameters for all possible

mouth shapes• 16 standard visemes in facial

animations

Page 21: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

21

Advanced Rendering• Shaded face model• ~3000 polygons, 2k x 2k textures

Page 22: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

22

Real-time Rendering• GPU based rendering with vertex and pixel

shaders • Skeletal and blend-shape animation• Dynamic shadow-casting lights• Bump mapping• Skin shading with subsurface scattering• Global illumination by cube maps and

ambient maps• Crease maps

Page 23: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

23

Skin Texture Maps

Page 24: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

24

Skin Shading• Real-time skin

shading with subsurfacescattering

• Dynamic shadows including self-shadowing

Page 25: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

25

Skin Shading (Diffuse)

Page 26: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

26

Skin Shading (Specular)

Page 27: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

27

Skin Shading (Normal)

Page 28: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

28

Facial Expressions

Page 29: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

29

Eye Controls

Page 30: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

30

Hair Modeling• Hairstyling

– Geometry of the hair, density, distribution, and orientation of hair strands.

• Hair simulation– Dynamic motion of hair, collision detection

between the hair and objects

• Hair rendering – Color, shadows, light scattering effects,

transparency

Page 31: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

31

Hair Animation

• By fitting a skeleton

• Simple but not suitable for all hairstyles

Page 32: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

32

Hair Styling

Page 33: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

33

Hair Styling

Page 34: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

34

Applications• Can be incorporated into any 3D rendering

environment– PC and console games – TV and movies– Mobile messaging– Lip reading

Page 35: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

35

PC & Console Games• Many game genres need human animation:

– Role-playing games (RPG)– Puzzle-adventure– First-person shooter (FPS)

• In-game cinematic cutscenes with dialogues

• Momentum’s Culpa Innata released worldwide in Fall 2007– 40+ main characters– 1400+ cutscenes– Animation localization for 5 languages

Page 36: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

36

www.momentum-dmt.comwww.culpainnata.com

Page 37: 3D BODY & FACE MODELING & ANIMATION · 3D BODY & FACE MODELING & ANIMATION Dr. Tanju Erdem May 2, 2008. 2 Outline • 3D Human Modeling • Facial Expressions • Lip Synchronization

37

Work Started on CI2