Recognition, Analysis and Synthesis of Gesture

Expressivity

George CaridakisIVML-ICCS

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Overview Corpus Image processing module Gesture Recognition Expressivity Analysis Expressivity Synthesis Applications

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Overview

Video sequences varying in activation-evaluation

Head and hand tracking

Hand Coordinates/Arm configuration

Synthetic expressive gesture reconstruction

Gesture expressivity

features extractionHMM classifier

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Corpus mint-IVML

• 7 subjects

• 7 gesture classes

• 20 gesture variations (3 quadrants)

• 20’ minutes – 30000 frames

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Corpus EmoTV

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Corpus GEMEP (on going…)

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Head detection Detect candidate facial areas Validate using skin probability Conclude on number of persons

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Hand Detection•Skin probability

•Thresholding & Morphology Operations

•Distance Transform

•Frame difference

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Tracking Scoring system based on:

• Skin region size• Distance wrt the previous position • Optical flow alignment • Spatial constraints

Thresholding scores Periodical re-initialization

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Head & Hand Tracking

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

HMM parameters for gestures States are head and hands coordinates

• XL-XR XH-XR XH-XL YL-YR YH-YR YH-YL 6 output states Bakis left-to-right models Continuous output distribution

• 3 Gaussian mixtures Arbitrary training initial estimation of

transition probabilities

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Recognition via HMM (Why HMMs?) Stochastic models fit the nature of the

gestures Fast convergence due to effective training

algorithms Sufficient modeling of the temporal aspect

of gestures Continuous HMMs suitable for gesture-

level classification

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

HMM overview

λ2=(A 2, B 2, π )

λ1=(A 1, B 1, π )

λM =(A M , B M , π )

Select Maxim un

Feature VectorSequence υ*= argm ax[P (O |λ υ)]

υ

P(O | λ1 )

P(O | λ2 )

P(O | λ M )

.

.

.

.

.

.

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Recognition via HMM

0 10 20 30 40 50 60 70 80 90-300

-250

-200

-150

-100

-50

0

50Xf - Xlh

Xf - XrhXlh - Xrh

0 10 20 30 40 50 60 70 80 90-400

-300

-200

-100

0

100

200Yf - Ylh

Yf - Yrh

Ylh - Yrh

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Results

Gesture Attention Clapping ExplainOne

hand Oh my

god Wave Go away

Unclassified

Attention 90.9 0 0 0 0 0 9.1 0

Clapping 0 75 25 0 0 0 0 0

Explain 0 0 60 0 0 0 0 40

One hand 0 0 0 83.3 0 16.7 0 0

Oh my god 0 0 8.3 0 83.3 0 08.3

Wave 25 0 0 0 0 62.5 12.5 0Go away 25 0 0 0 0 0 75 0

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Expressivity features analysis Overall activation Spatial extent Temporal Fluidity Power/Energy Repetitivity

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Overall activation Considered as the quantity of movement

during a conversational turn Computed as the sum of the motion

vectors’ norm

n

i

ilirOA0

|)(||)(|

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Spatial extent Modeled by expanding or condensing

the entire space in front of the agent that is used for gesturing

Calculated as the maximum Euclidean distance of the position of the two hands

The average spatial extent is also calculated for normalization reasons

max(| ( ( ) ( )) |)SE d r i l i

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Temporal The temporal parameter of the gesture

determines the speed of the arm movement of a gesture’s meaning carrying stroke phase and also signifies the duration of movements (e.g., quick versus sustained actions)

OATEduration

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Fluidity Differentiates smooth/graceful from

sudden/jerky ones. This concept seeks to capture the continuity between movements, the arms’ trajectory paths as well as the acceleration and deceleration of the limbs

To extract this feature from the input image sequences we calculate the sum of the variance of the norms of the motion vectors

0

var(| ( ) |) var(| ( ) |)n

i

FL r i l i

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Power/Energy The power is actually identical with the

first derivative of the motion vectors calculated in the first steps

| ( ) | | ( ) |d r x d l xPOdx dx

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Results of expressivity analysis

EF variation Overall Activation

Spatial Extent

Temporal Fluidity Power/Energy

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Expressive synthesis A system that

mimics user’s behaviour through the analysis of facial and gesture signals and expressivity

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Synthesis Greta Platform BAP calculation

• Plane assumption• Inverse kinematics• Manual adaptation

Expressivity features variations implemented in Greta’s BAP interpolation

humaine Summer School 2006, Genoa, IT Tutorial on Human Full-Body Movement and Gesture Analysis

Synthesis Results