- Sridhar Godavarthy. Expressions Microexpressions FACS Evolutionary Psychology Proposed method...

-Sridhar Godavarthy

Micro Expression Spotting in Video using Strain Patterns

Expressions Microexpressions FACS Evolutionary Psychology

Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field

Optic Flow Optic Strain

Datasets Current Work Future Work Results

Initial WACV Optic Flow Thresholding

Contents

ExpressionsMicroexpressionsFACSEvolutionary Psychology

Contents

Primary means of social emotion conveyanceNon verbalConveys emotional stateVoluntary or involuntaryMinuscule differences in muscle movement

Expressions

Expressions Contd…

• 6 primary expressions(Not all clearly distinguishable)

•Can you identify the ones in this picture?

What are microexpressions?Subtle movements of the human faceUsually caused when attempting to mask a

macro-expressionQuick enough to be completed within the

blink of an eyeLast from 1/25th to 1/5th of a secondRestricted to certain muscles(regions) of the

faceAlmost impossible to fake

Microexpressions

Examples:Raising an eyebrowShrugging of shouldersPout of lipsFast blinking of eye

Non Examples:TalkingSmilingLaughingAnger

Microexpressions Contd…

For compartmentalization and categorization of human expressions

32 Action Units (with muscle involvement)14 Action Descriptors ( ! -do-)Can be used to code any possible expressionWidely used in pain recognition and facial

expression simulations

The Facial Action Coding System (FACS)

1000 page manual.Devised and written by one man.Requires extensive training.Some success in automating [ Bartlett et. al

1999] .

FACS Contd…

FACS Examples

Study of everything we discussed until nowThe child of ONE man - Paul Ekman.

Over thirty years of researchOne of the world’s leading experts on lying.About 2 dozen books and innumerable

articlesDeveloped FACSScientific Advisor to “Lie to Me”Co creator of Microexpression Training Tool

(METTx)

Evolutionary Psychology




Datasets

Current Work

Future Work

Results Initial WACV Optic Flow Thresholding

Contents

OutlineFace Detection &

Translation

Optical Strain

Split into ROI

Motion Field Estimation(Optical Flow)

Eye Detection/Alignment

Thresholding for period, strain

Combine and count ROI

Video is a collection of individual images also known as frames

In reality: spatial and temporal compression using properties of the scene.

Any video can be decoded into a series of frames.

24/30 frames per second of video.

Video

The science of encoding a video in a manner such thatMinimum number of

bits are used Motion compensated

prediction can be performed from a previous frame.

Video Coding


Translation

Optical Strain

Split into ROI





Face Detection & ROI


Translation

Optical Strain

Split into ROI





2D vector field of velocities of the image points induced by the relative motion.

Motion Field

• Feature-based methodsExtract visual features (corners, textured areas) and

track them over multiple framesSparse motion fields, but more robust trackingSuitable when image motion is large (10s of pixels)

• Direct methodsDirectly recover image motion at each pixel from

spatio-temporal image brightness variationsDense motion fields, but sensitive to appearance

variationsSuitable for video and when image motion is small

Obtaining Motion Field

• Def: Optical Flow is the apparent motion of brightness patterns in the image

• Ideally, same as the motion field• Have to be careful: apparent motion can be

caused by lighting changes without any actual motion

Optical Flow

Brightness constancyUnder most circumstance, the

apparent brightness of moving objects remain constant

Optical Flow EquationRelation of the apparent

motion with the spatial and temporal derivatives of the image brightness

Aperture problem Only the component of the

motion field in the direction of the spatial image gradient can be determined

The component in the direction perpendicular to the spatial gradient is not constrained by the optical flow equation

Optical Flow Contd…

0),,(

dt

tyxdE

0 tyx EvEuE• Key assumptions

• Brightness constancy: projection of the same point looks the same in every frame

• Small motion: points do not move very far

• Spatial coherence: points move like their neighbors

Constant Flow Method Assumption: the motion field is well approximated by a

constant vector within any small region of the image plane

Solution: Least square of two variables (u,v) from NxN Equations – NxN (=5x5) planar patch

Condition: ATA is NOT singular (null or parallel gradients)

Weighted Least Square Method Assumption: the motion field is approximated by a

constant vector within any small region, and the error made by the approximation increases with the distance from the center where optical flow is to be computed

Solution: Weighted least square of two variables (u,v)

from NxN Equations – NxN patch Affine Flow Method

Assumption: the motion field is well approximated by a affine parametric model uT = ApT+b (a plane patch with arbitrary orientation)

Solution: Least square of 6 variables (A,b) from NxN

Equations – NxN planar patch

Estimating Optical Flow


Translation

Optical Strain

Split into ROI





ElasticityDifferent materials have different elasticityElasticity can be modeled

strain

stressElasticity Known

Calculate

What is Facial Strain?Strain on soft tissue when expressions are

made.Anatomical methodUses a pair of frames to measure deformation

Facial Strain

Finite Element MethodForward modeling when Dirichlet condition is satisfiedGood at handling irregular shapesComputationally expensiveThis method is an approximation to the solution

Finite Difference MethodStrain, a tensor, can be expressed derivatives of the

displacement vectorThis can be approximated by a Finite Difference

Method.Very efficient when carried out on a regular grid.This method is an approximation to the differential

equation

Strain Computation - methods

Finite Difference MethodCompute spatial derivatives from discrete

points. Forward Difference MethodCentral Difference MethodRichardson extrapolation

Strain Measurement

Why Optical Strain?

Optical Flow Optical Strain




DatasetsCurrent WorkFuture Work Results

Initial WACV Optic Flow Thresholding

Contents

Name #videos ~ #Sequences Total

Political 10 7-9 80

USF (30) 8-9 ~250

Found Videos 4(+10) 1-2 ~20

ASL 1 TBGt -

Datasets

Eye detection/face alignment to accommodate head movement/rotation.

Automatic thresholdingDataset collectionTesting on “interesting” videosTrying a different Optical Flow( Black and

Anandan)Run expression detection to remove Macro

expressions first.

Current work

Micro expression recognition

Future Work




Datasets

Current Work

Future Work

ResultsInitialWACVOptic FlowThresholding

Contents

RESULTS - Initial

RESULTS – Initial Contd…

Results – WACV (3 Regions)

Results - Optic Flow…

Images resized non-uniformly for presentation

Results -Thresholding

~22 frames

~5 frames

P. Ekman and W. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978

Malcolm Gladwell,” Blink: The Power of Thinking Without Thinking”, Back Bay Books (April 3, 2007)

G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski.

Classifying facial actions. IEEE Transactionson Pattern Analysis and Machine Intelligence,21(10):974–989, 1999

References

Sridhar GodavarthyDept. Of Computer Science and Engineering

University of South [email protected]

Merci Beaucoup!

- Sridhar Godavarthy. Expressions Microexpressions FACS Evolutionary Psychology Proposed method...

Documents

Transcript of - Sridhar Godavarthy. Expressions Microexpressions FACS Evolutionary Psychology Proposed method...