-Sridhar Godavarthy
Micro Expression Spotting in Video using Strain Patterns
Expressions Microexpressions FACS Evolutionary Psychology
Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field
Optic Flow Optic Strain
Datasets Current Work Future Work Results
Initial WACV Optic Flow Thresholding
Contents
ExpressionsMicroexpressionsFACSEvolutionary Psychology
Contents
Primary means of social emotion conveyanceNon verbalConveys emotional stateVoluntary or involuntaryMinuscule differences in muscle movement
Expressions
Expressions Contd…
• 6 primary expressions(Not all clearly distinguishable)
•Can you identify the ones in this picture?
What are microexpressions?Subtle movements of the human faceUsually caused when attempting to mask a
macro-expressionQuick enough to be completed within the
blink of an eyeLast from 1/25th to 1/5th of a secondRestricted to certain muscles(regions) of the
faceAlmost impossible to fake
Microexpressions
Examples:Raising an eyebrowShrugging of shouldersPout of lipsFast blinking of eye
Non Examples:TalkingSmilingLaughingAnger
Microexpressions Contd…
For compartmentalization and categorization of human expressions
32 Action Units (with muscle involvement)14 Action Descriptors ( ! -do-)Can be used to code any possible expressionWidely used in pain recognition and facial
expression simulations
The Facial Action Coding System (FACS)
1000 page manual.Devised and written by one man.Requires extensive training.Some success in automating [ Bartlett et. al
1999] .
FACS Contd…
FACS Examples
Study of everything we discussed until nowThe child of ONE man - Paul Ekman.
Over thirty years of researchOne of the world’s leading experts on lying.About 2 dozen books and innumerable
articlesDeveloped FACSScientific Advisor to “Lie to Me”Co creator of Microexpression Training Tool
(METTx)
Evolutionary Psychology
Expressions Microexpressions FACS Evolutionary Psychology
Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field
Optic Flow Optic Strain
Datasets
Current Work
Future Work
Results Initial WACV Optic Flow Thresholding
Contents
OutlineFace Detection &
Translation
Optical Strain
Split into ROI
Motion Field Estimation(Optical Flow)
Eye Detection/Alignment
Thresholding for period, strain
Combine and count ROI
OutlineFace Detection &
Translation
Optical Strain
Split into ROI
Motion Field Estimation(Optical Flow)
Eye Detection/Alignment
Thresholding for period, strain
Combine and count ROI
Video is a collection of individual images also known as frames
In reality: spatial and temporal compression using properties of the scene.
Any video can be decoded into a series of frames.
24/30 frames per second of video.
Video
The science of encoding a video in a manner such thatMinimum number of
bits are used Motion compensated
prediction can be performed from a previous frame.
Video Coding
OutlineFace Detection &
Translation
Optical Strain
Split into ROI
Motion Field Estimation(Optical Flow)
Eye Detection/Alignment
Thresholding for period, strain
Combine and count ROI
Face Detection & ROI
OutlineFace Detection &
Translation
Optical Strain
Split into ROI
Motion Field Estimation(Optical Flow)
Eye Detection/Alignment
Thresholding for period, strain
Combine and count ROI
2D vector field of velocities of the image points induced by the relative motion.
Motion Field
• Feature-based methodsExtract visual features (corners, textured areas) and
track them over multiple framesSparse motion fields, but more robust trackingSuitable when image motion is large (10s of pixels)
• Direct methodsDirectly recover image motion at each pixel from
spatio-temporal image brightness variationsDense motion fields, but sensitive to appearance
variationsSuitable for video and when image motion is small
Obtaining Motion Field
• Def: Optical Flow is the apparent motion of brightness patterns in the image
• Ideally, same as the motion field• Have to be careful: apparent motion can be
caused by lighting changes without any actual motion
Optical Flow
Brightness constancyUnder most circumstance, the
apparent brightness of moving objects remain constant
Optical Flow EquationRelation of the apparent
motion with the spatial and temporal derivatives of the image brightness
Aperture problem Only the component of the
motion field in the direction of the spatial image gradient can be determined
The component in the direction perpendicular to the spatial gradient is not constrained by the optical flow equation
Optical Flow Contd…
0),,(
dt
tyxdE
0 tyx EvEuE• Key assumptions
• Brightness constancy: projection of the same point looks the same in every frame
• Small motion: points do not move very far
• Spatial coherence: points move like their neighbors
Constant Flow Method Assumption: the motion field is well approximated by a
constant vector within any small region of the image plane
Solution: Least square of two variables (u,v) from NxN Equations – NxN (=5x5) planar patch
Condition: ATA is NOT singular (null or parallel gradients)
Weighted Least Square Method Assumption: the motion field is approximated by a
constant vector within any small region, and the error made by the approximation increases with the distance from the center where optical flow is to be computed
Solution: Weighted least square of two variables (u,v)
from NxN Equations – NxN patch Affine Flow Method
Assumption: the motion field is well approximated by a affine parametric model uT = ApT+b (a plane patch with arbitrary orientation)
Solution: Least square of 6 variables (A,b) from NxN
Equations – NxN planar patch
Estimating Optical Flow
OutlineFace Detection &
Translation
Optical Strain
Split into ROI
Motion Field Estimation(Optical Flow)
Eye Detection/Alignment
Thresholding for period, strain
Combine and count ROI
ElasticityDifferent materials have different elasticityElasticity can be modeled
strain
stressElasticity Known
Calculate
What is Facial Strain?Strain on soft tissue when expressions are
made.Anatomical methodUses a pair of frames to measure deformation
Facial Strain
Finite Element MethodForward modeling when Dirichlet condition is satisfiedGood at handling irregular shapesComputationally expensiveThis method is an approximation to the solution
Finite Difference MethodStrain, a tensor, can be expressed derivatives of the
displacement vectorThis can be approximated by a Finite Difference
Method.Very efficient when carried out on a regular grid.This method is an approximation to the differential
equation
Strain Computation - methods
Finite Difference MethodCompute spatial derivatives from discrete
points. Forward Difference MethodCentral Difference MethodRichardson extrapolation
Strain Measurement
Why Optical Strain?
Optical Flow Optical Strain
Expressions Microexpressions FACS Evolutionary Psychology
Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field
Optic Flow Optic Strain
DatasetsCurrent WorkFuture Work Results
Initial WACV Optic Flow Thresholding
Contents
Name #videos ~ #Sequences Total
Political 10 7-9 80
USF (30) 8-9 ~250
Found Videos 4(+10) 1-2 ~20
ASL 1 TBGt -
Datasets
Eye detection/face alignment to accommodate head movement/rotation.
Automatic thresholdingDataset collectionTesting on “interesting” videosTrying a different Optical Flow( Black and
Anandan)Run expression detection to remove Macro
expressions first.
Current work
Micro expression recognition
Future Work
Expressions Microexpressions FACS Evolutionary Psychology
Proposed method Outline Video Face Detection, Alignment and Splitting Motion Field
Optic Flow Optic Strain
Datasets
Current Work
Future Work
ResultsInitialWACVOptic FlowThresholding
Contents
RESULTS - Initial
RESULTS – Initial Contd…
Results – WACV (3 Regions)
Results - Optic Flow…
Images resized non-uniformly for presentation
Results -Thresholding
~22 frames
~5 frames
P. Ekman and W. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, 1978
Malcolm Gladwell,” Blink: The Power of Thinking Without Thinking”, Back Bay Books (April 3, 2007)
G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski.
Classifying facial actions. IEEE Transactionson Pattern Analysis and Machine Intelligence,21(10):974–989, 1999
References
Sridhar GodavarthyDept. Of Computer Science and Engineering
University of South [email protected]
Merci Beaucoup!
Top Related