Eyes Alive Sooha Park - Lee Jeremy B. Badler - Norman I. Badler University of Pennsylvania - The...
-
Upload
silvester-joseph -
Category
Documents
-
view
214 -
download
1
Transcript of Eyes Alive Sooha Park - Lee Jeremy B. Badler - Norman I. Badler University of Pennsylvania - The...
Eyes AliveEyes AliveSooha Park - Lee Jeremy B. Badler - Norman I. BadlerSooha Park - Lee Jeremy B. Badler - Norman I. Badler
University of Pennsylvania - The Smith-Kettlewell Eye Research University of Pennsylvania - The Smith-Kettlewell Eye Research InstituteInstitute
Presentation Prepared By:Presentation Prepared By:Chris WidmerChris Widmer
CSE 4280CSE 4280
OutlineOutline
IntroductionIntroduction MotivationMotivation BackgroundBackground Overview of SystemOverview of System DescriptionsDescriptions ResultsResults ConclusionsConclusions
IntroductionIntroduction
Eye movement an important Eye movement an important expression techniqueexpression technique
Statistical eye movement model Statistical eye movement model based on empirical databased on empirical data
MotivationMotivation
Natural look eye movement for Natural look eye movement for animations of close-up face viewsanimations of close-up face views
Traditionally difficult to attain accurate Traditionally difficult to attain accurate eye movement in animationseye movement in animations
No proposals for saccadic eye No proposals for saccadic eye movement for easy use in movement for easy use in speaking/expressive facesspeaking/expressive faces
Recent interest in construction of Recent interest in construction of human facial modelshuman facial models
BackgroundBackground
To build a realistic face model:To build a realistic face model:– Geometry modelingGeometry modeling– Muscle behaviorMuscle behavior– Lip synchronizationLip synchronization– Text synthesisText synthesis
Research has traditionally not Research has traditionally not focused on eye movement.focused on eye movement.
BackgroundBackground
Eyes are essential for non-verbal Eyes are essential for non-verbal communicationcommunication
– Regulate flow of conversationRegulate flow of conversation– Search for feedbackSearch for feedback– Express emotionExpress emotion– Influence of behaviorInfluence of behavior
New approach based on statistical data New approach based on statistical data and empirical studiesand empirical studies
SaccadesSaccades Rapid movements of both eyes from one gaze Rapid movements of both eyes from one gaze
position to another.position to another.– Only Eye Movement Executed ConsciouslyOnly Eye Movement Executed Consciously– Balance conflicting demands of speed and accuracyBalance conflicting demands of speed and accuracy
MagnitudeMagnitude – angle the eyeball rotates to – angle the eyeball rotates to change positionchange position
DirectionDirection – 2D axis of rotation, 0 degrees to – 2D axis of rotation, 0 degrees to the rightthe right
DurationDuration – Time of movement – Time of movement Inter-saccadic IntervalInter-saccadic Interval – time between – time between
saccadessaccades
SaccadesSaccades Example: Magnitude 10, 45 degreesExample: Magnitude 10, 45 degrees
– Rotating 10 degrees, right-upwardRotating 10 degrees, right-upward
Initial/Final Acceleration: 30,000 deg/secInitial/Final Acceleration: 30,000 deg/sec Peak Velocity – 400 – 600 deg/secPeak Velocity – 400 – 600 deg/sec Reaction Time: 180 – 220 msecReaction Time: 180 – 220 msec Duration and Velocity Functions of MagnitudeDuration and Velocity Functions of Magnitude
Magnitude ApproximationMagnitude Approximation D = DD = D0 + d * A + d * A D = Duration, A = Amplitude, d = increment in D = Duration, A = Amplitude, d = increment in
duration per degree (2-2.7 msec/deg), Dduration per degree (2-2.7 msec/deg), D0 = = Intercept (20-30 ms)Intercept (20-30 ms)
Often accompanied by head rotationOften accompanied by head rotation
BackgroundBackground
Three Functions of GazeThree Functions of Gaze– Sending Social SignalsSending Social Signals– Open Channel to Receive InformationOpen Channel to Receive Information– Regulate Flow of ConversationRegulate Flow of Conversation
Overview of SystemOverview of System
1.1. Eye-tracking images analyzed, Eye-tracking images analyzed, statistically based model generated statistically based model generated using Matlabusing Matlab
2.2. Lip movements/Eye Blinks/Head Lip movements/Eye Blinks/Head Rotation analyzed by Rotation analyzed by alterEGOalterEGO face face motion analysis systemmotion analysis system
Overview of SystemOverview of System
Face Animation Parameter (FAP) FileFace Animation Parameter (FAP) File Eye Movement Synthesis System Eye Movement Synthesis System
(EMSS)(EMSS)– Adds eye movement data to FAP fileAdds eye movement data to FAP file
Modified from face2face’s animator Modified from face2face’s animator plug-in for 3D Studio Maxplug-in for 3D Studio Max
Analysis of DataAnalysis of Data
Eye movements recorded with eye-Eye movements recorded with eye-tracking visor (ISCAN) – monocle and two tracking visor (ISCAN) – monocle and two miniature camerasminiature cameras
One views environment from left eye One views environment from left eye perspective, other is close-up of left eyeperspective, other is close-up of left eye
Eye image recordedEye image recorded Device tracks by comparing corneal Device tracks by comparing corneal
reflection of the light source relative to the reflection of the light source relative to the location of the pupil centerlocation of the pupil center– Reflection acts as reference point while pupil Reflection acts as reference point while pupil
changes during movementchanges during movement
Analysis of DataAnalysis of Data
Pupil position found using pattern Pupil position found using pattern mappingmapping
Default threshold grey level using Default threshold grey level using Canny Edge Detection operatorCanny Edge Detection operator
Positional histograms along X and Y Positional histograms along X and Y axis calculatedaxis calculated
Two center points with maximum Two center points with maximum correlation chosencorrelation chosen
Analysis of DataAnalysis of Data
Analysis of DataAnalysis of Data
Analysis of DataAnalysis of Data
Saccade MagnitudeSaccade Magnitude
Frequency of a specific magnitude Frequency of a specific magnitude (least mean squares distribution)(least mean squares distribution)
• d = Distance traversed by pupil center
• r = radius of eyeball (1/2 of xmax
• P = % chance to occur
• A = Saccade Magnitude (Degrees)
Analysis of DataAnalysis of Data
Saccade Duration measured with 40 Saccade Duration measured with 40 deg/sec thresholddeg/sec threshold
Used to derive instantaneous velocity Used to derive instantaneous velocity curve for every saccadecurve for every saccade
Duration of each movement Duration of each movement normalized to 6 framesnormalized to 6 frames
Two classes of Gaze: Two classes of Gaze: – MutualMutual– AwayAway
Talking vs. ListeningTalking vs. Listening
Synthesis of Eye MovementSynthesis of Eye Movement
Attention Attention Monitor (Monitor (AttMonAttMon))
Parameter Parameter Generator Generator ((ParGenParGen))
Saccade Saccade Synthesizer Synthesizer ((SacSynSacSyn))
Head Rotation MonitoringHead Rotation Monitoring
Synthesis of Natural Eye Synthesis of Natural Eye MovementMovement
AttMonAttMon determines mode, changes in determines mode, changes in head rotation, gaze statehead rotation, gaze state
ParGenParGen determines saccade determines saccade magnitude, direction, duration, and magnitude, direction, duration, and instantaneous velocityinstantaneous velocity
SacSynSacSyn synthesizes and codes synthesizes and codes movements into FAP valuesmovements into FAP values
Synthesis of Natural Eye Synthesis of Natural Eye MovementMovement
Magnitude determined by inverse of fitting Magnitude determined by inverse of fitting function shown earlier (Slide 16)function shown earlier (Slide 16)
Mapping guarantees same probability Mapping guarantees same probability distribution as empirical datadistribution as empirical data
Direction determined by head rotation Direction determined by head rotation (threshold), and distribution table(threshold), and distribution table– Uniform Distribution, 0 to 100Uniform Distribution, 0 to 100– 8 non-uniform intervals assigned to respective 8 non-uniform intervals assigned to respective
directionsdirections
Synthesis of Natural Eye Synthesis of Natural Eye MovementMovement
Duration determined by first equation, Duration determined by first equation, respective values for d and Drespective values for d and D0
Velocity determined by using fitted Velocity determined by using fitted instantaneous velocity curveinstantaneous velocity curve
SacSyn system calculates sequence of SacSyn system calculates sequence of coordinates for sys centerscoordinates for sys centers
Translated into FAP values, rendered in 3D Translated into FAP values, rendered in 3D Studio MAXStudio MAX
Face2face animation plug-in to render Face2face animation plug-in to render animations with correct parametersanimations with correct parameters
ResultsResults
3 Different Methods Tested3 Different Methods Tested Type 1 -> No Saccadic MovementsType 1 -> No Saccadic Movements Type 2 -> Random Eye MovementType 2 -> Random Eye Movement Type 3 -> Sampled from Estimated Type 3 -> Sampled from Estimated
Distributions (synchronized with Distributions (synchronized with head movements)head movements)
Tests were subjectiveTests were subjective
ResultsResults
Q1: Did the character on the screen Q1: Did the character on the screen appear interested in (5) or indifferent (1) appear interested in (5) or indifferent (1) to you?to you?
Q2: Did the character appear engaged (5) Q2: Did the character appear engaged (5) or (1) distracted during the conversation?or (1) distracted during the conversation?
Q3: Did the personality of the character Q3: Did the personality of the character look friendly (5) or not (1)?look friendly (5) or not (1)?
Q4: Did the face of the character look Q4: Did the face of the character look lively (5) or deadpan (1)?lively (5) or deadpan (1)?
Q5: In general, how would you describe Q5: In general, how would you describe the character?the character?
ResultsResults
ConclusionsConclusions
Saccade Model for Talking and Listening Saccade Model for Talking and Listening ModesModes
3 Different Eye Movements: Stationary, 3 Different Eye Movements: Stationary, Random, Model-basedRandom, Model-based
Model-based scored significantly higherModel-based scored significantly higher Eye Tracking Data recorded from a subjectEye Tracking Data recorded from a subject
– New recorded data for every characterNew recorded data for every character
This model allows any number of unique eye This model allows any number of unique eye movement sequences.movement sequences.
Drawbacks and Drawbacks and ImprovementsImprovements
Aliasing with small movementsAliasing with small movements Sensing of eye movement vs. head Sensing of eye movement vs. head
movement during data gatheringmovement during data gathering
Future EnhancementsFuture Enhancements– Eye/Eyelid dataEye/Eyelid data– More model gaze patternsMore model gaze patterns– More subjects for dataMore subjects for data– Scan-path model for close-up imagesScan-path model for close-up images
DevelopmentsDevelopments
J. Badler, Director, J. Badler, Director, Center for Human Center for Human Modeling and SimulationModeling and Simulation– Digital Human Modeling/BehaviorDigital Human Modeling/Behavior– ““Jack” SoftwareJack” Software– Simulation of workflow using virtual Simulation of workflow using virtual
peoplepeople
ReferencesReferences
Badler, Jeremy B., Badler, Norman I, Lee, Badler, Jeremy B., Badler, Norman I, Lee, Sooha Park, “Eyes AliveSooha Park, “Eyes Alive
http://www.cis.upenn.edu/~sooha/http://www.cis.upenn.edu/~sooha/home.htmlhome.html
http://www.cis.upenn.edu/~badler/http://www.cis.upenn.edu/~badler/ http://cg.cis.upenn.edu/hms/research.htmlhttp://cg.cis.upenn.edu/hms/research.html