Tracking Deformable Surfaces with Optical Flow in the ...€¦ · NORDIA’08, 27 June 2008 Optical...
Transcript of Tracking Deformable Surfaces with Optical Flow in the ...€¦ · NORDIA’08, 27 June 2008 Optical...
Image Processing DepartmentComputer Vision & Graphics Group
Berlin, Germany
• Anna Hilsmann
Tracking Deformable Surfaces with Optical Flow in the Presence of Self-Occlusion in Monocular
Image Sequences
NORDIA’08, 27-28 June 2008
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Outline
• Optical Flow Based Tracking and Formulation of the Motion Model
• Handling Optical Flow Constraints• Occlusion Handling• Experimental Results
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Optical Flow Constraint Equation
At each pixel in the image:
Minimize the error functional
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Optical Flow Constraint Equation
At each pixel in the image:
Minimize the error functional
BUT: one equation in two unknowns for each pixel !
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Optical Flow Constraint provides one equation for two unknownsRegularization by motion model
triangulated, regular 2D mesh with vertices
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Optical Flow Constraint provides one equation for two unknownsRegularization by motion model
triangulated, regular 2D mesh with vertices
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Back to the Optical Flow Constraint Equation
Minimize the error functional
Use the parameterized motion model to regularize optical flow field !
with
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Back to the Optical Flow Constraint Equation
Minimize the error functional
Use the parameterized motion model to regularize optical flow field !
with
Overdetermined equation system that can efficiently be solved in a linear least squares sense
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Incorporating spatial smoothing constraints
with
?
‘Laplacian vertex displacement deviation‘defined locally on the mesh topology
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Incorporating spatial smoothing constraints
with
?
Overdetermined equation system that can efficiently be solved in a linear least squares sense
‘Laplacian vertex displacement deviation‘defined locally on the mesh topology
Anna HilsmannNORDIA’08, 27 June 2008
Optical Flow Based Deformable Surface Tracking
Handling Optical Flow Constraints
At each pixel in the image:
Anna HilsmannNORDIA’08, 27 June 2008
Handling Optical Flow Constraints
At each pixel in the image:
However...
... only valid for small displacments between two successive frames hierarchical framework
Optical Flow Based Deformable Surface Tracking
Anna HilsmannNORDIA’08, 27 June 2008
Handling Optical Flow Constraints
At each pixel in the image:
However...
... only valid for small displacments between two successive frames hierarchical framework
... only valid for constant and uniform illumination
bandpass filtered images & model-to frame estimation
Optical Flow Based Deformable Surface Tracking
Anna HilsmannNORDIA’08, 27 June 2008
Handling Optical Flow Constraints... only valid for constant and uniform illumination
Optical Flow Based Deformable Surface Tracking
Illumination is not constant ...
bandpass filter
use a model frame ......and estimate transformationto next frame
warp it with current estimate...
Anna HilsmannNORDIA’08, 27 June 2008
Handling Optical Flow Constraints
At each pixel in the image:
Optical Flow Based Deformable Surface Tracking
However...
... only valid for small displacments between two successive frames hierarchical framework
... only valid for constant and uniform illumination
bandpass filtered images & model-to frame estimation
… aperture problem motion model
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
Normally, a 2D mesh folds under self-occlusions...
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
Normally, a 2D mesh folds under self-occlusions...
bad state in 2D monocular tracking
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
Normally, a 2D mesh folds under self-occlusions...
Idea: force the mesh to shrink instead of fold to stabelize the trackingGay-Bellile et al. ICCV 2007
bad state in 2D monocular tracking
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
We defined the mesh smoothness locally on vertex topology
with
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
We defined the mesh smoothness locally on vertex topology:
Idea: weight the smoothing constraints locally according to the occlusion of a region and force the mesh to shrink instead of fold
with
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
Occlusion maps from local shrinking regions in the mesh:
Anna HilsmannNORDIA’08, 27 June 2008
Self-Occlusion Handling
Anna HilsmannNORDIA’08, 27 June 2008
Results
T-Shirt Logo Sequence, 500 frames, 25fps
Anna HilsmannNORDIA’08, 27 June 2008
Results
T-Shirt Folding Sequence, 500 frames, 25fps Occlusion Map
Anna HilsmannNORDIA’08, 27 June 2008
Results
Paper Sequence, 100 frames, 25fps Occlusion Map
Anna HilsmannNORDIA’08, 27 June 2008
Results
without occlusion handling the mesh folds...
with occlusion handling the mesh shrinks at the boundary.Paper Sequence
Anna HilsmannNORDIA’08, 27 June 2008
Results
Knut Sequence, 100 frames, 25fps Occlusion Map
Anna HilsmannNORDIA’08, 27 June 2008
Results
Picasso Sequence, 500 frames, 25fps Augmentation
Anna HilsmannNORDIA’08, 27 June 2008
Results
Anna HilsmannNORDIA’08, 27 June 2008
Conclusion and applications
• straight-forward optical flow based algorithm for single view tracking
• mesh based motion model
• closed-form solution
• suitable for real-time applications:
e.g. Virtual Mirror, Virtual Clothing, Virtual Dressing Rooms
Anna HilsmannNORDIA’08, 27 June 2008
Thank You
The work presented in this paper has been developed with the support of the European Network of Excellence VISNET II (Contract IST-1-038398).