Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of...
-
Upload
shon-parrish -
Category
Documents
-
view
218 -
download
0
Transcript of Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of...
Modeling 3D Deformable and Articulated Shapes
Yu Chen, Tae-Kyun Kim, Roberto Cipolla
Department of EngineeringUniversity of Cambridge
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
Motivation
+
3D Shapes
UncertaintyMeasurements
2D Images
Tasks: – To recover deformable shapes from a single
image with arbitrary camera viewpoint.
Previous Work
Rigid shapes [Prasad’05, Rother’09, Yu’09, etc.]Problems: – Cannot handle self-deformation or articulations.
Category-specific articulated shapese.g., human bodies [Anguelov’05, Balan’07, etc.]Problems: – Requiring strong shape or anatomical knowledge of the
category, such as skeletons and joint angles.– Too many parameters to estimate;– Hard to be generalised to other object categories.
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
Our Contribution
A probabilistic framework for:– Modelling different shape variations of
general categories;– Synthesizing new shapes of the category
from limited training data;– Inferring dense 3D shapes of deformable or
articulated objects from a single silhouette;
Explanations on the Graphical Model
Shape Synthesis Matching Silhouettes
Pose Generator
Shape Generator
Joint Distribution:
Generating Shapes
Target: Simultaneous modelling two types of shape variations: – Phenotype variation:
fat vs. thin, tall vs. Short... – Pose variation:
articulation, self deformation, ...Training two GPLVMs:
– Shape generator (MS) for phenotype variation;
– Pose generator (MA) for pose variation.
Shape Generator (MS)– Training Set:
Shapes in the canonical pose.
– Pre-processing: Automatically register each instance with a common
3D template; 3D shape context matching and thin-plate spline
interpolation;Perform PCA on all registered 3D shapes.
– Input: PCA coefficients of all the data.
Generating Shapes
Generating ShapesPose Generator (MA)
– Training Set: Synthetic 3D poses
sequences.
– Pre-processing: Perform PCA on both spatial
positions of vertices and all vertex-wise Jacobian matrices.
– Input: PCA coefficients of all the
data
Shape Synthesis
ZeroShape
V0
Pose Generator
MA
ShapeGenerator
MS
VA VA
VSVS
ShapeSynthesis
V
V
Shape SynthesisModelling the local shape transfer
– Computing Jacobian matrices on the zero shape vertex-wisely.
Ji
Shape SynthesisSynthesizing fully-varied shape V from
phenotype-varied shape VS and pose-varied shape VA.
Probabilistic formulation: a Gaussian Approximation
Matching Silhouettes A two-stage process:
o Projecting the 3D shape onto the image plane
o Chamfer matching of silhouettes
Maximizing likelihood over latent coordinates xA, xS and camera parameters γko Optimizing the closed-form lower bound.o Adaptive line-search with multiple initialisations.
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
Experiments on Shape Synthesis
Task: – To synthesize shapes in different phenotypes
and poses with the mean shape μV.
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
Experiments on Single View Reconstruction
Training dataset:– Shark data:
MS: 11 3D models of different shark species .
MA: 11-frame tail-waving sequence from an animatable 3D MEX model.
– Human data:
MS: CAESAR dataset.
MA: Animations of different 3D poses of Sydney in Poser 7.
Testing: – Internet images (22 sharks and 20 humans in different
poses and camera viewpoints)Segmentation: GrabCut [Rother’04]
Experiments on Single View Reconstruction
Sharks:
Experiments on Single View Reconstruction
Humans:
Experiments on Single View Reconstruction
Examples of multi-modality
Experiments on Single View Reconstruction
Qualitative Results: Precision-Recall Ratios– SF: foreground regions
– SR: image projection of our result
A very good approximation to the results given by parametrical models
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
Pros and Cons:
Advantages
Fully data driven; Requiring no strong class-
specific prior knowledge, e.g., skeleton, joint angles;
Capable of modelling general categories;
Compact shape representation and much lower dimensions for efficient optimization;
Uncertainty measurements provided.
Disadvantages
Inaccurate at fine parts, e.g., hands.
Lower descriptive power on poses compared with parametric model, when training instances are not enough;
Training data are sometimes difficult to obtain.
Future Work
A compatible framework which allows incorporating category knowledge
Incorporating more cues: internal edges, texture, and colour;
Multiple view settings and video sequences;
3D object recognition and action recognition tasks.
Thanks!