Silhouette-based Object Phenotype Recognition using 3D Shape Priors
description
Transcript of Silhouette-based Object Phenotype Recognition using 3D Shape Priors
Silhouette-based Object Phenotype Recognition using 3D Shape Priors
Yu Chen1 Tae-Kyun Kim2 Roberto Cipolla1
University of Cambridge, Cambridge, UK1 Imperial College, London, UK2
Problem Description
Task: To identify the phenotype class of deformable objects.
Given a gallery of canonical-posed silhouettes in different phenotype classes.
Can we find out
?
Problem DescriptionMotivation:
– Pose recognition is widely investigated;– Phenotype recognition is somehow overlooked;– Applications?
Difficulty: – Pose and camera viewpoint variations are more
dominant than the phenotype variation.
Problem Description 2D approaches hardly work in this
case.
Our strategy: make use of the 3D shape prior of deformable objects.
Shall we use a purely generative approach?
No! Too expensive to perform for a recognition task!
Solution: Two-Stage Model Main Ideas:
Discriminative + Generative Two stages:1. Hypothesising
– Discriminative;– Using random forests;
2. Shape Synthesis and Verification– Generative;– Synthesising 3D shapes
using shape priors;– Silhouette verification.
Recognition by a model selection process.
• Use 3 RFs to quickly hypothesize phenotype, pose, and camera parameters.
• Learned on synthetic silhouettes generated by the shape priors.
Parameter Hypothesizing
FA: Pose classifier
FC: Camera pose classifier
FS: Phenotype classifier
(canonical pose)
Examples of Tree Classifiers
The phenotype classifier
The pose classifier
Training RF Classifiers Random Features:
– Rectangle pairs with random sizes and locations.
– Difference of mean intensity values[Shotton et al. 09]
– Feature error compensation for phenotype classifier;
Criteria Function:– Similarity-aware diversity
index.
Shape Synthesis and Verification
Generate 3D shapes V – From candidate parameters
given by RFs.– Use GPLVM shape priors
[Chen et al.’10].
Compare the projection of V with the query silhouette Sq.
– Oriented Chamfer matching (OCM). [Stenger et al’03]
ExperimentsTesting data:
– Manually segmented silhouettes;
Current Datasets– Human jumping jack
(13 instances, 170 images);– Human walking
(16 instances, 184 images);– Shark swimming
(13 instances, 168 images).Phenotype Categorisation
Comparative Approaches:
Learn a single RF phenotype classifier; Histogram of Shape Context (HoSC)
– [Agarwal and Triggs, 2006] Inner-Distance Shape Context (IDSC)
– [Ling and Jacob, 2007] 2D Oriented Chamfer matching (OCM)
– [Stenger et al. 2006] Mixture of Experts for the shape reconstruction
– [Sigal et al. 2007].– Modified into a recognition algorithm
Comparative Approaches: Internal comparisons:
– Proposed method with both feature error modelling and similarity-aware criteria function (G+D);
– Proposed method w.o feature error modelling (G+D–E);
– Proposed method w.o similarity-aware criteria function (G+D–S)
Using standard diversity index instead.
Recognition Performance Cross-validation by splitting the dataset instances. 5 phenotype categories for every test. Selecting one instance from each category.
Recognition Performance How the parameters of RFs affect the
performance?– Max Tree Depth dmax
– Tree Number NT
Qualitative Results of SVR Left: Input image/silhouette; Centre: Using RF-hypothesizes;Right: Using the optimisation-based approach.
Qualitative Results of SVR
Take-Home MessagesPhenotype recognition is difficult but still
possible;
Combing discriminative and generative cues can greatly speed up the inference;
A divide-and-conquer strategy can help improve the recognition rate.
Future WorkExplore the application on more
complicated poses and more categories.– E.g. Boxing, gardening, other sports, etc.
Data collection;
Automate the silhouette extraction.– E.g. Kinect.
The End
Questions?