Slide 1

Real-time Articulated Hand Pose Estimation using Semi-supervised Transductive Regression Forests Tsz-HoYu Danhang Tang T-KKim Sponsored by Slide 2 2 Slide 3 Motivation Multiple cameras with invserse kinematics [Bissacco et al. CVPR2007] [Yao et al. IJCV2012] [Sigal IJCV2011] Learning-based (regression) [Navaratnam et al. BMVC2006] [Andriluka et al. CVPR2010] Specialized hardware (e.g. structured light sensor, TOF camera) [ Shotton et al. CVPR11] [Baak et al. ICCV2011] [Ye et al. CVPR2011] [Sun et al. CVPR2012] Slide 4 Motivation Discriminative approaches (RF) have achieved great success in human body pose estimation. Efficient real-time Accurate frame-basis, not rely on tracking Require a large dataset to cover many poses Train on synthetic, test on real data Didnt exploit kinematic constraints Examples: Shotton et al. CVPR11, Girshick et al. ICCV11, Sun et al. CVPR12 Slide 5 Challenges for Hand? Labeling is difficult and tedious! Viewpoint changes and self occlusions Discrepancy between synthetic and real data is larger than human body Slide 6 Our method Hierarchical Hybrid Forest Transductive Learning Semi- supervised Learning Labeling is difficult and tedious! Viewpoint changes and self occlusions Discrepancy between synthetic and real data is larger than human body Slide 7 Existing Approaches Generative approaches Model-fitting No training is required Oikonomidis et al. ICCV2011 De La Gorce et al. PAMI2010 Hamer et al. ICCV2009 Motion capture Ballan et al. ECCV 2012 Slow Needs initialisation and tracking Discriminative approaches Similar solutions to human body pose estimation Performance on real data remains challenging Wang et al. SIGGRAPH2009 Stenger et al. IVC 2007 Keskin et al. ECCV2012 Discriminative approaches Similar solutions to human body pose estimation Performance on real data remains challenging Xu and Cheng ICCV 2013 Slide 8 Our method Hierarchical Hybrid Forest Labeling is difficult and tedious! Viewpoint changes and self occlusions Discrepancy between synthetic and real data is larger than human body Slide 9 Hierarchical Hybrid Forest STR forest: Qa View point classification quality (Information gain) Viewpoint Classification: Qa Q apv = Q a + (1-)Q P + (1-)(1-)Q V Slide 10 Hierarchical Hybrid Forest STR forest: Qa View point classification quality (Information gain) Qp Joint label classification quality (Information gain) Viewpoint Classification: Qa Finger joint Classification: Qp Q apv = Q a + (1-)Q P + (1-)(1-)Q V Slide 11 Hierarchical Hybrid Forest STR forest: Qa View point classification quality (Information gain) Qp Joint label classification quality (Information gain) Qv Compactness of voting vectors (Determinant of covariance trace) Viewpoint Classification: Qa Finger joint Classification: Qp Pose Regression: Qv Q apv = Q a + (1-)Q P + (1-)(1-)Q V Slide 12 Hierarchical Hybrid Forest STR forest: Qa View point classification quality (Information gain) Qp Joint label classification quality (Information gain) Qv Compactness of voting vectors (Determinant of covariance trace) (,) Margin measures of view point labels and joint labels Viewpoint Classification: Qa Finger Joint Classification: Qp Pose Regression: Qv Q apv = Q a + (1-)Q P + (1-)(1-)Q V Slide 13 Our method Transductive Learning Semi- supervised Learning Labeling is difficult and tedious! Viewpoint changes and self occlusions Discrepancy between synthetic and real data is larger than human body Slide 14 Transductive learning Training data D = {R l, R u, S}: labeled unlabeled Target space (Realistic data R) Realistic data R: Captured from Primesense depth sensor A small part of R, R l are labeled manually (unlabeled set R u ) Source space (Synthetic data S ) Synthetic data S: Generated from an articulated hand model. All labeled. Slide 15 Transductive learning Training data D = {R l, R u, S}: Realistic data R: Captured from Kinect A small part of R, R l are labeled manually (unlabeled set R u ) Synthetic data S: Generated from a articulated hand model, where |S| >> |R| Source space (Synthetic data S ) Target space (Realistic data R) Slide 16 Transductive learning Training data D = {R l, R u, S}: Similar data-points in R l and S are paired(if separated by split function give penalty) Source space (Synthetic data S ) Target space (Realistic data R) Slide 17 Semi-supervised learning Training data D = {R l, R u, S}: Similar data-points in R l and S are paired(if separated by split function give penalty) Introduce a semi-supervised term to make use of unlabeled real data when evaluating split function Source space (Synthetic data S ) Target space (Realistic data R) Slide 18 Kinematic refinement Slide 19 Experiment settings 19 Evaluation data: Three different testing sequences 1.Sequence A --- Single viewpoint(450 frames) 2.Sequence B --- Multiple viewpoints, with slow hand movements(1000 frames) 3.Sequence C --- Multiple viewpoints, with fast hand movements(240 frames) Training data: Synthetic data(337.5K images) Real data(81K images,