1. Semi-Supervised Learning in Vision Amir Saari, Christian Leistner, Horst BischofInstitute for Computer Graphics and Vision, Graz University of Technology CVPR San Francisco, June 18, 2010

2. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedOutline1 Semi-Supervised Learning2 Self-Training3 Generative Models4 Margin Assumption5 Cluster and Manifold Assumption6 Multi-View Learning7 Large-Scale, Multi-Class SSL, and Online Learning8 Related Topics Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 3. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedSupervised and Semi-Supervised LearningSupervised learning is all about nding mappings from input (feature)space to output space:f :X YGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 4. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedSupervised and Semi-Supervised LearningSupervised learning is all about nding mappings from input (feature)space to output space:f :X YGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 5. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedSupervised and Semi-Supervised LearningSupervised learning is all about nding mappings from input (feature)space to output space:f (x; ) F : X YGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 6. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedSupervised and Semi-Supervised LearningIn Semi-supervised learning we wish to nd mappings by using bothlabeled and unlabeled data:f (x; ) F : X YGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 7. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhy Should SSL Make Sense?Learnerf (x; ) F : X YLabeled dataDl = {(x, y) X Y }Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 8. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhy Should SSL Make Sense?Learnerf (x; ) F : X YLabeled dataDl = {(x, y) X Y }Unlabeled data Du = {x X }Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 9. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhy Should SSL Make Sense?Learnerf (x; ) F : X YLabeled dataDl = {(x, y) X Y }Unlabeled data Du = {x X }Unsupervised learning: the goal is to recover the structure of thedata, eg. clusters, manifolds, low dimensional embeddings, density ...Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 10. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhy Should SSL Make Sense?Learnerf (x; ) F : X YLabeled dataDl = {(x, y) X Y }Unlabeled data Du = {x X }Unsupervised learning: the goal is to recover the structure of thedata, eg. clusters, manifolds, low dimensional embeddings, density ...Density p(x)Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 11. SSLSelf-TrainingGenerative Models Margin Manifold Multi-ViewLarge-ScaleRelatedWhen unlabeled data is going to help?Bayes rule: p(y) p(x|y) prior likelihoodp(y|x) =p(x)posterior evidenceGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 12. SSLSelf-TrainingGenerative Models Margin ManifoldMulti-View Large-ScaleRelatedWhen unlabeled data is going to help?Bayes rule: p(y) p(x|y) prior likelihood p(y|x) =p(x)posterior evidenceMAP decision rule:p(k) p(x|k) y =arg max p(k |x) = arg max =kk p(x)=arg max p(k) p(x|k)kGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 13. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhen unlabeled data is going to help?When we expect that p(x) (structure of the data) is related tothe p(y|x), ie. they share parameters. Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 14. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhen unlabeled data is going to help?When we expect that p(x) (structure of the data) is related tothe p(y|x), ie. they share parameters.In other words, a better estimation of p(x) can improve theestimation of p(y|x). Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 15. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhen unlabeled data is going to help?When we expect that p(x) (structure of the data) is related tothe p(y|x), ie. they share parameters.In other words, a better estimation of p(x) can improve theestimation of p(y|x). Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 16. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhen unlabeled data is going to help?When we expect that p(x) (structure of the data) is related tothe p(y|x), ie. they share parameters.In other words, a better estimation of p(x) can improve theestimation of p(y|x). Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 17. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedWhen unlabeled data is going to help?When we expect that p(x) (structure of the data) is related tothe p(y|x), ie. they share parameters.In other words, a better estimation of p(x) can improve theestimation of p(y|x). Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 18. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedSemi-Supervised Learning AssumptionsSemi-supervised learning often is eective, if the assumptionsregarding the relationship between the structure of the data p(x) andthe posterior p(y|x) are true for a given problem.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 19. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedComputer Vision Problems: Object RecognitionStructure: similar images may contain similar objects.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 20. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedComputer Vision Problems: Object DetectionStructure: similar image patches may contain similar objects. Veryclose patches (over the 2D image neighborhood) may contain thesame object.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 21. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedComputer Vision Problems: Object TrackingStructure: similar image patches may contain similar objects. Veryclose patches (over the 2D image neighborhood and time) maycontain the same object.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 22. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedComputer Vision Problems: Object SegmentationStructure: similar pixels may correspond to the same object. Veryclose pixels (over the 2D image neighborhood and time) may belongto the same object.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 23. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedDierent Semi-Supervised Learning SettingsSemi-supervised classication: f : X Y , Y = {1, , K }. Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 24. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedDierent Semi-Supervised Learning SettingsSemi-supervised classication: f : X Y , Y = {1, , K }.Semi-supervised regression: f : X Y , Y = R. Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 25. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedDierent Semi-Supervised Learning SettingsSemi-supervised classication: f : X Y , Y = {1, , K }.Semi-supervised regression: f : X Y , Y = R.Semi-supervised clustering: constrainted clustering, clusteringwith pair-wise must-link and cannot-link constraints. Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 26. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedTransductive and Inductive SSLTransductive: nd f : Dl Du Y |Dl Du | . Can not be used forany future example which was not in the training set.Inductive: nd f : X Y . Can be used for any future example,beyond the training set. Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 27. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedInteractive Segmentation Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 28. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedOther Uses of Unlabeled Data Unsupervised preprocessing: normalization, standardization, PCA, ICA, ...Graz University of Technology[Torralba ICCV 2009, Mobahi et al. ICML 2009, Ranzato NIPS 2007]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 29. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedOther Uses of Unlabeled Data Unsupervised preprocessing: normalization, standardization, PCA, ICA, ... Feature extraction: bag-of-wordsGraz University of Technology[Torralba ICCV 2009, Mobahi et al. ICML 2009, Ranzato NIPS 2007]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 30. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedOther Uses of Unlabeled Data Unsupervised preprocessing: normalization, standardization, PCA, ICA, ... Feature extraction: bag-of-words Unsupervised feature learning: deep learning and sparse codingGraz University of Technology[Torralba ICCV 2009, Mobahi et al. ICML 2009, Ranzato NIPS 2007]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 31. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedThe Simplest Approach to SSLSelf-training is a meta learning (wrapper) semi-supervised method.Self-Training Inputs: learning algorithm T, labeled set Dl , and unlabeled set Du . For n = 1 to N: 1 Train using the labeled set: f n = T (Dl ). 2 Use f n to classify the unlabeled set: cu = f n (Du ). 3 Create the set of m most condent examples from the unlabeled set: C Du . 4 Update the labeled set: Dl Dl {(x, f n (x ))| x C}. 5 Update the unlabeled set: Du Du C .Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 32. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: 1-NN ClassierGraz University of Technology[Zhu TPCL 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 33. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: 1-NN ClassierGraz University of Technology[Zhu TPCL 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 34. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with RF Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 35. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Self-Training with RF (Iteration 5) Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 36. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Self-Training with RF (Iteration 10) Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 37. SSLSelf-TrainingGenerative Models Margin Manifold Multi-ViewLarge-ScaleRelatedGenerative ModelsJoint Distribution p(x, y|, ) = p(x|y; )p(y| )mixture componentyGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 38. SSLSelf-TrainingGenerative Models Margin Manifold Multi-ViewLarge-ScaleRelatedGenerative ModelsJoint Distribution p(x, y|, ) = p(x|y; )p(y| )mixture componentyPosteriory p(x|y; ) p(y|x; , ) = k k p(x|k; )Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 39. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExpectation Maximization ApproachLog-likelihood (, ; Dl , Du ) =log y p(x|y; ) + log k p(x|k; ) (x,y)Dl x Du klabeled data unlabeled dataGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 40. SSLSelf-TrainingGenerative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedExpectation Maximization ApproachLog-likelihood (, ; Dl , Du ) =log y p(x|y; ) +log k p(x|k; ) (x,y)Dlx Du k labeled dataunlabeled dataExpectation Maximization (EM) can be used to estimate the modelparameters.EM Inputs: labeled set Dl , and unlabeled set Du , initial model parameters 0 , 0 . For n = 1 to N: 1 E step: Estimate the posterior p (y | x; n1 , n1 ) for Du . 2 M step: Update the model parameters, given the posteriors for unlabeled data: n , n = arg max (, ; Dl , Du ). , Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 41. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Two Gaussians Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 42. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Supervised GMM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 43. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Semi-Supervised EM with GMM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 44. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedAssumptions[Jaakkola et al.] Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 45. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedAssumptions[Jaakkola et al.] Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 46. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLinear Classier Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 47. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMaximum Margin Linear Classier Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 48. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedSVM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 49. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedS3VM and TSVM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 50. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedS3VMSVM with soft margins: binary case y {1, 1} without the biasterm 1 min w 2 + w 22 max(0, 1 y w, x )|Dl | (x,y)Dl margin hinge lossGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 51. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedS3VMSVM with soft margins: binary case y {1, 1} without the biasterm 1 min w 2 + w 22 max(0, 1 y w, x )|Dl | (x,y)Dl margin hinge lossPrediction rule:y = sign( w, x )Margin of an unlabeled data:mu (w, x) = | w, x |Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 52. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedS3VM: Loss Functions Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 53. SSL Self-Training Generative ModelsMarginManifoldMulti-ViewLarge-ScaleRelatedS3VM: FormulationS3VM 1 minw 2w2 2 + max(0, 1 y w, x ) + |Dl | (x,y)D l margin hinge loss + max(0, 1 | w, x |) |Du | xDu sym. hinge Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 54. SSLSelf-Training Generative ModelsMarginManifoldMulti-ViewLarge-ScaleRelatedS3VM: FormulationS3VM 1min w 2 w22 + max(0, 1 y w, x ) +|Dl | (x,y)Dlmarginhinge loss + max(0, 1 | w, x |)|Du | xDusym. hingeBalancing constraint: 11 y = |Du | w, x ||Dl | (x,y)D lxDu[Vapnik 1998, Joachims ICML 1999] Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 55. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedEntropy Regularization Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 56. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Linear SVM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 57. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Linear TSVM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 58. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Linear TSVMGraz University of Technology[Zhu TPCL 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 59. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster Assumption and Margin Maximization Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 60. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster KernelsLinear SVM: f (x) = w, xGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 61. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster KernelsLinear SVM: f (x) = w, xNonlinear SVM:f (x ) = w, (x ) = y x (x), (x )(x,y)Dl nonlinear mappingK (x,x )Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 62. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster KernelsLinear SVM: f (x) = w, xNonlinear SVM:f (x ) = w, (x ) = y x (x), (x )(x,y)Dl nonlinear mappingK (x,x )Cluster Kernel: p I(c(x) == c(x )) Kc (x, x ) = K (x, x ) n[Weston et al. NIPS 2003]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 63. SSLSelf-TrainingGenerative Models MarginManifold Multi-ViewLarge-ScaleRelatedBoostingBoosting Mf (x) = wm g(x; m )m =1 Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 64. SSLSelf-TrainingGenerative Models MarginManifold Multi-ViewLarge-ScaleRelatedBoostingBoosting Mf (x) = wm g(x; m )m =1Base Learner g(x; m ) G : X Y Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 65. SSLSelf-TrainingGenerative Models MarginManifold Multi-ViewLarge-ScaleRelatedBoostingBoosting Mf (x) = wm g(x; m )m =1Base Learner g(x; m ) G : X YBoosting is a linear classier over the space of base learners G : f (x) = G (x; )w Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 66. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedBoosting: Learning with Functional Gradient Descent M1f (x; ) = arg min (x, y; f ) f (x; ) = wm g(x; m ) |Xl | (x,y)X l m =1Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 67. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedBoosting: Learning with Functional Gradient Descent M1f (x; ) = arg min (x, y; f ) f (x; ) = wm g(x; m ) |Xl | (x,y)X l m =1Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 68. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedBoosting with PriorPriorsx Du , k Y : q(k |x) 1minw, |Dl | (x, y; w, ) +(x,y)Dl Supervised |Du | x u + p (x, q; w, )DPrior[Saari et al. ECCV 2008, CVPR 2009, ECCV 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 69. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster PriorGraz University of Technology[Saari et al. CVPR 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 70. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster PriorGraz University of Technology[Saari et al. CVPR 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 71. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCluster PriorGraz University of Technology[Saari et al. CVPR 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 72. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: SSL Boosting with Cluster Prior0.60VOC20060.550.50Class. Acc.0.450.400.35 RMSB SER0.30 AML SVM TSVM0.25 0.00.10.2 0.30.40.5 Labeled Samp. Ratio, rGraz University of Technology[Saari et al. CVPR 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 73. SSLSelf-Training Generative Models Margin Manifold Multi-ViewLarge-ScaleRelatedExample: SSL Boosting with Cluster Prior 10000 Computation Time r =0.1 r =0.5 8000Time (sec) 6000 4000 20000RMSB SER TSVM Graz University of Technology[Saari et al. CVPR 2009]Amir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 74. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Boosting Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 75. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Cluster Prior Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 76. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedManifold Assumption[Hein and von Luxburg MLSS 2007]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 77. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedManifold Assumption[Hein and von Luxburg MLSS 2007]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 78. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedManifold Assumption[Hein and von Luxburg MLSS 2007]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 79. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLabel Propagation: Supervised[Zhu TPCL 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 80. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLabel Propagation: Semi-Supervised[Zhu TPCL 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 81. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLabel Propagation: Semi-Supervised[Kveton et al. CVPR OLCV 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 82. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLabel Propagation: Semi-Supervised[Kveton et al. CVPR OLCV 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 83. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMincut Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 84. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMincutMincut energy:min {y x } xD (x ,y )D s(x, x )(yx y )2 + s(x, x )(yx yx )2 ulx DuGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 85. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedRandom Walks on GraphGraz University of Technology[Zhu TPCL 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 86. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedManifold Regularization: LapSVMLapSVM 1minw2 w22 + max(0, 1 y w, x )+|Dl | (x,y)D l+|Du |2 s(x, x )( w, x w, x )2 xDu x Dl Du Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 87. SSLSelf-Training Generative ModelsMarginManifoldMulti-ViewLarge-ScaleRelatedManifold Regularization: LapSVMLapSVM 1 minw2 w22 + max(0, 1 y w, x )+|Dl | (x,y)Dl+|Du |2 s(x, x )( w, x w, x )2 xDu x Dl DuGraph Laplacian 1 min w 2 w 2 2 + max(0, 1 y w, x )+ |Dl | (x,y)D l +f, Lf |Du |2where f is the response vector of the classier to both labeled andunlabeled data.Graz University of Technology[Belkin et al. JMLR 2006]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 88. SSL Self-TrainingGenerative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedBoosting with Priors and Manifolds1minw, (x, y; w, ) + |Dl | (x,y)D l Superviseds(x, x ) + p (x, q; w, ) +(1 ) z(x) m (x, x ; w, ) |Du | xDux Du Priorx =xManifoldGraz University of TechnologyUnsupervised Amir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 89. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Boosting with Cluster and Manifold PriorsMethods-Dataset g241cg241d Digit1 USPS COIL BCI1-NN44.0543.22 23.4719.8265.91 48.74SVM 47.3246.66 30.6020.0368.36 49.85RF (weak) 47.5148.44 42.4222.9075.72 49.35GBoost-RF 46.7746.61 38.7020.8969.85 49.12TSVM24.7150.08 17.7725.2067.50 49.15Cluster Kernel48.2842.05 18.7319.4167.32 48.31LapSVM46.2145.15 08.9719.05N/A 49.25ManifoldBoost 42.1742.80 19.4219.97N/A 47.12SemiBoost-RF48.4147.19 10.5715.8363.39 49.77MCSSB-RF49.7748.57 38.5022.9569.96 49.12GPMBoost-RF 15.6939.45 11.1914.9262.60 49.27[Saari et al. 2010] Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 90. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Boosting with Cluster and Manifold PriorsMethods-Dataset g241cg241d Digit1 USPS COIL BCI1-NN40.2837.49 06.1207.6423.27 44.83SVM 23.1124.64 05.5309.7522.93 34.31RF (weak) 44.2345.02 17.8016.7334.26 43.78GBoost-RF 31.8432.38 06.2413.8121.88 40.08TSVM18.4622.42 06.1509.7725.80 33.25Cluster Kernel13.4904.95 03.7909.6821.99 35.17LapSVM23.8226.36 03.1304.70N/A 32.39ManifoldBoost 22.8725.00 04.2906.65N/A 32.17SemiBoost-RF41.2639.14 02.5605.9215.31 47.12MCSSB-RF45.1140.26 13.1912.3131.09 47.64GPMBoost-RF 12.8012.59 02.3506.3314.49 45.41[Saari et al. 2010] Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 91. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Cluster Prior Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 92. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Cluster Prior and Manifolds Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 93. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLearning from Dierent ViewsData is represented by multiple views: x = [x1 | |xV ] T T T Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 94. SSLSelf-TrainingGenerative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedLearning from Dierent Views Data is represented by multiple views: x = [x1 | |xV ] TT T There is a classier per view: F = { f v }V=1 vMulti-View LearningF = arg min (x, y; F ) + (x; F )F (x,y)Dl xDu Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 95. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCo-TrainingCo-TrainingInputs: learning algorithm T, labeled set Dl , and unlabeled setDu .For n = 1 to N: 1 1 21 Train using the labeled set: f n = T (Dl ) and f n = T (Dl ).22 Use f n1 and f 2 to classify the unlabeled set: c1 = f 1 (D 1 ) and n un u c2 = f n (Du ).u2 2 3 Create the set of m most condent examples from each view of the unlabeled set: C 1 Du andC 2 Du .12 4 Update the labeled set: Dl Dl {(x, f n (x))| x C 1 } {(x, f n (x))| x C 2 }. 1 2 5 Update the unlabeled set: Du Du (C 1 C 2 ).[Blum and Mitchell COLT 1998]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 96. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCo-Training and AssumptionsWe can represent the data in two views: x = [x1 , x2 ]. Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 97. SSLSelf-TrainingGenerative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedCo-Training and Assumptions We can represent the data in two views: x = [x1 , x2 ]. We can train a good classier only using either x1 or x2 .[Blum and Mitchell COLT 1998, Sindhwani et al. ICML 2005] Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 98. SSLSelf-TrainingGenerative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedCo-Training and Assumptions We can represent the data in two views: x = [x1 , x2 ]. We can train a good classier only using either x1 or x2 . x1 and x2 are conditionally independent given the class.[Blum and Mitchell COLT 1998, Sindhwani et al. ICML 2005] Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 99. SSLSelf-TrainingGenerative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedCo-Training and Assumptions We can represent the data in two views: x = [x1 , x2 ]. We can train a good classier only using either x1 or x2 . x1 and x2 are conditionally independent given the class.[Blum and Mitchell COLT 1998, Sindhwani et al. ICML 2005] Graz University of TechnologyAmir Saari, Christian Leistner, Horst BischofSemi-Supervised Learning in Vision 100. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCo-Training: Visual and EEG DataGraz University of Technology[Kapoor et al. CVPR 2008]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 101. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedCo-Training: Visual and EEG Data[Kapoor et al. CVPR 2008]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 102. SSLSelf-Training Generative ModelsMarginManifoldMulti-ViewLarge-ScaleRelatedCo-Training: Visual and EEG DataGraz University of Technology[Kapoor et al. CVPR 2008]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 103. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-View Boosting with PriorsMulti-View Priors 1 V 1 sqv (k |xv ) =ps (k |xs ), v {1, , V }, k Y=vGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 104. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-View Boosting with PriorsMulti-View Priors 1 V 1 s qv (k |xv ) = ps (k |xs ), v {1, , V }, k Y=vBoosting with Priors: 1 arg min (x, y; w, ) +|Dl | (x,y)Dw, lSupervised + p (x, q; w, )|Du | xDu Prior[Saari et al. ECCV 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 105. SSL Self-Training Generative Models Margin ManifoldMulti-ViewLarge-ScaleRelatedRobust Loss Functions q + =0.555 0-1 KL Hinge SKL4 Exponential 4 JS Logit Savage33(x,y;f)(x,q;f)22110 4202 4 0 4 2 0 2 4 m(x,y;f) f + (x) q + =0.75 5 KL SKL 4 JS 3(x,q;f) 2 1 0 4 2 0 2 4f + (x) Graz University of Technology[Saari et al. ECCV 2010] Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 106. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with RF Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 107. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Linear SVM Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 108. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Multi-ViewClassiers Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 109. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedExample: Interactive Segmentation with Multi-ViewClassiers Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 110. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLarge-Scale Applications Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 111. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLarge-Scale Applications Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 112. SSL Self-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLarge-Scale Applications Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 113. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLearning from Gigantic Image CollectionsOriginal Objective J (f) = (f y)T A(f y) + f T LfObjective with eigenvectors using f = UJ () = (U y)T A(U y) + T TGraz University of Technology[Fergus et al. NIPS 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 114. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLearning from Gigantic Image Collections80 million tiny images[Torralba PAMI 2008, Fergus et al. NIPS 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 115. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLearning from Gigantic Image CollectionsGraz University of Technology[Fergus et al. NIPS 2009]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 116. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedDeep LearningEmbedding loss ( f ( x ), y ) + s(x, x ) ( f (x), f (x )) (x,y)Dl xDu x DGraz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 117. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedDeep LearningSemantic Role Labeling:The cat eats the sh in the pond TheARG0 catARG0 eatsREL theARG1 shARG1 inARGM-LOCtheARGM-LOC pondARGM-LOC Trained with stochastic gradient descent on 1 million labeled dataand 631 million unlabeled data from Wikipedia.Graz University of Technology[Weston et al. ICML 2008]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 118. SSLSelf-TrainingGenerative Models Margin Manifold Multi-ViewLarge-ScaleRelatedOnline Learning and SSLMore about online semi-supervised learning in the second part of thetutorial.[Goldberg et al. ECML 2009, Zeisl et al. CVPR 2010, Leistner et al. PR 2010, Saari et al. ECCV 2010, Kveton CVPROLCV 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 119. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-Class ProblemsMany interesting problems are multi-class.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 120. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-Class ProblemsWhat is wrong with 1-vs-all?Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 121. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-Class ProblemsWhat is wrong with 1-vs-all? Computational problems.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 122. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-Class ProblemsWhat is wrong with 1-vs-all? Computational problems. Calibration problems [B. Schoelkopf, A. Smola, 2002].Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 123. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedMulti-Class ProblemsWhat is wrong with 1-vs-all? Computational problems. Calibration problems [B. Schoelkopf, A. Smola, 2002]. Articial unbalanced binary problems.Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 124. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedBoosting with Priors and ManifoldsBoosting with Priors and Manifolds is an inherently multi-classalgorithm.Graz University of Technology[Saari et al. CVPR 2009, CVPR 2010, ECCV 2010]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 125. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedTransfer Learning[Pan and Yang TKDE 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 126. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedTransfer Learning[Pan and Yang TKDE 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 127. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedTransfer Learning with Attributes[Farhadi CVPR 2009, Lampert CVPR 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 128. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedTransfer Learning with AttributesGraz University of Technology[Farhadi et al. CVPR 2010]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 129. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedTransfer Learning with Attributes[Rohbach et al. CVPR 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 130. SSLSelf-Training Generative ModelsMarginManifoldMulti-ViewLarge-ScaleRelatedLearning from Latent (Hidden) VariablesLatent SVM f w (x) = max w, (x, z) z 1 min w 2 w 2 2 + max(0, 1 y f w (x)) |Dl | (x,y)DlGraz University of Technology[Felzenszwalb et al. CVPR 2008]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 131. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedIndoor Scene Labeling[Wang et al. ECCV 2010]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 132. SSLSelf-Training Generative ModelsMarginManifoldMulti-ViewLarge-ScaleRelatedLatent Structural SVM[Yu and Joachims 2009]Graz University of TechnologyAmir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 133. SSLSelf-TrainingGenerative Models MarginManifoldMulti-ViewLarge-ScaleRelatedLatent Structural SVM for Indoor Scene LabelingGraz University of Technology[Wang et al. ECCV 2010]Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision 134. SSL Self-TrainingGenerative Models MarginManifold Multi-View Large-ScaleRelatedMultiple Instance Learning--- -+ -- + ----+- Graz University of Technology Amir Saari, Christian Leistner, Horst Bischof Semi-Supervised Learning in Vision