Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera...
Transcript of Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera...
![Page 1: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/1.jpg)
20 – 07 ‐ 2010
Author: Antonio HernándezVela
Supervisors: Sergio EscaleraPetia Radeva
![Page 2: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/2.jpg)
2
![Page 3: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/3.jpg)
IntroductionM h d l Methodology Spatio‐Temporal GrabCut Human segmentaton Active Appearance models Face fitting Conditional Random Field Limb recovery
V lid i Validation Existent public data sets New Human Body Limb data set
Conclusions and future work
3
![Page 4: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/4.jpg)
IntroductionM h d l Methodology Spatio‐Temporal GrabCut Human segmentaton Active Appearance models Face fitting Conditional Random Field Limb recovery
V lid i Validation Existent public data sets New Human Body Limb data set
Conclusions and future work
4
![Page 5: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/5.jpg)
Video oriented
Human segmentation and pose recovery
5
![Page 6: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/6.jpg)
Human segmentation for human pose [1] and face human pose [1] and face recovery [2] assistance
[1] Ramanan, D. "Learning to Parse Images of Articulated Bodies ", In NIPS, 2006.[2] T. Cootes, J.Edwards and C. Taylor, "Active Appearance models.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998. 6
![Page 7: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/7.jpg)
IntroductionM h d l Methodology Spatio‐Temporal GrabCut Human segmentaton Active Appearance models Face fitting Conditional Random Field Limb recovery
V lid i Validation Existent public data sets New Human Body Limb data set
Conclusions and future work
7
![Page 8: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/8.jpg)
Person facefittingfitting
Person poseInitialization STGrabCut
Person poseestimation
8
![Page 9: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/9.jpg)
HOG‐based [4] person detectorSeedinitialization
STGrabCut
PPerson poseestimation
Backgroundseeds
Unknownregion
Person facefitting
9
[4] Navneet Dalal and Bill Triggs, "Histograms of Oriented Gradients for Human Detection" , In CVPR, 2005.
![Page 10: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/10.jpg)
d Face detection [5] Foreground
seedsSeedinitialization
STGrabCut
PPerson poseestimation
Person facefitting
10
[5] Paul Viola and Michael Jones, "Robust real‐time face detection" , International Journal of Computer Vision, 2004.
![Page 11: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/11.jpg)
GrabCutSeed
initialization
Pixel‐levelsegmentation
STGrabCut
GrabCut segmentation
segmentationrefinement
P GrabCut segmentationGrabCut
Person poseestimation
Seed init fornext frame
Person facefitting
11
![Page 12: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/12.jpg)
Sl S Toy example Log‐likelihood of
GMM models(BGD,FGD) over
S RGB
T
Minimum cut == Pi l diffTminimum energy
Ci
iC ||
Pixel differencesbased on RGB
Euclidean distance
12[3] C Rother, V Kolmogorov, A Blake. "Grabcut: Interactive foreground extraction using iterated graph cuts" , ACM Transactions on Graphics, 2004.
Ci
![Page 13: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/13.jpg)
Iterative procedure
Until convergence
GMM Assign pixelstoGMM Segmentation
modelling toGMM components
g(min‐cut)
[3] C Rother, V Kolmogorov, A Blake. "Grabcut: Interactive foreground extraction using iterated graph cuts" , ACM Transactions on Graphics, 2004.
13
![Page 14: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/14.jpg)
GrabCutSeed
initialization
Pixel‐levelsegmentation
STGrabCut
GrabCut segmentation
segmentationrefinement
P GrabCut segmentation Mean‐shift GMM initialization
GrabCutPerson poseestimation
(spatial coherence)Seed init fornext frame
Person facefitting
14
![Page 15: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/15.jpg)
Seedinitialization GrabCut
STGrabCut Pixel‐levelsegmentation
P
segmentationrefinement
Biased to convex regions!Person poseestimation GrabCut
Biased to convex regions!
B k d d f i lPerson face
fittingSeed init fornext frame
Background seeds for pixels:
15
![Page 16: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/16.jpg)
GrabCutSeed
initialization
Pixel‐levelsegmentation
STGrabCutsegmentationrefinement
PGrabCut
GrabCut re‐iterationI f t ti ith
Person poseestimation
Seed init fornext frame
Infer segmentation with new background seeds
Person facefitting
16
![Page 17: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/17.jpg)
GrabCutSeed
initialization
Pixel‐levelsegmentation
STGrabCutsegmentationrefinement
PGrabCut
Initialization for next frame
Person poseestimation
Seed init fornext frame
Person facefitting
17
![Page 18: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/18.jpg)
CRF‐based [1] pose recoverySeedinitialization
STGrabCut
PPerson poseestimation
Person facefitting
18
[1] Ramanan, D. "Learning to Parse Images of Articulated Bodies ", In NIPS, 2006.
![Page 19: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/19.jpg)
AAM‐based [2] face recoverySeedinitialization
STGrabCut
PPerson poseestimation 3 meshes: 1 frontal, 2 lateral
Person facefitting
19
[2] T. Cootes, J.Edwards and C. Taylor, "Active Appearance models.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998.
![Page 20: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/20.jpg)
IntroductionM h d l Methodology Spatio‐Temporal GrabCut Human segmentaton Active Appearance models Face fitting Conditional Random Field Limb recovery
V lid ti Validation Existent public data sets New Human Body Limb data set
Conclusions and future work
20
![Page 21: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/21.jpg)
CVSG data set [6]Vid f Video sequence: 307 frames Ground truth
Undergraduate thesis defenseg 4 video sequences: 720 frames each
21
[6] F. Tiburzi, M. Escudero, J. Bescos, and J. Martinez.”A ground‐truth for motion‐based video‐object segmentation” IEEE International Conference on Image Processing (Workshop on Multimedia Information Retrieval, 2008
![Page 22: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/22.jpg)
New human body limb data set 227 images 25 different people5 p p Ground truth
22http://www.maia.ub.es/~sergio/Code.html
![Page 23: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/23.jpg)
Quantitative results (CVSG data set)
0,8747
0,80,91
0,53560,54240,6229
0,50,60,7
GrabCut
Mean‐shift
pping%
0 20,30,4,5
Morphology
ST‐GrabCut
Ove
rlap
00,10,2
23
![Page 24: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/24.jpg)
Qualitative results (CVSG data set)
24
![Page 25: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/25.jpg)
l l d Qualitative results (UB data set)
25
![Page 26: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/26.jpg)
d Body pose recovery
nmen
tatio
nNo segm
utT‐GrabC
u
26
S
![Page 27: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/27.jpg)
F Face recovery
27
![Page 28: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/28.jpg)
Temporal joint body and face recovery
28
![Page 29: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/29.jpg)
Face and body pose recovery (CVSG)
0 96360,896
0,7919
0,96360,876
0,8
1
0,6
No segmentationpping%
0,4No segmentation
STGrabCut
Ove
rlap
0
0,2
29
Face Body pose
![Page 30: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/30.jpg)
Body pose recovery
200 Trunk
Up‐arms
#overlapping wins
100
150Up arms
Up‐legs
Low‐arms
50Low‐legs
Head
0
STGrabCut*
30
No segmentation* No temporal information is used
![Page 31: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/31.jpg)
IntroductionM h d l Methodology Spatio‐Temporal GrabCut Human segmentaton Active Appearance models Face fitting Conditional Random Field Limb recovery
V lid i Validation Public data sets Human Body Limb data set
Conclusions and future work
31
![Page 32: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/32.jpg)
Extension of GrabCut for human segmentationF ll i h d Fully automatic method Spatio‐temporal coherence Segmentation convexity problem
Face recovery with temporal coherence New human body limb database New human body limb database Human segmentation helps to retrieve face and b dbody pose
32
![Page 33: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/33.jpg)
Oral paper at AMFG Workshop (CVPR Oral paper at AMFG Workshop (CVPR 2010) Antonio Hernández Miguel Reyes Sergio Antonio Hernández, Miguel Reyes, Sergio Escalera and Petia Radeva, “Spatio‐temporal GrabCut HumanSegmentationtemporal GrabCut HumanSegmentationfor Face and Pose Recovery”
Submission to Machine Vision and ApplicationsApplications
33
![Page 34: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/34.jpg)
Include spatio‐temporal coherence inside graphc ts frame orkcuts framework Extended graph for image volumes New spatio‐temporal potential
Improve segmentation using face and pose recovery feedbackrecovery feedback
34
![Page 35: Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:](https://reader034.fdocuments.in/reader034/viewer/2022042920/5f66a550dfc99a12a4181e97/html5/thumbnails/35.jpg)
20 – 07 ‐ 2010
Author: Antonio HernándezVela
Supervisors: Sergio EscaleraPetia Radeva