MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise...
-
date post
21-Dec-2015 -
Category
Documents
-
view
214 -
download
1
Transcript of MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise...
![Page 1: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/1.jpg)
1 MSRIUniversity of California Berkeley
Recovering Human Body Configurations using Pairwise Constraints between PartsRecovering Human Body Configurations using Pairwise Constraints between Parts
Xiaofeng Ren, Alex Berg, Jitendra MalikXiaofeng Ren, Alex Berg, Jitendra Malik
![Page 2: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/2.jpg)
2 MSRIUniversity of California Berkeley
Finding PeopleFinding People
Challenges: Pose, Clothing, Lighting, Clutter, …
![Page 3: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/3.jpg)
3 MSRIUniversity of California Berkeley
Previous WorkPrevious Work• Related Domains
– Tracking People– Detecting Pedestrians– ... …
• Localizing Human Figures– Exemplar-based:
[Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], …
– Part-based:[Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], …
– … …
• Related Domains– Tracking People– Detecting Pedestrians– ... …
• Localizing Human Figures– Exemplar-based:
[Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], …
– Part-based:[Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], …
– … …
![Page 4: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/4.jpg)
4 MSRIUniversity of California Berkeley
Beyond “Trees”Beyond “Trees”
• A hard problem! More information is needed.
• Important cues that are NOT in the tree model:– Symmetry of clothing/color
– “V-shape” formed by the upper legs
– Distance/smooth connection between arms and legs
– ……
• A hard problem! More information is needed.
• Important cues that are NOT in the tree model:– Symmetry of clothing/color
– “V-shape” formed by the upper legs
– Distance/smooth connection between arms and legs
– ……
?
![Page 5: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/5.jpg)
5 MSRIUniversity of California Berkeley
Our ApproachOur Approach• Preprocessing with Constrained Delaunay Triangulation
• Detecting Candidate Parts from Bottom-up
• Learning Pairwise Constraints between Parts
• Assembling Parts by Integer Quadratic Programming (IQP)
• Preprocessing with Constrained Delaunay Triangulation
• Detecting Candidate Parts from Bottom-up
• Learning Pairwise Constraints between Parts
• Assembling Parts by Integer Quadratic Programming (IQP)
![Page 6: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/6.jpg)
6 MSRIUniversity of California Berkeley
Constrained Delaunay TriangulationConstrained Delaunay Triangulation
– Detect edges with Pb (Probability of Boundary)
– Trace contours with Canny’s hysteresis
– Recursively split contours into piecewise straight lines
– Complete the partial graph with Constrained Delaunay Triangulation
– Detect edges with Pb (Probability of Boundary)
– Trace contours with Canny’s hysteresis
– Recursively split contours into piecewise straight lines
– Complete the partial graph with Constrained Delaunay Triangulation
![Page 7: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/7.jpg)
7 MSRIUniversity of California Berkeley
Detecting Parts using ParallelismDetecting Parts using Parallelism
(L1,1)
(L2,2)T
N
• Candidate parts as parallel line segments (Ebenbreite)
• (Scale-invariant) Features for parallelism:|Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2|
• Logistic Classifier
• Candidate parts as parallel line segments (Ebenbreite)
• (Scale-invariant) Features for parallelism:|Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2|
• Logistic Classifier
C1
C2
![Page 8: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/8.jpg)
8 MSRIUniversity of California Berkeley
Pairwise Constraints between PartsPairwise Constraints between Parts
• Scale (width) consistency– Use anthropometric data as groundtruth
• Symmetry of appearance (color)
• Orientation consistency
• Connectivity– Short distance between adjacent parts
– “Smooth” connection between non-adjacent parts• short “gaps” on shortest path (on CDT graph)
• small maximum angle on the shortest path
• few T-junctions/turns on the shortest path
• Scale (width) consistency– Use anthropometric data as groundtruth
• Symmetry of appearance (color)
• Orientation consistency
• Connectivity– Short distance between adjacent parts
– “Smooth” connection between non-adjacent parts• short “gaps” on shortest path (on CDT graph)
• small maximum angle on the shortest path
• few T-junctions/turns on the shortest path
C1
C2
![Page 9: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/9.jpg)
9 MSRIUniversity of California Berkeley
Learning Pairwise ConstraintsLearning Pairwise Constraints
15 hand-labeled images from a skating sequence
Empirical distributions of some pairwise features
For simplicity, assume all features are Gaussian (future work here as they are clearly non-Gaussian)
![Page 10: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/10.jpg)
10 MSRIUniversity of California Berkeley
Assembling Parts as AssignmentAssembling Parts as Assignment
Candidates {Ci} Parts {Lj}
(Lj1,Ci1=(Lj1))
(Lj2,Ci2=(Lj2))
Cost for a partial assignment {(Lj1,Ci1), (Lj2,Ci2)}:
assignment
2
)22)(11( 2,1
2,1)22(),11(
kijij jj
k
jjk
ijijkf
H
![Page 11: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/11.jpg)
11 MSRIUniversity of California Berkeley
Assignment by IQPAssignment by IQP• Suppose there are m parts and n candidates, the optimal
assignment minimizes a quadratic function• Suppose there are m parts and n candidates, the optimal
assignment minimizes a quadratic function
Q(x)=xTHxQ(x)=xTHxwhere x is a mn1 indicator vector and H is of size mnmn.where x is a mn1 indicator vector and H is of size mnmn.
• This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions.
• We choose an approximation scheme which solves mn linear programs followed by gradient descent.
• The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems.
• We have m=9 and n~150; the total time is less than a minute.
• This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions.
• We choose an approximation scheme which solves mn linear programs followed by gradient descent.
• The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems.
• We have m=9 and n~150; the total time is less than a minute.
![Page 12: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/12.jpg)
12 MSRIUniversity of California Berkeley
![Page 13: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/13.jpg)
13 MSRIUniversity of California Berkeley
![Page 14: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/14.jpg)
14 MSRIUniversity of California Berkeley
![Page 15: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/15.jpg)
15 MSRIUniversity of California Berkeley
ConclusionConclusion
• To find people under general conditions, we need to go beyond the traditional tree-based model;
• Most important constraints for the human body are between pairs of body parts;
• Pairwise constraints may be learned from a small set of training examples;
• Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.
• To find people under general conditions, we need to go beyond the traditional tree-based model;
• Most important constraints for the human body are between pairs of body parts;
• Pairwise constraints may be learned from a small set of training examples;
• Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.
![Page 16: MSRI University of California Berkeley 1 Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d5e5503460f94a3d0eb/html5/thumbnails/16.jpg)
16 MSRIUniversity of California Berkeley