Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and...
Transcript of Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and...
![Page 1: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/1.jpg)
Object Detection by 3D Aspectlets and Occlusion Reasoning
Yu Xiang
University of Michigan
Silvio Savarese
Stanford University
In the 4th International IEEE Workshop on 3D Representation and Recognition (3dRR), 2013.
![Page 2: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/2.jpg)
Occlusions in Object Detection
test image
Occlusion changes the appearances of objects.
training images
?
2 car mixture detector
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 3: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/3.jpg)
Object Context for Occlusion Reasoning
test image
Consider all the objects in the scene jointly by estimating their 3D spatial layout.
3D spatial layout
3 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 4: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/4.jpg)
3D Parts for Handling Occlusion
4
test image
3D Parts provides evidences of partial observations from different views.
3D object detector
training images
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 5: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/5.jpg)
Our Method
5 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 6: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/6.jpg)
Our Method
• Top-down occlusion reasoning by contextualizing objects in 3D
• Bottom-up evidences provided by part-based 3D object detectors (3D Aspectlets).
6 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 7: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/7.jpg)
Outline • Related work
• 3D aspectlets
• Spatial layout model
• Experiments
• Conclusion
7 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 8: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/8.jpg)
Outline • Related work
• 3D aspectlets
• Spatial layout model
• Experiments
• Conclusion
8 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 9: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/9.jpg)
Related Work: 3D Object Detection • Use 3D models, learn object appearances from
training images for robust 2D matching
• Hard to handle complicated scenes with occlusions and truncations
From Liebelt et al. CVPR’08
From Pepik et al. CVPR’12 From Fidler et al., NIPS’12 9
From Xiang & Savarese, CVPR’12
![Page 10: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/10.jpg)
Related Work: Object Context for Object Detection
Hoiem et al. use 3D scene geometry, CVPR’06
Desai et al. use object co-occurrences, ICCV’09
Hedau et al. use room layout, ECCV’10 10
Choi et al. use geometry phases, CVPR’13
![Page 11: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/11.jpg)
Related work: 2D Occlusion Reasoning
HOG-LBP human detector, Wang et al. ICCV’09
Segmentation-aware detector, Gao et al. CVPR’11
11
Occlusion boundaries recovery, Hoiem et al. ICCV’07
Occlusion Patterns, Pepik et al. CVPR’13 Occlusion masks, Zia et al. CVPR’13
![Page 12: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/12.jpg)
Related work: 3D occlusion reasoning in object detection
Wojek et al. CVPR’11
Difference: Instead of a simpilifed 2.5D structure of depth layers, we handle occlusion using a true 3D representation of object.
Wu and Nevatia, ICCV’05
12
![Page 13: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/13.jpg)
Outline • Related work
• 3D aspectlets
• Spatial layout model
• Experiments
• Conclusion
13 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 14: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/14.jpg)
3D Aspectlet • Aspect part [Xiang & Savarese, CVPR’12] is
good at handling self-occlusion, but not good for occlusion between objects
[Xiang & Savarese, CVPR’12] 14
![Page 15: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/15.jpg)
15
3D Aspectlet • Atomic aspect part for handling occlusion
[Xiang & Savarese, 3dRR’13]
![Page 16: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/16.jpg)
3D Aspectlets • Atomic aspect parts are hard to detect, group
them to form “bigger parts” – 3D aspectlets
Geometrically close to each other in 3D
Discriminative
Yu Xiang and Silvio Savarese. 3dRR'13 16
![Page 17: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/17.jpg)
17
3D Aspectlets
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 18: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/18.jpg)
18
3D Aspectlets • Each 3D aspectlet is modeled by a two
level tree structure as in Aspect Layout Model [Xiang & Savarese, CVPR’12]
p1 p2 p3
a
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 19: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/19.jpg)
Outline • Related work
• 3D aspectlets
• Spatial layout model
• Experiments
• Conclusion
19 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 20: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/20.jpg)
20
Spatial Layout Model
• Posterior distribution
1 ( , )
( ) ( ) ( | , , ) ( , | , , )M
i i j
i i j
P C P P o C I P o o C I
O O O
camera prior 3D objects prior
unary 2D projection likelihood
pairwise 2D projection likelihood
a single input image
( , , | )P C Io O
2D projections 3D objects camera
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 21: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/21.jpg)
21
Spatial Layout Model
• Camera prior
Virtual intrinsic camera matrix
( ) ( ) ( ) ( )P C P a P e P d
O1
O2
O3
anchor object
image
d
a e X
Y
Z camera
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 22: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/22.jpg)
22
Spatial Layout Model
• 3D objects prior
“ground plane” constraint
3D space constraint
1 2
1 ( , )
( ) exp ( ) ( , )M
i i j
i i j
P V O V O O
O
2
1 2( )
2
ii
ZV O
2 ( , )i j
i j
i j
O OV O O
O O
O1
O2
O3 X
Y
Z
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 23: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/23.jpg)
23
Spatial Layout Model
• Unary 2D projection likelihood
likelihood from the full-object model
likelihood from the kth 3D aspectlet
weights proportional to the number of visible atomic aspect parts
a1 a2 a3
r
p1 p2 p3 p4
p1 p2 p3
a
( | , , )iP o C IO 0( | , , )i iP o O C I1
N
k
( | , , )k i iP o O C I( , )kw CO
1
s.t. ( , ) 1N
k
k
w C
O
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 24: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/24.jpg)
24
Spatial Layout Model
• Pairwise 2D projection likelihood
Penalizes wrong occlusion order
Reduces false alarms
( | , , )( , | , , ) exp
( | , , )
if occludes and ( | , , )
j
i j
i
i j i
P o C IP o o C I
P o C I
O O P o C I threshold
OO
O
O
Yu Xiang and Silvio Savarese. 3dRR'13
jO
iO
![Page 25: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/25.jpg)
25
Spatial Layout Model
• Training
Unsupervised learning for selecting 3D aspectlets
Structural SVM for parameter estimation of 3D aspectlets
• Inference
RJMCMC sampling
Object hypotheses from unary 2D projection likelihood without occlusion reasoning
Add moves, delete moves, switch moves
Log-odds ratios from MAP as 2D detection scores
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 26: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/26.jpg)
Outline • Related work
• 3D aspectlets
• Spatial layout model
• Experiments
• Conclusion
26 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 27: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/27.jpg)
Training Datasets • Car: 3DObject Dataset [Savarese & Fei-Fei, ICCV’07]
• Bed, Chair, Sofa and Table: Subset of ImageNet Dataset [Xiang & Savarese, CVPR’12]
Yu Xiang and Silvio Savarese. 3dRR'13 27
![Page 28: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/28.jpg)
Test Datasets • Two new datasets with occlusion (online)
– An outdoor-scene dataset with cars (200 images)
– An indoor-scene dataset with beds, chairs, sofas and tables (300 images)
Category Car Bed Chair Sofa Table
#objects 659 202 235 273 222
#occluded 235 81 112 175 61
#truncated 135 86 41 99 80
28 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 29: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/29.jpg)
29
Detection APs
[1] Y. Xiang and S. Savarese. Estimating the aspect layout of object categories. In CVPR, 2012. [2] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 2010.
Category Car Bed Chair Sofa Table
ALM [1] 46.6 28.9 14.2 41.1 19.2
DPM [2] 57.0 34.8 14.4 38.3 15.1
SLM Aspectlets 59.2 35.8 15.9 45.5 24.3
SLM Full 63.0 39.1 19.0 48.6 28.6
Yu Xiang and Silvio Savarese. 3dRR'13
SLM Aspectlets: using 3D aspectlets in Hough voting without occlusion reasoning SLM Full: our full model using 3D aspectlets and occlusion reasoning
![Page 30: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/30.jpg)
30
Detection APs
Dataset Outdoor-scene Indoor-scene
% occlusion < 0.3 0.3 – 0.6 > 0.6 <0.2 0.2-0.4 >0.4
# images 66 68 66 77 111 112
ALM [1] 72.3 42.9 35.5 38.5 25.0 20.2
DPM [2] 75.9 58.6 44.6 38.0 22.9 21.9
SLM Aspectlets 78.7 59.7 47.7 41.9 30.8 24.8
SLM Full 80.2 63.3 52.9 45.9 34.5 28.0
Yu Xiang and Silvio Savarese. 3dRR'13
[1] Y. Xiang and S. Savarese. Estimating the aspect layout of object categories. In CVPR, 2012. [2] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 2010.
%occlusion: percentage of occluded area of the object computed from ground truth annotation.
![Page 31: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/31.jpg)
Yu Xiang and Silvio Savarese. 3dRR'13 31
3D Localization Evaluation Ground truth 2D annotations Ground truth 3D spatial layout
2D object detections Predicted 3D spatial layout
Mean absolute difference in pairwise distances
![Page 32: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/32.jpg)
• 3D localization errors on the outdoor-scene dataset according to the best recalls of ALM, DPM and SLM.
3D Localization
Recall 54.8 64.4 76.8
ALM [1] 1.90 - -
DPM [2] 2.07 2.39 -
SLM 1.64 1.86 2.33
[1] Y. Xiang and S. Savarese. Estimating the aspect layout of object categories. In CVPR, 2012. [2] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part-based models. TPAMI, 2010.
32 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 33: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/33.jpg)
33
Anecdotal Results
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 34: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/34.jpg)
34
Anecdotal Results
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 35: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/35.jpg)
Outline • Related work
• 3D aspectlets
• Spatial layout model
• Experiments
• Conclusion
35 Yu Xiang and Silvio Savarese. 3dRR'13
![Page 36: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/36.jpg)
36
Conclusion
• 3D object representation
Atomic aspect part
3D aspectlet
• 3D object recognition
Spatial Layout Model (SLM)
Top-down occlusion reasoning
Bottom-up evidence from 3D aspectlets
Yu Xiang and Silvio Savarese. 3dRR'13
![Page 37: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/37.jpg)
Yu Xiang and Silvio Savarese. 3dRR'13 37
3DObjectPose: Large Scale Dataset with 3D Poses
12 rigid categories in PASCAL VOC
PASCAL VOC 2012 train and validation set
8505 images Subset of ImageNet 22394 images
Bounding box Pose: azimuth and elevation
3D CAD model Anchor points
Annotations
![Page 38: Object Detection by 3D Aspectlets and Occlusion Reasoning · Object Detection by 3D Aspectlets and Occlusion Reasoning Yu Xiang University of Michigan Silvio Savarese Stanford University](https://reader034.fdocuments.in/reader034/viewer/2022042709/5f442af5f3e94e4e3f2b6ef9/html5/thumbnails/38.jpg)
38
Thank you!
Acknowledgments
Yu Xiang and Silvio Savarese. 3dRR'13