3D LayoutCRF Derek Hoiem Carsten Rother John Winn.
-
Upload
ethel-phelps -
Category
Documents
-
view
217 -
download
0
Transcript of 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.
![Page 1: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/1.jpg)
3D LayoutCRF
Derek Hoiem
Carsten Rother
John Winn
![Page 2: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/2.jpg)
2
Goal 1: Object Description
Object Description:
• Bounding Box
• Viewpoint
• Color
• Pose
• Subclass
![Page 3: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/3.jpg)
3
Goal 2: Object Segmentation
![Page 4: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/4.jpg)
4
• Combine object-level and pixel-level reasoning
Key Idea
![Page 5: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/5.jpg)
5
Recognition Requires Object-Level Reasoning
• Position
• Shape/Size
• Viewpoint/Pose
• Style/Color
![Page 6: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/6.jpg)
6
Recognition Requires Object-Level Reasoning
![Page 7: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/7.jpg)
7
Solution: Window Detector?
• 45 degree range of viewpoints
• Minor scale/position variation
![Page 8: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/8.jpg)
8
What if we have a really good model?
![Page 9: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/9.jpg)
9
Recognition Requires Part-Level Reasoning
• Propose good global model
![Page 10: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/10.jpg)
10
Recognition Requires Part-Level Reasoning
• Propose good global model
• Occlusions
![Page 11: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/11.jpg)
11
Context Requires Both Object and Part-Level Info
• Size relationships require object model
![Page 12: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/12.jpg)
12
Context Requires Both Object and Part-Level Info
• Surface relationships require occlusion info
Visibly sitting on ground
Not visibly sitting on ground
![Page 13: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/13.jpg)
13
Our Object/Part Model
Ti = {
hj object parts
bounding box, viewpoint, color model, instance cost }
part consistency
occlusions
Tm
h1 h2 h3 h4
h5 h6 h7 h8
h9 h10 h11 hn
x
…
…
Extension from [Winn Shotton 2006]
T1…
![Page 14: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/14.jpg)
14
Modeling Viewpoint
Parameterized by Bounding Box and Corner
![Page 15: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/15.jpg)
15
Assigning Parts from Model
Training Image
FL
Training Annotation
Assigned Parts3D Parts Model
![Page 16: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/16.jpg)
16
Part Assignment Consistency
![Page 17: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/17.jpg)
17
Relabeling
• Allowing slight deformations, relabel training data
Training Image
Original Labels
New Labels
![Page 18: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/18.jpg)
18
Eight Viewpoint/Scale Ranges
Height Range
• Appearance (but not location) constant within each range
![Page 19: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/19.jpg)
20
Modeling Part Appearance
• Template patches (normalized xcorr)
• Intensity / Color
Image Edges (DT)
![Page 20: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/20.jpg)
21
Modeling Part Appearance
• Randomized decision trees– 25 trees, 250 leaf nodes
• Once:– Learn structure on 50,000 object / 50,000 background
pixels
• For each appearance model:– Learn parameters on all pixels (850 LabelMe images)
![Page 21: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/21.jpg)
22
Inference
Input Image
![Page 22: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/22.jpg)
23
Inference
Input Image
Proposals
• One per appearance model
• Objects proposed by connected components
![Page 23: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/23.jpg)
24
Proposal Stage Model
hi object parts
part consistency
occlusions
h1 h2 h3 h4
h5 h6 h7 h8
h9 h10 h11 hn
x
…
…
• CRF Inference (TRW-BP)
![Page 24: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/24.jpg)
25
Inference
Refinement
• One per proposal
• Incorporate viewpoint, size information
Proposals
Input Image
![Page 25: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/25.jpg)
26
Refinement Stage Model
Ti = {
hi object parts
bounding box, viewpoint }
part consistency
occlusions
T1
h1 h2 h3 h4
h5 h6 h7 h8
h9 h10 h11 hn
x
…
…
![Page 26: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/26.jpg)
27
Inference
Refinement
Proposals
Arbitration
• Includes color model, instance penalty (graph cuts)
Input Image
![Page 27: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/27.jpg)
28
Preliminary Results on UIUC
• Trained on 20, tested on rest• Quantitatively comparable to best
![Page 28: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/28.jpg)
29
Preliminary Results on UIUC
Without Instance Cost
With Instance Cost
T1
h1 h2 h3 h4
h5 h6 h7 h8
h9 h10 h11 hn
x
…
…
![Page 29: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/29.jpg)
30
Preliminary Results on PASCAL’06
• 25 images– One proposal (viewpoint within 45 degrees,
scale of 26-38 pixels)
![Page 30: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/30.jpg)
31
Preliminary Results on PASCAL’06
![Page 31: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/31.jpg)
32
Preliminary Results on PASCAL’06
![Page 32: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/32.jpg)
33
Preliminary Results on PASCAL’06
Without Color Model
With Color Model
![Page 33: 3D LayoutCRF Derek Hoiem Carsten Rother John Winn.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e425503460f94b34b5c/html5/thumbnails/33.jpg)
34
Conclusion
• Combined object-level and pixel-level reasoning – Object-level: Position/Size, Viewpoint, Color– Pixel-level: Part appearance, Occlusion
reasoning
• Good preliminary results