Putting Objects in Perspective

42
Putting Objects in Perspective Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University Robotics Institute

description

Putting Objects in Perspective. Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University Robotics Institute. Understanding an Image. Today: Local and Independent. What the Detector Sees. Local Object Detection. True Detection. False Detections. Missed. Missed. - PowerPoint PPT Presentation

Transcript of Putting Objects in Perspective

Page 1: Putting Objects in Perspective

Putting Objects in PerspectivePutting Objects in Perspective

Derek Hoiem

Alexei A. Efros

Martial Hebert

Carnegie Mellon University

Robotics Institute

Page 2: Putting Objects in Perspective

Understanding an ImageUnderstanding an Image

Page 3: Putting Objects in Perspective

Today: Local and IndependentToday: Local and Independent

Page 4: Putting Objects in Perspective

What the Detector SeesWhat the Detector Sees

Page 5: Putting Objects in Perspective

Local Object DetectionLocal Object Detection

True Detection

True Detections

MissedMissed

False Detections

Local Detector: [Dalal-Triggs 2005]

Page 6: Putting Objects in Perspective

Work in Context Work in Context

• Image understanding in the 70’sGuzman (SEE) 1968

Hansen & Riseman (VISIONS) 1978

Barrow & Tenenbaum 1978

Yakimovsky & Feldman 1973

Brooks (ACRONYM) 1979

Marr 1982

Ohta & Kanade 1973

• Recent work in 2D context

Kumar & Hebert 2005

Torralba, Murphy, Freeman 2004

Fink & Perona 2003

He, Zemel, Cerreira-Perpiñán 2004

Carbonetto, Freitas, Banard 2004

Winn & Shotton 2006

Page 7: Putting Objects in Perspective

Real Relationships are 3DReal Relationships are 3D

Close

Not Close

Page 8: Putting Objects in Perspective

Recent Work in 3DRecent Work in 3D

[Torralba, Murphy & Freeman 2003]

[Han & Zu 2003]

[Oliva & Torralba 2001]

[Han & Zu 2005]

Page 9: Putting Objects in Perspective

• Biederman’s Relations among Objects in a Well-Formed Scene (1981):

– Support

– Size

– Position

– Interposition

– Likelihood of Appearance

Objects and ScenesObjects and Scenes

Hock, Romanski, Galie, & Williams 1978

Page 10: Putting Objects in Perspective

• Biederman’s Relations among Objects in a Well-Formed Scene (1981):

Hock, Romanski, Galie, & Williams 1978

– Support

– Size

– Position

– Interposition

– Likelihood of Appearance

Contribution of this PaperContribution of this Paper

Page 11: Putting Objects in Perspective

Object SupportObject Support

Page 12: Putting Objects in Perspective

Surface EstimationSurface Estimation

Image Support Vertical Sky

V-Left V-Center V-Right V-Porous V-Solid

[Hoiem, Efros, Hebert ICCV 2005]

Software available online

ObjectSurface?

Support?

Page 13: Putting Objects in Perspective

Object Size in the ImageObject Size in the Image

Image World

Page 14: Putting Objects in Perspective

Input Image

Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint

Loose Viewpoint Prior

Page 15: Putting Objects in Perspective

Input Image

Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint

Loose Viewpoint Prior

Page 16: Putting Objects in Perspective

Object Position/Sizes Viewpoint

Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint

Page 17: Putting Objects in Perspective

Object Position/Sizes Viewpoint

Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint

Page 18: Putting Objects in Perspective

Object Position/Sizes Viewpoint

Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint

Page 19: Putting Objects in Perspective

Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint

Object Position/Sizes Viewpoint

Page 20: Putting Objects in Perspective

What does surface and viewpoint say about objects?What does surface and viewpoint say about objects?

Image

P(object) P(object | surfaces)

P(surfaces) P(viewpoint)

P(object | viewpoint)

Page 21: Putting Objects in Perspective

Image

P(object | surfaces, viewpoint)

What does surface and viewpoint say about objects?What does surface and viewpoint say about objects?

P(object)

P(surfaces) P(viewpoint)

Page 22: Putting Objects in Perspective

Scene Parts Are All InterconnectedScene Parts Are All Interconnected

Objects

3D SurfacesCamera Viewpoint

Page 23: Putting Objects in Perspective

Input to Our AlgorithmInput to Our Algorithm

Surface Estimates Viewpoint Prior

Surfaces: [Hoiem-Efros-Hebert 2005]

Local Car Detector

Local Ped Detector

Object Detection

Local Detector: [Dalal-Triggs 2005]

Page 24: Putting Objects in Perspective

Scene Parts Are All InterconnectedScene Parts Are All Interconnected

Objects

3D SurfacesViewpoint

Page 25: Putting Objects in Perspective

Our Approximate ModelOur Approximate Model

Objects

3D SurfacesViewpoint

Page 26: Putting Objects in Perspective

s1

o1

θ

on...

sn…

Local Object Evidence

Local Surface Evidence

Local Object Evidence

Local Surface Evidence

Viewpoint

Objects

Local Surfaces

Inference over Tree Easy with BP Inference over Tree Easy with BP

Page 27: Putting Objects in Perspective

Viewpoint estimationViewpoint estimation

Viewpoint Prior

HorizonHeight Height Horizon

Like

liho

od

Like

liho

od

Viewpoint Final

Page 28: Putting Objects in Perspective

Object detectionObject detection

4 TP / 2 FP

3 TP / 2 FP

4 TP / 1 FP

Ped Detection

Car Detection

Local Detector: [Dalal-Triggs 2005]

4 TP / 0 FP

Car: TP / FP

Ped: TP / FP

Initial (Local) Final (Global)

Page 29: Putting Objects in Perspective

Experiments on LabelMe DatasetExperiments on LabelMe Dataset

• Testing with LabelMe dataset: 422 images

– 923 Cars at least 14 pixels tall

– 720 Peds at least 36 pixels tall

Page 30: Putting Objects in Perspective

Each piece of evidence improves performanceEach piece of evidence improves performance

Local Detector from [Murphy-Torralba-Freeman 2003]

Car Detection Pedestrian Detection

Page 31: Putting Objects in Perspective

Can be used with any detector that outputs confidencesCan be used with any detector that outputs confidences

Local Detector: [Dalal-Triggs 2005] (SVM-based)

Car Detection Pedestrian Detection

Page 32: Putting Objects in Perspective

Accurate Horizon EstimationAccurate Horizon Estimation

Median Error:

8.5% 4.5% 3.0%

90% Bound:

[Murphy-Torralba-Freeman 2003]

[Dalal- Triggs 2005]

Horizon Prior

Page 33: Putting Objects in Perspective

Qualitative ResultsQualitative Results

Initial: 2 TP / 3 FP Final: 7 TP / 4 FP

Local Detector from [Murphy-Torralba-Freeman 2003]

Car: TP / FP Ped: TP / FP

Page 34: Putting Objects in Perspective

Qualitative ResultsQualitative Results

Local Detector from [Murphy-Torralba-Freeman 2003]

Car: TP / FP Ped: TP / FP

Initial: 1 TP / 14 FP Final: 3 TP / 5 FP

Page 35: Putting Objects in Perspective

Qualitative ResultsQualitative Results

Car: TP / FP Ped: TP / FP

Local Detector from [Murphy-Torralba-Freeman 2003]

Initial: 1 TP / 23 FP Final: 0 TP / 10 FP

Page 36: Putting Objects in Perspective

Qualitative ResultsQualitative Results

Local Detector from [Murphy-Torralba-Freeman 2003]

Car: TP / FP Ped: TP / FP

Initial: 0 TP / 6 FP Final: 4 TP / 3 FP

Page 37: Putting Objects in Perspective

Summary & Future WorkSummary & Future Work

meters

met

ers Ped

Ped

Car

Reasoning in 3D:

• Object to object

• Scene label

• Object segmentation

Page 38: Putting Objects in Perspective

ConclusionConclusion

• Image understanding is a 3D problem

– Must be solved jointly

• This paper is a small step

– Much remains to be done

Page 39: Putting Objects in Perspective

Thank youThank you

Page 40: Putting Objects in Perspective
Page 41: Putting Objects in Perspective

A Return to Scene UnderstandingA Return to Scene Understanding

• Guzman (SEE), 1968

• Hansen & Riseman (VISIONS), 1978

• Barrow & Tenenbaum 1978

• Brooks (ACRONYM), 1979

• Marr, 1982

• Ohta & Kanade, 1978

• Yakimovsky & Feldman, 1973

[Ohta & Kanade 1978]

Page 42: Putting Objects in Perspective

ImagesImages