Presented by: Hadar Elor

61
Presented by: Hadar Elor Stereo Vision

description

Stereo Vision. Presented by: Hadar Elor. נתחיל בסגירת חוב. מבוסס על השקפים של טל הסנר. Geometric vision. Goal: Recovery of 3D structure Structure and depth are inherently ambiguous from single views. Geometric vision. Goal: Recovery of 3D structure - PowerPoint PPT Presentation

Transcript of Presented by: Hadar Elor

Page 1: Presented by:  Hadar Elor

Presented by: Hadar Elor

Stereo Vision

Page 2: Presented by:  Hadar Elor

חוב בסגירת ...נתחיל

Page 3: Presented by:  Hadar Elor

Geometric vision• Goal: Recovery of 3D structure

– Structure and depth are inherently ambiguous from single views.

הסנר טל של השקפים על מבוסס

Page 4: Presented by:  Hadar Elor

Geometric vision• Goal: Recovery of 3D structure

– What cues in the image allow us to do this?

Slide credit: Svetlana Lazebnik

Page 5: Presented by:  Hadar Elor

Shading

[Figure from Prados & Faugeras 2006]

Page 6: Presented by:  Hadar Elor

Focus/defocus

[figs from H. Jin and P. Favaro, 2002]

Images from same point of view, different camera parameters

3d shape / depth estimates

Page 7: Presented by:  Hadar Elor

Texture

[From A.M. Loh. The recovery of 3-D structure using visual texture patterns. PhD thesis]

Page 8: Presented by:  Hadar Elor

Perspective effects

Image credit: S. Seitz

Page 9: Presented by:  Hadar Elor

Motion

Figures from L. Zhang http://www.brainconnection.com/teasers/?main=illusion/motion-shape

Page 10: Presented by:  Hadar Elor

Estimating scene shape • “Shape from X”: Shading, Texture, Focus, Motion…

• Stereo: – shape from “motion” between two views– infer 3d shape of scene from two (multiple) images

from different viewpoints

scene point

optical center

image plane

Main idea:

Page 11: Presented by:  Hadar Elor

Geometry for a Simple Stereo System

• First, assuming parallel optical axes, known camera parameters (i.e., calibrated cameras):

11B. LeibeSlide credit: Kristen Grauman

Page 12: Presented by:  Hadar Elor

12B. Leibe

baseline

optical center (left)

optical Center (right)

Focal length

World point

Depth of pimage point (left) image point

(right)

Slide credit: Kristen Grauman

Page 13: Presented by:  Hadar Elor

Geometry for a Simple Stereo System• Assume parallel optical axes, known camera

parameters (i.e., calibrated cameras). We can triangulate via:

13B. Leibe

Similar triangles (pl, P, pr) and (Ol, P, Or):

r lT x x T

Z f Z

lr xx

TfZ

disparity

Slide credit: Kristen Grauman

Page 14: Presented by:  Hadar Elor

Disparity ועומק

מצלמות

הבדל במיקומי הנקודה (disparityבתמונות )

( ת

מוצל

מה

מק

חמר

dept

h)

lr xx

TfZ

disparity

Page 15: Presented by:  Hadar Elor

Depth From Disparity

15B. Leibe

Image I(x,y) Image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y), y)

Page 16: Presented by:  Hadar Elor

General Case With Calibrated Cameras

• The two cameras need not have parallel optical axes.

16B. Leibe

vs.

Slide credit: Kristen Grauman, Steve Seitz

Page 17: Presented by:  Hadar Elor

Stereo Correspondence Constraints

• Given p in the left image, where can the corresponding point p’ in the right image be?

17B. LeibeSlide credit: Kristen Grauman

Page 18: Presented by:  Hadar Elor

Stereo Correspondence Constraints

• Given p in the left image, where can the corresponding point p’ in the right image be?

18B. LeibeSlide credit: Kristen Grauman

Page 19: Presented by:  Hadar Elor

Stereo Correspondence Constraints

19B. LeibeSlide credit: Kristen Grauman

Page 20: Presented by:  Hadar Elor

Stereo Correspondence Constraints

• Geometry of two views allows us to constrain where the corresponding pixel for some image point in the first view must occur in the second view.

• Epipolar constraint: Why is this useful?– Reduces correspondence problem to 1D search along conjugate

epipolar lines.

20B. Leibe

epipolar planeepipolar lineepipolar line

Slide adapted from Steve Seitz

Page 21: Presented by:  Hadar Elor

Epipolar Geometry

21

• Epipolar Plane

• Epipoles • Epipolar Lines

• Baseline

Slide adapted from Marc Pollefeys

Page 22: Presented by:  Hadar Elor

Example

23B. LeibeSlide credit: Kristen Grauman

Page 23: Presented by:  Hadar Elor

• For a given stereo rig, how do we express the epipolar constraints algebraically?

24B. Leibe

Page 24: Presented by:  Hadar Elor

ההכרחית המטריצה בניית

נגדיר •סיבוב • מטריצת P, הקשר בין מיקום Rעבור

במערכת הקואורדינטות השמאלית לימנית הוא:

lO rO

P

l rlp

rp

מישור פולרי- י אפ�

r l T O O

r l P R P T

Page 25: Presented by:  Hadar Elor

Rotation MatrixExpress 3d rotation as series of rotations around coordinate axes by angles

,,

Overall rotation is product of these elementary rotations:

zyx RRRR

Slide credit: Kristen Grauman

Page 26: Presented by:  Hadar Elor

ההכרחית המטריצה בניית

• - על , נמצאים ו הוקטורים שלושת : האפיפולרי המישור המישור אותו

lO rO

P

l rlp

rp

מישור פול- י אפ�רי

lPT) )l P T

Page 27: Presented by:  Hadar Elor

Cross Product & Dot Product

• Vector cross product takes two vectors and returns a third vector that’s perpendicular to both inputs.

• So here, c is perpendicular to both a and b, which means the dot product = 0.

Slide credit: Kristen Grauman

Page 28: Presented by:  Hadar Elor

ההכרחית המטריצה בניית

• - על , נמצאים ו הוקטורים שלושת : האפיפולרי המישור המישור אותו

למישור • הניצב וקטור הוא מכאן:••- ו היותאזי:•• : ונקבל במשואה נציב

lO rO

P

l rlp

rp

מישור פול- י אפ�רי

lPT) )l P T

lT P

0T

l l P T T P

r l P R P T

Tr l R P P T

0TT T

r l r l R P T P P RT P

Page 29: Presented by:  Hadar Elor

Matrix Form of Cross Product

30Slide credit: Kristen Grauman

Page 30: Presented by:  Hadar Elor

ההכרחית המטריצה בניית

מטריצות • כפל באמצעות נשכתב– נגדיר –ונקבל: –

נקראת Eהמטריצה •המטריצה ההכרחית

)Essential Matrix)

0TT T

r l r l R P T P P RT P

0TT T

r l r x l R P T P P R T P

xE R T

0Tr l P EP

lO rO

P

l rlp

rp

מישור פול- י אפ�רי

Page 31: Presented by:  Hadar Elor

ההכרחית המטריצה

נתונות • התמונות במישורי ונקודות היות ,' נחליף הומ (שכן זהות p ב- Pבקואורדינטות

עד לכדי כפל בקבוע(ישר במישור התמונה הימנית אשר הוא •

מובטח כי מכיל את הנקודה•E שימושית כאשר נתונות לנו קואורדינטות נקודות

התמונה.במישורלנו יש קואורדינטות פיקסלים בתמונה...•

0Tr l p Ep

r lu Ep

rp

Page 32: Presented by:  Hadar Elor

היסודית המטריצה

היסודית • Fundamental Matrix Fמטריצהדומה באופייה למטריצה היסודית אך הפעם ו- •

פיקסליםבקואורדינטות • עבור ו- מטריצות פנימיות של שתי •

המצלמותאלג' שמונה חישוב המטריצה היסודית באמצעות "•

"הנקודות

0Tr l p Fp

lprp

1Tr l F M EM

lMlM

Page 33: Presented by:  Hadar Elor

Fundamental matrix

• Relates pixel coordinates in the two views

• More general form than essential matrix: we remove need to know intrinsic parameters

• If we estimate fundamental matrix from correspondences in pixel coordinates, can reconstruct epipolar geometry without intrinsic or extrinsic parameters

Grauman

Page 34: Presented by:  Hadar Elor

Computing F from correspondences

• Cameras are uncalibrated: we don’t know E or left or right Mint matrices

• Estimate F from 8+ point correspondences.

0leftright pFp

1int.int,

leftright EMMF

Grauman

Page 35: Presented by:  Hadar Elor

Computing F from correspondences

0leftright pFp

Each point correspondence generates one constraint on F

Grauman

We can re-write as:

Page 36: Presented by:  Hadar Elor
Page 37: Presented by:  Hadar Elor

סטראו מערכת כיול

So, where to start with uncalibrated cameras?Need to find fundamental matrix F and the correspondences (pairs of points (u’,v’) ↔

(u,v)).

1) Find interest points in image2) Compute correspondences3) Compute epipolar geometry4) Refine

Example from Andrew Zisserman

Page 38: Presented by:  Hadar Elor

1) Find interest points

Stereo pipeline with weak calibration

Grauman

Page 39: Presented by:  Hadar Elor

2) Match points only using proximity

Stereo pipeline with weak calibration

Grauman

Page 40: Presented by:  Hadar Elor

Putative matches based on correlation search

Grauman

Page 41: Presented by:  Hadar Elor
Page 42: Presented by:  Hadar Elor

עעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעעע

Page 43: Presented by:  Hadar Elor
Page 44: Presented by:  Hadar Elor
Page 45: Presented by:  Hadar Elor
Page 46: Presented by:  Hadar Elor
Page 47: Presented by:  Hadar Elor
Page 48: Presented by:  Hadar Elor
Page 49: Presented by:  Hadar Elor
Page 50: Presented by:  Hadar Elor
Page 51: Presented by:  Hadar Elor
Page 52: Presented by:  Hadar Elor
Page 53: Presented by:  Hadar Elor
Page 54: Presented by:  Hadar Elor

RANSAC for robust estimation of the fundamental matrix

• Select random sample of correspondences

• Compute F using them– This determines epipolar constraint

• Evaluate amount of support – inliers within threshold distance of epipolar line

• Choose F with most support (inliers)

Grauman

Page 55: Presented by:  Hadar Elor

Putative matches based on correlation search

Grauman

Page 56: Presented by:  Hadar Elor

Pruned matches• Correspondences consistent with epipolar geometry

Grauman

Page 57: Presented by:  Hadar Elor

• Resulting epipolar geometry

Grauman

Page 58: Presented by:  Hadar Elor

? היום ראינו מה אז

Page 59: Presented by:  Hadar Elor

סיכום

סטראו • מערכת של אנטומיהאפיפולרית • גיאומטריה

– , האפיפולרי, המישור האפיפולרי הישר האפיפול

• ' , מע וכיול היסודית ההכרחית המטריצהסטראו

Page 60: Presented by:  Hadar Elor

יסודית: המטריצה שיר !לסיכום

The Fundamental Matrix Song

Page 61: Presented by:  Hadar Elor

שקפים מקורות

• , על מבוססים רבים שקפים שצוין כל מלבד: של אלו

–B. Leibe–K. Grauman– D. Low–S. Lazebnik–A. Torralba–T. Darrell