3D Geometry and Camera Calibration. 3D Coordinate Systems Right-handed vs. left-handedRight-handed...

3D Geometry and3D Geometry andCamera CalibrationCamera Calibration

3D Coordinate Systems3D Coordinate Systems

• Right-handedRight-handed vs. left-handed vs. left-handed

xx

yy zz

xx

yyzz

2D Coordinate Systems2D Coordinate Systems

• yy axis up axis up vs. vs. yy axis down axis down

• Origin at center vs. Origin at center vs. cornercorner

• Will often write (Will often write (uu, , vv) for image ) for image coordinatescoordinates

uu

vv

uu

uu

vvvv

3D Geometry Basics3D Geometry Basics

• 3D points = column vectors3D points = column vectors

• Transformations = pre-multiplied Transformations = pre-multiplied matricesmatrices

z

y

x

p

z

y

x

p

z

y

x

ihg

fed

cba

p

T

z

y

x

ihg

fed

cba

p

T

RotationRotation

• Rotation about the Rotation about the zz axis axis

• Rotation about Rotation about xx, , yy axes similar axes similar(cyclically permute (cyclically permute xx, , yy, , zz))

100

0cossin

0sincos

zR

100

0cossin

0sincos

zR

Arbitrary RotationArbitrary Rotation

• Any rotation is a composition of Any rotation is a composition of rotations about rotations about xx, , yy, and , and zz

• Composition of transformations =Composition of transformations =matrix multiplication (watch the order!)matrix multiplication (watch the order!)

• Result: orthonormal matrixResult: orthonormal matrix– Each row, column has unit lengthEach row, column has unit length

– Dot product of rows or columns = 0Dot product of rows or columns = 0

– Inverse of matrix = transposeInverse of matrix = transpose

Arbitrary RotationArbitrary Rotation

• Rotate around Rotate around xx, , yy, then , then zz::

• Don’t do this! Compute simple matrices Don’t do this! Compute simple matrices and multiply them!and multiply them!

yxyxy

zyxzxzyxzxzy

zyxzxzyxzxzy

coscoscossinsin

sinsincoscossinsinsinsincoscossincos

cossincossinsincossinsinsincoscoscos

R

yxyxy

zyxzxzyxzxzy

zyxzxzyxzxzy

coscoscossinsin

sinsincoscossinsinsinsincoscossincos

cossincossinsincossinsinsincoscoscos

R

ScaleScale

• Scale in Scale in xx, , yy, , zz::

z

y

x

s

s

s

00

00

00

S

z

y

x

s

s

s

00

00

00

S

ShearShear

• Shear parallel to Shear parallel to xyxy plane: plane:

100

10

01

y

x

xy

σ

100

10

01

y

x

xy

σ

TranslationTranslation

• Can translation be represented by Can translation be represented by multiplying by a 3multiplying by a 33 matrix?3 matrix?

• No.No.

• Proof:Proof:

00:

AA 00:

AA

Homogeneous CoordinatesHomogeneous Coordinates

• Add a fourth dimension to each point:Add a fourth dimension to each point:

• To get “real” (3D) coordinates, divide by To get “real” (3D) coordinates, divide by ww::

w

z

y

x

z

y

x

w

z

y

x

z

y

x

wzwywx

w

z

y

x

wzwywx

w

z

y

x

Translation inTranslation inHomogeneous CoordinatesHomogeneous Coordinates

• After divide by After divide by ww, this is just a , this is just a translationtranslationby (by (ttx x , t, ty y , t, tzz))

w

wtz

wty

wtx

w

z

y

x

t

t

t

z

y

x

z

y

x

1000

100

010

001

w

wtz

wty

wtx

w

z

y

x

t

t

t

z

y

x

z

y

x

1000

100

010

001

Perspective ProjectionPerspective Projection

• What does 4What does 4thth row of matrix do? row of matrix do?

• After divide,After divide,

z

z

y

x

w

z

y

x

0100

0100

0010

0001

z

z

y

x

w

z

y

x

0100

0100

0010

0001

1zyzx

z

z

y

x

1zyzx

z

z

y

x

Perspective ProjectionPerspective Projection

• This is projection onto the This is projection onto the zz=1 plane=1 plane

• Add scaling, etc. Add scaling, etc. pinhole camera model pinhole camera model

((xx,,yy,,zz))

((xx//zz,,yy//zz,1),1)

(0,0,0)(0,0,0)zz=1=1

Putting It All Together:Putting It All Together:A Camera ModelA Camera Model

xcamcamcampiximg

TRPST xcamcamcampiximg

TRPST

3D point3D point(homogeneous coords)(homogeneous coords)

CameraCameralocationlocation

CameraCameraorientationorientation

PerspectivePerspectiveprojectionprojection

Scale toScale topixel sizepixel size

TranslateTranslateto imageto imagecentercenter

Then performThen performhomogeneoushomogeneous

divide, anddivide, andget (u,v) coordsget (u,v) coords


xcamcamcampiximg


TRPST

ExtrinsicsExtrinsicsIntrinsicsIntrinsics


xcamcamcampiximg


TRPST

World coordinatesWorld coordinates

Camera coordinatesCamera coordinatesNormalized device coordinatesNormalized device coordinates

Eye coordinatesEye coordinates

Image coordinatesImage coordinatesPixel coordinatesPixel coordinates

More General Camera ModelMore General Camera Model

• Multiply all these matrices togetherMultiply all these matrices together

• Don’t care about “Don’t care about “zz” after transformation” after transformation

• Scale ambiguity Scale ambiguity 11 free parameters 11 free parameters

lkzjyix

hgzfyexlkzjyix

dczbyax

z

y

x

lkji

hgfe

dcbashomogeneou

divide

1

lkzjyix

hgzfyexlkzjyix

dczbyax

z

y

x

lkji

hgfe

dcbashomogeneou

divide

1

Radial DistortionRadial Distortion

• Radial distortion can not be representedRadial distortion can not be representedby matrixby matrix

• ((ccuu, , ccvv) is image center,) is image center,

uu**imgimg= = uuimgimg– – ccuu, , vv**

imgimg= = vvimgimg– – ccvv,,

is first-order radial distortion coefficient is first-order radial distortion coefficient

)(1

)(12*2**

2*2**

imgimgimgvimg

imgimgimguimg

vuvcv

vuucu

)(1

)(12*2**

2*2**

imgimgimgvimg

imgimgimguimg

vuvcv

vuucu

Camera CalibrationCamera Calibration

• Determining values for camera Determining values for camera parametersparameters

• Necessary for any algorithm that requiresNecessary for any algorithm that requires3D 3D 2D mapping 2D mapping

• Method used depends on:Method used depends on:– What data is availableWhat data is available

– Intrinsics only vs. extrinsics only vs. bothIntrinsics only vs. extrinsics only vs. both

– Form of camera modelForm of camera model

Camera Calibration – Example 1Camera Calibration – Example 1

• Given:Given:– 3D 3D 2D correspondences 2D correspondences

– General perspective camera modelGeneral perspective camera model(11-parameter, no radial distortion)(11-parameter, no radial distortion)

• Write equations:Write equations:

1111

111

1111

111

vlkzjyix

hgzfyex

ulkzjyix

dczbyax

1111

111

1111

111

vlkzjyix

hgzfyex

ulkzjyix

dczbyax


• Linear equationLinear equation

• Overconstrained (more equations than unknowns)Overconstrained (more equations than unknowns)

• Underconstrained (rank deficient matrix – any Underconstrained (rank deficient matrix – any multiple of a solution, including 0, ismultiple of a solution, including 0, isalso a solution)also a solution)

0

l

c

b

a

uzuyuxuzyx

uzuyuxuzyx

uzuyuxuzyx

uzuyuxuzyx

2222222222

2222222222

1111111111

1111111111

10000

00001

10000

00001

0

l

c

b

a

uzuyuxuzyx

uzuyuxuzyx

uzuyuxuzyx

uzuyuxuzyx

2222222222

2222222222

1111111111

1111111111

10000

00001

10000

00001


• Standard linear least squares methods Standard linear least squares methods forforAx=0 will give the solution x=0Ax=0 will give the solution x=0

• Instead, look for a solution with |x|= 1Instead, look for a solution with |x|= 1

• That is, minimize |Ax|That is, minimize |Ax|22 subject to |x| subject to |x|22=1=1


• Minimize |Ax|Minimize |Ax|22 subject to |x| subject to |x|22=1=1

• |Ax||Ax|22 = (Ax) = (Ax)TT(Ax) = (x(Ax) = (xTTAATT)(Ax) = x)(Ax) = xTT(A(ATTA)xA)x

• Expand x in terms of eigenvectors of Expand x in terms of eigenvectors of AATTA:A:

x = x = 11ee11+ + 22ee22+…+…

x xTT(A(ATTA)x = A)x = 111122++2222

22+…+…

|x||x|22 = = 1122++22

22+…+…


• To minimizeTo minimize1111

22++222222+…+…

subject tosubject to 11

22++2222+… = 1+… = 1

set set minmin= 1 and all other = 1 and all other ii=0=0

• Thus, least squares solution is minimum Thus, least squares solution is minimum (non-zero) eigenvalue of A(non-zero) eigenvalue of ATTAA


• Incorporating additional constraints into Incorporating additional constraints into camera modelcamera model– No shear, no scale (rigid-body motion)No shear, no scale (rigid-body motion)

– Square pixelsSquare pixels

– etc.etc.

• These impose nonlinear constraints on These impose nonlinear constraints on camera parameterscamera parameters


• Option 1: solve for general perspective Option 1: solve for general perspective model, then find closest solution thatmodel, then find closest solution thatsatisfies constraintssatisfies constraints

• Option 2: nonlinear least squaresOption 2: nonlinear least squares– Usually “gradient descent” techniquesUsually “gradient descent” techniques

– Common implementations availableCommon implementations available(e.g. Matlab optimization toolbox)(e.g. Matlab optimization toolbox)


• Incorporating radial distortionIncorporating radial distortion

• Option 1:Option 1:– Find distortion first (straight lines inFind distortion first (straight lines in

calibration target)calibration target)

– Warp image to eliminate distortionWarp image to eliminate distortion

– Run (simpler) perspective calibrationRun (simpler) perspective calibration

• Option 2: nonlinear least squaresOption 2: nonlinear least squares


• What if 3D points are not known?What if 3D points are not known?

• Structure from motion problem!Structure from motion problem!

• As we saw last time, can often be As we saw last time, can often be solved since # of knowns > # of solved since # of knowns > # of unknownsunknowns

Multi-Camera GeometryMulti-Camera Geometry

• Epipolar geometry – relationship Epipolar geometry – relationship between observed positions of points in between observed positions of points in multiple camerasmultiple cameras

• Assume:Assume:– 2 cameras2 cameras

– Known intrinsics and extrinsicsKnown intrinsics and extrinsics

Epipolar GeometryEpipolar Geometry

PP

CC11 CC22

pp22pp11


PP

CC11 CC22

pp22pp11

ll22


PP

CC11 CC22

pp22pp11

ll22

Epipolar lineEpipolar line

EpipolesEpipoles


• Goal: derive equation for lGoal: derive equation for l22

• Observation: P, CObservation: P, C11, C, C22 determine a plane determine a plane

PP

CC11 CC22

pp22pp11

ll22


• Work in coordinate frame of CWork in coordinate frame of C11

• Normal of plane is T Normal of plane is T Rp Rp22, where T is , where T is

relative translation, R is relative rotationrelative translation, R is relative rotation

PP

CC11 CC22

pp22pp11

ll22


• pp11 is perpendicular to this normal: is perpendicular to this normal:

p p11 (T (T Rp Rp22) = 0) = 0

PP

CC11 CC22

pp22pp11

ll22


• Write cross product as matrix Write cross product as matrix multiplicationmultiplication

PP

CC11 CC22

pp22pp11

ll22

0

0

0

, **

xy

xz

yz

TT

TT

TT

xxT TT

0

0

0

, **

xy

xz

yz

TT

TT

TT

xxT TT


• pp11 T T* * R pR p22 = 0 = 0 p p11TT EE pp22 = 0 = 0

• E is the E is the essential matrixessential matrix

PP

CC11 CC22

pp22pp11

ll22

Essential MatrixEssential Matrix

• E depends only on camera geometryE depends only on camera geometry

• Given E, can derive equation for line lGiven E, can derive equation for line l22

PP

CC11 CC22

pp22pp11

ll22

Fundamental MatrixFundamental Matrix

• Can define Can define fundamental matrixfundamental matrix F F analogously, operating on pixel analogously, operating on pixel coordinates instead of camera coordinates instead of camera coordinatescoordinates

uu11TT FF uu22 = 0 = 0

• Advantage: can sometimes estimate F Advantage: can sometimes estimate F without knowing camera calibrationwithout knowing camera calibration

3D Geometry and Camera Calibration. 3D Coordinate Systems Right-handed vs. left-handedRight-handed...

Documents

Transcript of 3D Geometry and Camera Calibration. 3D Coordinate Systems Right-handed vs. left-handedRight-handed...