Post on 21-Dec-2015
CS 395/495-25: Spring 2003CS 395/495-25: Spring 2003
IBMR: Week 7A IBMR: Week 7A
The Camera MatrixThe Camera MatrixContinued...Continued...
Jack TumblinJack Tumblin
jet@cs.northwestern.edujet@cs.northwestern.edu
IBMR-Related SeminarsIBMR-Related Seminars
3D Scanning for Cultural Heritage Applications3D Scanning for Cultural Heritage Applications Holly Rushmeier, IBM TJ Watson Holly Rushmeier, IBM TJ Watson
FridayFriday May 16 3:00pm, Rm 381, CS Dept. May 16 3:00pm, Rm 381, CS Dept.
no title yet ...<BRDF, BRSSDF capture? no title yet ...<BRDF, BRSSDF capture? Optics of Hair? Inverse Rendering?>Optics of Hair? Inverse Rendering?>
Steve Marschner, Cornell University Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.Friday May 23 3:00pm, Rm 381, CS Dept.
RemindersReminders
• ProjA graded: Good Job! 90,95, 110ProjA graded: Good Job! 90,95, 110• ProjB graded: Good! minor H confusions...ProjB graded: Good! minor H confusions...• MidTerm graded:MidTerm graded:• ProjC posted tonight; Due next Friday, May 15ProjC posted tonight; Due next Friday, May 15
• Start Watson’s Late Policy: Grade -(3Start Watson’s Late Policy: Grade -(3nn) points;) points;
n=# of class meetings late n=# of class meetings late• ProjD coming Thurs May 15, due Thurs May 29ProjD coming Thurs May 15, due Thurs May 29• Take-Home Final Exam: Thurs June 5, due June 11Take-Home Final Exam: Thurs June 5, due June 11
Basic Cameras RevisitedBasic Cameras Revisited
ffCC
yycc
zzcc
xxcc
pp
Plenty of TerminologyPlenty of Terminology::
• Image Plane or Focal PlaneImage Plane or Focal Plane
• Focal Distance Focal Distance ff
• Camera Center Camera Center CC
• Principal Point Principal Point pp
• Principal Axis Principal Axis zzcc
• Principal Plane Principal Plane (?!?! DOESN’T touch principle point!?!?)(?!?! DOESN’T touch principle point!?!?)
• Camera Coords (xCamera Coords (xcc,y,ycc,z,zcc))
• Image Coords (x’,y’)Image Coords (x’,y’)
Recall Basic Camera PRecall Basic Camera P00: :
• Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix:
• To translate To translate imageimage origin ( origin (x’,y’x’,y’) away from ) away from zzcc axis: axis:
Shift principal point Shift principal point pp from ( from (0,00,0) to () to (ppxx,p,pyy))
– does does NOTNOT use ‘homogeneous’ use ‘homogeneous’ 11 term in term in X X
– is is NOTNOT obvious: scales obvious: scales zzc c (think in P(think in P22 z zcc=x3 = 1,...)=x3 = 1,...)
x’x’y’y’z’z’
xxcc
yycc
zzcc
11
==
PP00 X = x X = x
yyy’y’
ffzz
CC
yycc
xxcc
zzcc
pp
ff 0 0 ppxx 00
0 0 ff ppyy 00
0 0 0 0 11 0 0
Basic Camera PBasic Camera P00: P: P33PP22 ((or camera or camera RR33))
• Basic Camera PBasic Camera P0 0 is a 3x4 matrix:is a 3x4 matrix:
• Non-square pixels? change scaling Non-square pixels? change scaling ((xx, , yy))
• Parallelogram pixels? set nonzero skew Parallelogram pixels? set nonzero skew ss
K matrixK matrix: “(internal) camera calib. matrix”: “(internal) camera calib. matrix”
xxyyzz
xxcc
yycc
zzcc
11
==
PP00 X = x X = x
yyy’y’
ffzz
CC
yycc
xxcc
zzcc
pp
xxff ss ppxx 00
0 0 yyff ppyy 00
0 0 0 0 11 0 0
KK (3x3 submatrix)(3x3 submatrix)
[K 0] = P[K 0] = P00
Complete Camera Matrix PComplete Camera Matrix P
• K matrixK matrix: “: “internalinternal camera calib. matrix” camera calib. matrix”
• RR··T matrixT matrix: “: “externalexternal camera calib. matrix” camera calib. matrix” – T matrix: Translate world origin to cam. origin,T matrix: Translate world origin to cam. origin,– R matrix: then Rotate world to fit cam. axesR matrix: then Rotate world to fit cam. axes
Combine: writeCombine: write
(P(P00··RR··T)T)··X = X = xx
asas
PP··X = X = xx
Output: Output: xxPP22 Camera Image Camera Image
Input:Input: XXPP33 World Space World Space
zz
yy
xx
XX (world space)(world space)
zz cc
yy cc
xx cc
CC
xx(c
amer
a sp
ace)
(cam
era
spac
e)
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
ColumnsColumns of P matrix = image of world-space axes: of P matrix = image of world-space axes:
• pp11,p,p22,p,p33 == image of x,y,z axis vanishing points == image of x,y,z axis vanishing points
– Direction D = [1 0 0 0]Direction D = [1 0 0 0]TT = point on P = point on P33’s x’s x11 axis, at inifinity axis, at inifinity
– PD = 1PD = 1stst column of P = P column of P = P11. Repeat for y and z axes. . Repeat for y and z axes.
• pp44 == image of the world-space origin pt. == image of the world-space origin pt.– Proof: let Proof: let DD = [0 0 0 1]T = world origin = [0 0 0 1]T = world origin– PDPD = 4 = 4thth column of column of PP = image of origin pt. = image of origin pt.
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
pp11 pp22 pp33 pp44
ff CC
yycc
zzcc
xxcc
pp
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
RowsRows of P matrix: camera planes in world space of P matrix: camera planes in world space• row 1 = row 1 = PP1T1T = image x-axis plane = image x-axis plane• row 2 = row 2 = PP2T2T = image y-axis plane = image y-axis plane• row 1 = row 1 = PP3T3T = camera’s principal plane = camera’s principal plane
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
PP1T1T
PP2T2T
PP3T3T
ff CC
yycc
zzcc
xxcc
pp
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• row 1 = row 1 = PP33 = camera’s principal plane = camera’s principal plane
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
ff CC
yycc
zzcc
xxcc
pp
PP1T1T
PP2T2T
PP3T3T
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• row 1 = row 1 = PP33 = camera’s principal plane = camera’s principal plane
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
ff CC
yycc
zzcc
xxcc
pp
PP1T1T
PP2T2T
PP3T3T
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• Careful!Careful! Shifting the Shifting the
image origin by image origin by ppxx, p, pyy
shifts the x,y axis planes!shifts the x,y axis planes!
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
ff CC
yycc
zzcc
xxcc
pp
PP1T1T
PP2T2T
PP3T3T
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• row 3 = row 3 = PP33 = camera’s principal plane = camera’s principal plane
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
ff CC
yycc
zzcc
xxcc
pp
PP1T1T
PP2T2T
PP3T3T
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• row 3 = row 3 = PP33 = camera’s principal plane = camera’s principal plane
– princip. plane princip. plane PP33= = [p[p3131 p p3232 p p3333 p p3434]]TT
– its normal direction: its normal direction: [p[p3131 p p3232 p p33 33 0 ]0 ]TT
– Why is it normal? It’s the world-space Why is it normal? It’s the world-space P P33 direction of the direction of the zzcc axis axis
BOOK has yet another inconsistent notation for this.....BOOK has yet another inconsistent notation for this.....
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
ff CC
yycc
zzcc
xxcc
pp
Principal Principal plane pplane p33
PP1T1T
PP2T2T
PP3T3T
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
• Principal Axis Vector (zPrincipal Axis Vector (zcc)) in in worldworld space: space:– Normal of principal plane: = Normal of principal plane: = mm33 = = [p[p3131 p p3232 p p33 33 ]]TT
– PP33 Scaling Scaling Ambiguous direction!! +/- m Ambiguous direction!! +/- m3 3 ? ? – Solution: use Solution: use det(M)det(M)··mm33 as front of cameraas front of camera
• Principal Point Principal Point p p in in imageimage space: space:– image of (infinity point on zimage of (infinity point on zcc axis = m axis = m33))
MM··mm33 = = pp = x = x0 0 (book renames p as x(book renames p as x00))
CC
yycc
zzcc
xxcc
pp
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
pp44
MM
mm11TT
mm22TT
mm33TT
xx00 ff
••••••00
••••••11
The Pieces of Camera Matrix The Pieces of Camera Matrix PP
Where is the Camera?Where is the Camera?
• Camera center Camera center CC is at camera origin is at camera origin(x(xcc,y,ycc,z,zcc)=0=C)=0=C
• What point What point CC will the camera will the camera PP transform to transform to C=0 C=0 ??Answer: Answer: CC is the Null Space of is the Null Space of P :P :
P C = C = 0 P C = C = 0 (solve for C. SVD works, but here’s an easier way:)(solve for C. SVD works, but here’s an easier way:)
Finite Camera: Finite Camera: C = C = Affine Camera: C Affine Camera: C = =
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or
~~~~ --MM-1-1··pp44 dd(where (where Md=0)Md=0)
~~
~~
PP = =• • •• • • ••• • •• • • ••• • • • • • • •
MM
mm11TT
mm22TT
mm33TT
ff CC
yycc
zzcc
xxcc
pp
pp44
Uses for Camera Matrix Uses for Camera Matrix PP
Each world space Each world space RR33 point point d = (x,y,z)d = (x,y,z) defines a world-space defines a world-space PP33 directiondirection D = (x,y,z,0)D = (x,y,z,0)..
What is the image point What is the image point xxdd from direction from direction DD? ?
xxdd = PD = [M | p= PD = [M | p44] D = M] D = M dd
((PP44 column has no effect, because of D’s zero): column has no effect, because of D’s zero):
xxdd = M = M d or Md or M-1-1xxdd = d = dff CC
yycc
zzcc
xxcc
pp
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
MM
xxdd
~~
pp44
zzyy xx
XX (world space)(world space)
dd
Uses for Camera Matrix Uses for Camera Matrix PP
Given image pointGiven image point xx00 and camera matrix and camera matrix PP,,Find ray Find ray X(X()) in world space through both: in world space through both:Slow, Obvious way: Slow, Obvious way:
‘‘Pseudo-invert’ Pseudo-invert’ PP, apply to x, apply to x00::• Define pseudo-inverse Define pseudo-inverse PP++ as as = P= PTT(PP(PPTT))-1-1
(note (note P·PP·P++ = I = I))• Find a world-space point on ray: XFind a world-space point on ray: X0 0 = P= P+ + xx00
• LIRP with world-space camera point C: LIRP with world-space camera point C:
X(X() = C + (X) = C + (X00-C)-C)ff CC
yycc
zzcc
xxcc
pp
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
MM
xx00
X(X())
~~
pp44
~~ ~~~~
~~
Uses for Camera Matrix Uses for Camera Matrix PP
Given image pointGiven image point xx00 and camera matrix and camera matrix PP,,Find ray Find ray X(X()) in world space through both: in world space through both:
Better way: ray from Better way: ray from CC to point to point DD00 at infinity: at infinity:• Point Point xx00 is the image of is the image of (unknown)(unknown) world-space direction world-space direction
DD00 = (x,y,z,0) = (x,y,z,0)TT. Define a point . Define a point dd00 = (x,y,z) = (x,y,z)TT. . • Recall: we can find Recall: we can find dd00 from the image: from the image: MM-1-1xx00 = d = d00 • Recall: world-space camera Recall: world-space camera C = -MC = -M-1-1pp44
X(X() = ) = MM-1-1xx00 + C = M + C = M-1-1 ((xx00 – – pp44))
0 1 10 1 1
ff CC
yycc
zzcc
xxcc
pp
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
MM
xx00
X(X())
~~
pp44
~~
~~
~~
Uses for Camera Matrix Uses for Camera Matrix PP
Given world-space pointGiven world-space point XX00, camera matrix , camera matrix PP,,
Find camera depth Find camera depth zz00::
– XX00 = [x = [xww, y, yww, z, zww, t, tww]]TT seen thru camera seen thru camera PP is is
XX00··PP = = xx00 = [x = [xcc, y, ycc, 1], 1]TT··wwcc
– Then Then signedsigned depth depth zz00 is: is:
zz00 = = wwcc sign(det sign(det MM))
ttww || || mm33 || ||
ff CC
yycc
zzcc
xxcc
pp
xxcc
yycc
zzcc
xxww
yyww
zzww
ttww
==• • • •• • • •• • • •• • • •• • • •• • • •
PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •
xx00
XX00
zz00
pp44
MM
mm11TT
mm22TT
mm33TT
Skipped:Skipped:
• P = [K|0]P = [K|0]··RR··TT How can we separate K, R, T? How can we separate K, R, T?– Answer: K is triangular; use QR decompositionAnswer: K is triangular; use QR decomposition
• Cameras at Infinity:Cameras at Infinity:– Orthographic or ‘Parallel Projection’ CamerasOrthographic or ‘Parallel Projection’ Cameras– Transition to Orthographic:Transition to Orthographic:
• Weak Perspective projection camerasWeak Perspective projection cameras• the ‘zoom’ lens (variable f)the ‘zoom’ lens (variable f)
– Moving line-scan or ‘pushbroom’ cameras Moving line-scan or ‘pushbroom’ cameras • Translation Scan: aerial/sattelite camerasTranslation Scan: aerial/sattelite cameras• Cylindrical Scan: panoramic camerasCylindrical Scan: panoramic cameras• UNC ‘HiBall Tracker’: 6 tiny self-locating line-scan camerasUNC ‘HiBall Tracker’: 6 tiny self-locating line-scan cameras
Chapter 6 Chapter 6 In Just One Slide:In Just One Slide:
Given point correspondence sets Given point correspondence sets (x(xii X Xii)), How , How do you find camera matrix do you find camera matrix P ? P ? (full 11 DOF)(full 11 DOF)
Surprise! You already know how !Surprise! You already know how !• DLT method: DLT method:
-rewrite H x = x’ as Hx -rewrite H x = x’ as Hx x’ = 0 x’ = 0--rewrite rewrite P X = xP X = x as as PX PX x = 0 x = 0-vectorize, stack, solve Ah = 0 for h vector-vectorize, stack, solve Ah = 0 for h vector-vectorize, stack, solve -vectorize, stack, solve Ap = 0Ap = 0 for for pp vector vector
--Normalizing step removes origin dependenceNormalizing step removes origin dependence
• More data More data better results (at least 28 point pairs) better results (at least 28 point pairs)• Algebraic & Geometric Error, Sampson Error…Algebraic & Geometric Error, Sampson Error…
ENDEND