Computer Vision : CISC4/689 Camera terminology a camera is defined by an optical center c and an...
-
date post
21-Dec-2015 -
Category
Documents
-
view
232 -
download
0
Transcript of Computer Vision : CISC4/689 Camera terminology a camera is defined by an optical center c and an...
Computer Vision : CISC4/689
Camera terminology
• a camera is defined by an optical center c and an image plane• image plane (or focal plane) is at distance f from the camera center
– f is called the focal length• camera center = optical center• principal axis = the line through camera center orthogonal to image plane• principal point = intersection of principal axis with image plane
– an indication of the camera center in the image (but NOT the camera center!)• principal plane = the plane through camera center parallel to image plane (XY plane in the fig.)
FOCAL POINT - (1) The central or principal point of focus. (2) The optical center of a lens when it is focused on infinity.
Computer Vision : CISC4/689
Pinhole Camera Terminology
Camera center/ pinhole
Principal point/image center
Image point
Camera point
Focal length
Optical axis
Image plane
Computer Vision : CISC4/689
The equation of projection
(Camera center)
Computer Vision : CISC4/689
The equation of projection
• Cartesian coordinates:– We have, by similar
triangles, that (x, y, z) -> (f x/z, f y/z, -f)
– Ignore the third coordinate, and get
(x,y, z) ( fxz
, fyz)
Computer Vision : CISC4/689
The camera matrix
• Turn previous expression into HC’s– HC’s for 3D point are
(X,Y,Z,T)
– HC’s for point in image are (U,V,W)
U
V
W
1 0 0 0
0 1 0 0
0 0 1f 0
X
Y
Z
T
Computer Vision : CISC4/689
Camera parameters
• Issue– camera may not be at the origin, looking down the z-axis
• extrinsic parameters• freeing the origin from the camera center involves a translation C• freeing the z-axis from the principal axis involves a rotation R
– one unit in camera coordinates may not be the same as one unit in world coordinates
• intrinsic parameters - focal length, principal point, aspect ratio, angle between axes, etc.
U
V
W
Transformation
representing
intrinsic parameters
1 0 0 0
0 1 0 0
0 0 1 0
Transformation
representing
extrinsic parameters
X
Y
Z
T
Note the matrix dimensions
Computer Vision : CISC4/689
Camera calibration
• Issues:– what are intrinsic
parameters of the camera?– what is the camera matrix?
(intrinsic+extrinsic)
• General strategy:– view calibration object– identify image points– obtain camera matrix by
minimizing error– obtain intrinsic parameters
from camera matrix
• Error minimization:– Linear least squares
• easy problem numerically• solution can be rather bad
– Minimize image distance• more difficult numerical
problem• solution usually rather
good, • start with linear least
squares
– Numerical scaling is an issue
Computer Vision : CISC4/689
Homogeneous Coordinates
• “Expanded” form is called homogeneous coordinates or projective space
• Change to projective space by adding a scale factor (usually but not always 1):
Computer Vision : CISC4/689
Homogeneous Coordinates: Projective Space
• Equivalence is defined up to scale ¸ (non-zero for finite points)
• Think of projective points in P2 as rays in R3, where z coordinate is scale factor– All Euclidean points along ray are “same” in this sense
Computer Vision : CISC4/689
Leaving Projective Space
• Can go back to non-homogeneous representation by dividing by scale factor and dropping extra coordinate:
• This is the same as saying “Where does the ray intersect
the plane defined by z = 1”?• Analogy to perspective projection, where f=1 (image plane) and lambda is z of
any point in the ray. For different lambda’s along the line, projected point is the same, thus Equivalence class
Computer Vision : CISC4/689
Ideal Points and Vanishing Points
• Ideal points on the projective plane are located at infinity, and have coordinates of the form (x1,x2,0). There is just one free parameter in the coordinates of an ideal point because the scale of a homogeneous vector is arbitrary [x1(1,x2/x1,0)]- thus the set of all ideal points on the projective plane constitutes a line, called the ideal line. In the same way, the ideal points of projective 3-space have the form (x1,x2,x3,0).
• The image of an ideal point under a projectivity is called a vanishing point, the image of an ideal line is called a vanishing line, and so on.
An algebraic analysis of this configuration would describe the following:parallel lines meet at infinity at an ideal point; the image these lines meeting at the ideal point is at the vanishing point V (the image of the ideal point is V) - note that concurrency is preserved under a projective transformation, so concurrency at the ideal point in the world is reflected in concurrency at V in the image;
Courtesy, CVonline
Computer Vision : CISC4/689
Courtesy, CVonline
Computer Vision : CISC4/689
Courtesy, CVonline
Figure: (a) One set of parallel lines on the plane is concurrent at the ideal point I1, and the other set is concurrent at I2 . The line through I1 and I2 is the ideal line (the horizon) of the plane. The image of I1 is the vanishing point V1, the image of I2 is the vanishing point V2, and the image of the ideal line is the vanishing line L. (b) Note that the plane formed by C and L has the same normal as the plane.
Application: image stabilization!Project!!! (2005 paper)
Computer Vision : CISC4/689
Onto 3D
• Coordinate systems• 3-D homogeneous transformations
– Translation, scaling, rotation
• Changes of coordinates– Rigid transformations
Computer Vision : CISC4/689
Vector Projection
• The projection of vector a onto u is that component
of a in the direction of u
Computer Vision : CISC4/689
Vector Cross Product
• Definition: If a = (xa, ya, za)T and
• b = (xb, yb, zb)T, then:
c = a X b
c is orthogonal to both a and b
from Hill
Computer Vision : CISC4/689
• Let x = (x, y, z)T be a point in 3-D space (R3). What do these values mean?
• A coordinate system in Rn is defined by an origin o and n orthogonal basis vectors – In R3, positive direction of each axis X, Y, Z is indicated by unit vector i, j, k,
respectively, where k = i X j (in a right-handed system)– Coordinate is length of projection of vector from origin to point onto axis basis
vector—e.g., x = x . i
Coordinate System: Definitions
x
o
Computer Vision : CISC4/689
3-D Camera Coordinates• Right-handed system• From point of view of camera looking out into scene:
+X right, {X left
+Y down, {Y up
+Z in front of camera, {Z behind
Computer Vision : CISC4/689
Going from 2-D to 3-D
• Points: Add z coordinate
• Transformations: Become 4 x 4 matrices with extra
row/column for z component—e.g., translation:
Computer Vision : CISC4/689
3-D Scaling
Computer Vision : CISC4/689
3-D Rotations
• In 2-D, we are always rotating in the plane of the image, but in 3-D the axis of rotation itself is a variable
• Three canonical rotation axes are the
coordinate axes X, Y, Z• These are sometimes referred to • in aviation terms: (X)roll, (Y)pitch, and (Z)yaw, respectively
from Hill
from Hill
Pitch is the angle that its longitudinal axis (running from tail to nose and along n) makes with horizontal plane.
Courtesy: Wikepedia
Computer Vision : CISC4/689
3-D Euler Rotation Matrices
• Similar to 2-D rotation matrices, but with coordinate corresponding to rotation axis held constant
• E.g., a rotation about the X axis of µ radians:
Computer Vision : CISC4/689
3-D Rotation Matrices
• General form is:
• Properties
– RT = R-1
– Preserves vector lengths, angles between vectors
– Upper-left block R3£3 is orthogonal matrix• Rows form orthonormal basis (as do columns): Length = 1, mutually
orthogonal
• So R3£3 x projects point x onto unit vectors represented by rows of
R3£3
Computer Vision : CISC4/689
• Camera coordinates C: Origin at center of camera, Z axis pointed in viewing direction
• World coordinates W: Arbitrary origin, axes– Way to specify camera location, orientation (aka pose) in same frame as
scene objects (we like to move camera to world, so as to convert world coordinates into camera coordinates)
• Cx, Wx,: Same point in different coordinates
Coordinate System Conversion
Computer Vision : CISC4/689
• Camera coordinates C: Origin at center of camera, Z axis pointed in viewing direction
• World coordinates W: Arbitrary origin, axes– Way to specify camera location, orientation (aka pose) in same frame as
scene objects
• Cx, Wx,: Same point in different coordinates
Coordinate System Conversion
Computer Vision : CISC4/689
• Camera coordinates C: Origin at center of camera, Z axis pointed in viewing direction
• World coordinates W: Arbitrary origin, axes– Way to specify camera location, orientation (aka pose) in same frame as
scene objects
• Cx, Wx,: Same point in different coordinates
Coordinate System Conversion
Computer Vision : CISC4/689
Change of Coordinates: Special Case of Same Axes
• Distinct origins, parallel basis vectors: If B is camera, Bx (camera) can be obtained by Ax (world) translated by its CG.
note: camera=calibrationestimated.
Computer Vision : CISC4/689
Change of Coordinates: Special Case of Same Origin
• Just need to rotate basis vectors so that they are aligned• Rotation matrix is projection of basis vectors in new frame Ex. World axes is (ia 0 0,0 ja 0,0 0 ka) and camera is (1 0 0, 0 1 0, 0 0 1)
Check by multing(ia 0 0), etc. the rotationMatrix formed by worldAxes.
(ia 0 0).(ia 0 0)(0 ja 0).(0 ja 0)(0 0 ka).(0 0 ka)=[1 0 0], i.e, transformedto camera coords.
(rotation of world in camera)
Computer Vision : CISC4/689
3-D Rigid Transformations
• Combination of rotation followed by translation without scaling
• “Moves” an object from one 3-D position and orientation (pose) to another
T R M
Computer Vision : CISC4/689
3-D Transformations: Arbitrary Change of Coordinates
• A rigid transformation can be used to represent a general change in the coordinate system that “expresses” a point’s location
Computer Vision : CISC4/689
• Points in one coordinate system are transformed to the other as follows:
• takes the camera to the world origin, transforming world coordinates to camera coordinates
• If A is camera and B is world, inverse translationand inverse rotation
Rigid Transformations: Homogeneous Coordinates
Computer Vision : CISC4/689
Camera Projection Matrix
• Using homogeneous coordinates, we can describe perspective projection as the result of multiplying by a 3 x 4 matrix P:
(by the rule for converting between homogeneous and regular coordinates—this is perspective division)
Computer Vision : CISC4/689
Camera Projection Matrix: Image Offsets
Center of CCD matrixusually does not coincide with the principalpoint C0. This addsu0 and v0 to definein pixel units of C0 inretinal coordinate system.
Computer Vision : CISC4/689
Factoring the Camera Matrix
• Another way to write it:
P = K ( Id 0 )
Camera calibrationmatrix
Identity form of rigid transformation
(with 4th row dropped)
Computer Vision : CISC4/689
Camera Calibration Matrix• More general matrix allows:
– Image coordinates with an offset origin (e.g., convention of upper left corner)– Non-square pixels = Different effective horizontal vs. vertical focal length
• These four variables are known as the camera’s intrinsic parameters
fu=f*sufv=f*sv
Computer Vision : CISC4/689
Dealing with World Coordinates
• Thus far we have assumed that points are in camera coordinates
• Recall the definition of the world-to-camera coordinate rigid transformation:
• In simpler form: And droplast row
Computer Vision : CISC4/689
Combining Intrinsic & Extrinsic Parameters
• The transformation performed by a pinhole camera on an arbitrary point in world coordinates can be written as:
3 x 4 projective camera matrix P has 10 degrees of freedom (DOF): 4 intrinsic, 3 rotation, 3 translation
Computer Vision : CISC4/689
Skew ignored
• The textbook has skew parameter included (pp. 29).
• Since the camera coordinate system may also be skewed due to some manufacturing error, the angle between the two image axes is not equal (maybe close to 90 degrees). This adds up another unknown parameter
• Easy to incorporate, just makes it 11 unknowns
Computer Vision : CISC4/689
Applications
• Estimates of the camera matrix parameters are critical in order to:– Know where the camera is and how it is moving
– Deduce structural characteristics of the scene (i.e., 3-D information)
– Place known objects (e.g., computer graphics) into a camera image correctly
Computer Vision : CISC4/689
Camera Matrix
• Linear systems of equations• Least-squares estimation• Application: Estimating the camera matrix
Computer Vision : CISC4/689
Linear System
• A general set of m simultaneous linear equations
in n variables can be written as:
Computer Vision : CISC4/689
Matrix Form of Linear System
• This can be represented as a matrix-vector product:
• Compactly, we write this as A x = b
Computer Vision : CISC4/689
Solving Linear Systems
• If m = n (A is a square matrix), then we can obtain the solution by simple inversion:
• If m > n, then the system is over-constrained and A is not invertible – Use the pseudoinverse A+ = (ATA)-1AT to obtain least-
squares solution x = A+b(Ax=B, multiply both sides by A^t,
etc.)
Computer Vision : CISC4/689
Fitting Lines
• A 2-D point x = (x, y) is on a line with slope m and
intercept b if and only if y = mx + b • Equivalently,
• So the line defined by two points x1, x2 is the solution to
the following system of equations:
Computer Vision : CISC4/689
Fitting Lines
• With more than two points, there is no guarantee that they will all be on the same line
• Least-squares solution obtained from pseudoinverse is line that is “closest” to all of the points
courtesy ofVanderbilt U.
Computer Vision : CISC4/689
Example: Fitting a Line
• Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3)
• Then
and x = A+b = (0.3571, 0.2857)T
Computer Vision : CISC4/689
Example: Fitting a Line
Computer Vision : CISC4/689
Homogeneous Systems of Equations
• Suppose we want to solve A x = 0• There is a trivial solution x = 0, but we don’t want this. For
what other values of x is A x close to 0?• This is satisfied by computing the singular value
decomposition (SVD) A = UDVT (a non-negative diagonal matrix between two orthogonal matrices) and taking x as the last column of V (unit singular vector) corresponding to the least eigenvalue.– Note that Matlab returns [U, D, V] = svd(A)-This is usually subject to constraints such as norm of x=1
Computer Vision : CISC4/689
Line-Fitting as a Homogeneous System
• A 2-D homogeneous point x = (x, y, 1)T is on the
line l = (a, b, c)T only when ax + by + c = 0
• We can write this equation with a dot product: x ¢ l = 0, and hence the following system is implied for multiple
points x1, x2, ..., xn:
Computer Vision : CISC4/689
Example: Homogeneous Line-Fitting
• Again we have 4 points, but now in homogeneous form: (2, 1, 1), (5, 2, 1), (7, 3, 1), and (8, 3, 1)
• Our system is:
• Taking the SVD of A, we get: b=-1, a=.3571, c=0.2857..scaled differently
C ompare to x = (0.3571, 0.2857)T
Computer Vision : CISC4/689
Camera Calibration
• Camera calibration is the name given to the process of discovering the projection matrix (and its decomposition into camera matrix and the position and orientation of the camera) from an image of a controlled scene. For ex., we might set up the camera to view a calibrated grid of some sort.
Computer Vision : CISC4/689
A Vision Problem: Estimating P
• Given a number of correspondences between 3-D
points and their 2-D image projections Xi $ xi, we
would like to determine the camera projection
matrix P such that xi = PXi for all i
Computer Vision : CISC4/689
A Calibration Target
courtesy of B. Wilburn
XZ
Y
Xi
xi
Computer Vision : CISC4/689
Estimating P: The Direct Linear Transformation (DLT) Algorithm
• xi = PXi is an equation involving homogeneous vectors
(powers are equal), so PXi and xi need only be in the same
direction, not strictly equal
• We can specify “same directionality” by using a cross product formulation:
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Preliminaries
• Let the image point xi = (xi, yi, wi)T
(remember that Xi has 4 elements)
• Denoting the jth row of P by pjT (a 4-element row vector), we have:
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Step 1
• Then by the definition of the cross product, xi £
PXi is:
Definition of cross product:U x V = uy vz – uz vy, uz vx – ux vz, ux vy – uy vx
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Step 2
• The dot product commutes, so pjT Xi = XT
i pj,
and we can rewrite the preceding as:
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Step 3
• Collecting terms, this can be rewritten as a matrix product:
where 0T = (0, 0, 0, 0). This is a 3 x 12
matrix times a 12-element column vector p = (p1T, p2T, p3T)T
Computer Vision : CISC4/689
What We Just Did
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Step 4
• There are only two linearly independent rows here
– The third row is obtained by adding xi times the first row to yi times the
second and scaling the sum by -1/wi
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Step 4
• So we can eliminate one row to obtain the following linear matrix equation for the ith pair of corresponding points:
• Write this as Ai p = 0
Computer Vision : CISC4/689
DLT Camera Matrix Estimation: Step 5
• Remember that there are 11 unknowns which generate the 3 x 4 homogeneous matrix P (represented in vector
form by p)• Each point correspondence yields 2 equations (the two
rows of Ai) We need at least 5 ½ point correspondences to solve for
p• Stack Ai to get homogeneous linear system A p = 0
Computer Vision : CISC4/689
Direct Linear Transform (DLT)(summary)
ii PXx ii PXx
rank-2 matrix
Computer Vision : CISC4/689
Direct Linear Transform (DLT)
Minimal solution
Over-determined solution
5½ correspondences needed (say 6)
P has 11 dof, 2 independent eq./points
n 6 points (usually, around 30 points needed?)
Apminimize subject to constraint
1p
use SVD
Computer Vision : CISC4/689
Degenerate configurations
(i) Points are collinear or single line passing through projection center
(ii) Camera and points on a twisted cubic
Computer Vision : CISC4/689
Scale data to values of order 1
1. move center of mass to origin2. scale to yield order 1 values
Data normalization
D3
D2
Computer Vision : CISC4/689
Geometric error
Computer Vision : CISC4/689
Gold Standard algorithmObjective
Given n≥6 2D to 3D point correspondences {Xi↔xi’}, determine the Maximum Likelihood Estimation of P
Algorithm
(i) Linear solution:
(a) Normalization:
(b) DLT
(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:
(iii) Denormalization:
ii UXX~ ii Txx~
UP~
TP -1
~ ~~
Computer Vision : CISC4/689
Calibration example
(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners
typically precision <1/10
(H&Z rule of thumb: 5n constraints for n unknowns)
Computer Vision : CISC4/689
Errors in the world
Errors in the image and in the world
ii XPx
iX
Errors in the image
iPXx̂
i
(standard case)
Computer Vision : CISC4/689
Radial distortion
• Due to spherical lenses (cheap)
• Model:
R
yxyxKyxKyx ...))()(1(),( 44
2
22
1R
http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpgstraight lines are not straight anymore
pincushion dist.
barrel dist.
Computer Vision : CISC4/689
Radial distortion example
Computer Vision : CISC4/689
Some typical calibration algorithmsTsai calibration
Zhangs calibration
http://research.microsoft.com/~zhang/calib/
Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.
Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.
Jean-Yves Bouguet’s matlab implementation:http://www.vision.caltech.edu/bouguetj/calib_doc/
Reg Willson’s implementation: http://www-2.cs.cmu.edu/~rgw/TsaiCode.html
Computer Vision : CISC4/689
Recovery of world position
• Given u,v we cannot uniquely determine the position of the point in the world.
• Each observed image point (u,v) gives us two equations in three unknowns (X,Y,Z). These equations define a line (i.e, ray) in space, on which the world point must lie.
• For general 3D scene interpretation, we need to use more than one view. Later in this course we will take a detailed look at stereo vision and structure from motion.