Lecture 17: Geometric Image Transformations

Harvey RhodyChester F. Carlson Center for Imaging Science

Rochester Institute of Technologyrhody@cis.rit.edu

November 11, 2004

AbstractGeometric transformations are widely used for image registration

and the removal of geometric distortion. Common applications includeconstruction of mosaics, geographical mapping, stereo and video.

DIP Lecture 17

Triangulation

Triangulation is a common tool for the description of a surface that isdefined on a random set of points.

Linear interpolation over the triangular regions is easily accomplished. Thisleads to a set of triangular facets that approximate the surface.

Consider the surfacef(x, y) = e−(x2+y2)

whose values are known on the set of random points shown in the diagram(next page).

How would one construct an approximation to f(x, y) for any (x, y)?

DIP Lecture 17 1

DeLaunay Triangles

Sample Points DeLaunay Triangles

DeLaunay triangles have the property that each the enclosing circle of eachtriangle encloses no other points.

DIP Lecture 17 2

Related Topic: Veronoi DiagramVoronoi Diagrams are usedextensively throughout geometry,computer science, and in manypractical applications. Althoughexamples of Voronoi diagramswere used in astronomy as earlyas 1644, Lejeune Dirichlet (1805-1859) was credited with theirfirst discovery in 1850. However,Voronoi diagrams were namedafter Georgy Voronoi (1868-1908).

Delaunay triangulations are the dual graph of Voronoi diagrams.

http://en.wikipedia.org/wiki/Voronoi_diagram

DIP Lecture 17 3

Triangular Surface Facets

The surface can be approximated by triangular facets. These form linearinterpolations between the values of f(xk, yk) at the known data points.

The facets are evident on these renderings because the number of points issmall.

DIP Lecture 17 4

Triangulation with 100 random points

A much smoother rendition of the function

f(x, y) = e−(x2+y2)

DIP Lecture 17 5

Image SharpeningThe image on theright is alow-resolutionversion of the imageon the left. It wasformed from onesample of each 8x8patch of the original.

We can combineseverallow-resolutionimages to constructa sharpened image.

How many boats are in the image on the right? How many people?

DIP Lecture 17 6

Image SharpeningSuppose that an image is known at two sets of points, such as thoseidentified by � and ∗ below.

We can construct a Delaunay triangulation over the combined sets asshown on the right. The image value can then be found at any point byinterpolation. The combined samples all contribute to the result.

DIP Lecture 17 7

Image Sharpening

The effect of combining four images using triangular interpolation is shownbelow. One has a chance of counting the boats and resolving some of thepeople in the image on the right.

Sampled 1:64 Original Sharpened (4)

DIP Lecture 17 8

Image Sharpening

The same technique can be used to combine more images. Resolutionincreases with more observations.

Sharpened (8) Sharpened (16) Sharpened (32)

DIP Lecture 17 9

Image Sharpening

The sharpening is evident by comparing a single subsampled image with theoriginal and one that has been sharpened with many observations.

Single Image Original Sharpened (32)

DIP Lecture 17 10

Reconstruction from Random Pixels

It is not necessary for the sub-images to be sampled on a regular grid or thateach of them has samples that are related to the others in any particularway.

The following slide shows images that were reconstructed from 1%, 10%,20%, 30%, 50% and 70% of the pixels of the original image.

The pixels used for reconstruction were selected randomly.

This is not a common situation, but it does demonstrate the flexibility ofthe method.

DIP Lecture 17 11

Interpolation from Random Pixels

DIP Lecture 17 12

Interpolation Algorithm

The interpolation process can be done with just a few statements in IDLby using the TRIANGULATE and TRIGRID routines. (See IDL Online Helpfor details.)

Suppose that samples of a scene A are obtained by two observations. Let(x1(n), y1(n)) be the coordinates for observation A1 and (x2(n), y2(n)) bethe coordinates for observation A2. The size of the desired output array isN ×M pixels. We can combine them as

X=[X1,X2]Y=[Y1,Y2]B=[A1,A2]TRIANGULATE,X,Y,TR ;Construct Delaunay trianglationC=TRIGRID(X,Y,B,TR,NX=N,NY=M) ;Interpolate on NxM grid

The image can be displayed as usual:

Window,/free,xsize=N,ysize=MTV,C

DIP Lecture 17 13

Coordinate Transformation

Two or more images cannot be combined until they are associated with acommon coordinate system.

Associating images with a common coordinate system requires:

• Selecting a base coordinate system.

• Determining how the coordinate system for each image array is relatedto the base coordinate system.

• Mapping the pixel coordinates to the base coordinates.

The mapping is a coordinate transformation that is chosen to match theproperties of the system. Its parameters are determined by image analysis.

DIP Lecture 17 14

Coordinate Transformation – Translation

We will look at some basic coordinate transformations and relatedmathematical representations. We begin with translation, scaling androtation.

TranslationTranslation changes the coordinates of each point (x, y, z) to (x + Dx, y +Dy, z + Dz). This can be represented mathematically in homogeneous 3Dcoordinates as

x′

y′

z′

1

=

1 0 0 Dx

0 1 0 Dy

0 0 1 Dz

0 0 0 1

xyz1

The use of a 4×4 transformation matrix allows translation to be representedas multiplication of a vector by a matrix, and enables translation, rotationand scaling to be described by a common system.

The position coordinates are the first three components of the homogeneouscoordinate vector.

DIP Lecture 17 15

Scaling

Scaling changes the coordinates of each point (x, y, z) to (xSx, ySy, zSz).This is like zooming in or out. The scale factors do not have to be thesame, but unequal values produce nonuniform scaling.

The transformation for scaling isx′

y′

z′

1

=

Sx 0 0 00 Sy 0 00 0 Sz 00 0 0 1

xyz1

DIP Lecture 17 16

Combined Transformations

Translation followed by scaling can be represented by a combinedtransformation.

x′′ = Sx′ = STx

where

ST =

Sx 0 0 00 Sy 0 00 0 Sz 00 0 0 1

1 0 0 Dx

0 1 0 Dy

0 0 1 Dz

0 0 0 1

=

Sx 0 0 SxDx

0 Sy 0 SyDy

0 0 Sz SzDz

0 0 0 1

This is not the same as scaling followed by translation.

TS =

1 0 0 Dx

0 1 0 Dy

0 0 1 Dz

0 0 0 1

Sx 0 0 00 Sy 0 00 0 Sz 00 0 0 1

=

Sx 0 0 Dx

0 Sy 0 Dy

0 0 Sz Dz

0 0 0 1

DIP Lecture 17 17

Rotation

Rotation about the origin can be composed of rotations about the x, y andz axes.

Rx(θ) =

1 0 0 00 cos θ − sin θ 00 sin θ cos θ 00 0 0 1

Ry(φ) =

cos φ 0 sinφ 0

0 1 0 0− sinφ 0 cos φ 0

0 0 0 1

Rz(ω) =

cos ω − sinω 0 0sinω cos ω 0 0

0 0 1 00 0 0 1

DIP Lecture 17 18

Composition of Transformations

A sequence of transformations can be constructed by multiplying thematrices.

Each of the transformations is invertible so that the product is also invertible.

The combined transformation is of the form

A =

r11 r12 r13 Dx

r21 r22 r23 Dy

r31 r32 r33 Dy

0 0 0 1

A rigid transformation from one coordinate system to another can berepresented by

x′

y′

z′

1

=

r11 r12 r13 Dx

r21 r22 r23 Dy

r31 r32 r33 Dz

0 0 0 1

xyz1

DIP Lecture 17 19

Solving for A

How do we find the transformation matrix A for a given problem?

Let P = [p1,p2, . . .pn] be a set of points on one image and let Q =[q1,q2, . . .qn] be the corresponding points on another image. pi and qi

are matching homogeneous coordinate vectors.

If the images differ only in 3D translation, rotation and scaling then theremust be an array A such that

Q = AP

We can solve for A from the n points in each image by

QPT = APPT

QPT (PPT )−1 = A(PPT )(PPT )−1 = A

The quantity P † = PT (PPT )−1 is the pseudo-inverse of the array P .

DIP Lecture 17 20

Example

The figure shown on the left is defined by ten vertices. It is converted intothe figure on the right by a rotation of 45◦ about the x−axis.

DIP Lecture 17 21

Example (cont)

P =

0. 5. 5. 2. 0. 0. 5. 5. 2. 0.0. 0. 2. 3. 2. 0. 0. 2. 3. 2.0. 0. 0. 0. 0. 5. 5. 5. 5. 5.1. 1. 1. 1. 1. 1. 1. 1. 1. 1.

A =

1.000 0.000 0.000 0.0000.000 0.707 −0.707 0.0000.000 0.707 0.707 0.0000.000 0.000 0.000 1.000

Q = AP

=

0.00 5.00 5.00 2.00 0.00 0.00 5.00 5.00 2.00 0.000.00 0.00 1.41 2.12 1.41 −3.54 −3.54 −2.12 −1.41 −2.120.00 0.00 1.41 2.12 1.41 3.54 3.54 4.95 5.66 4.951.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

DIP Lecture 17 22

Example (cont)

(PPT ) =

108 32 60 2432 34 35 1460 35 125 2524 14 25 10

(PPT )−1 =

0.020 0.002 −0.000 −0.0510.002 0.070 −0.000 −0.1030.000 0.000 0.016 −0.040

−0.051 −0.103 −0.040 0.466

We can now estimate the transformation matrix A by using only theobservations P and Q by computing the pseudo-inverse.

P † = PT (PPT )−1

DIP Lecture 17 23

Example (cont)

P † =

−0.051 −0.103 −0.040 0.4660.049 −0.092 −0.040 0.2120.053 0.048 −0.040 0.006

−0.004 0.111 −0.040 0.056−0.046 0.037 −0.040 0.260−0.051 −0.103 0.040 0.266

0.049 −0.092 0.040 0.0120.053 0.048 0.040 −0.194

−0.004 0.111 0.040 −0.144−0.046 0.037 0.040 0.060

A = QP † =

1.000 0.000 −0.000 −0.0000.000 0.707 −0.707 0.000

−0.000 0.707 0.707 −0.000−0.000 0.000 −0.000 1.000

DIP Lecture 17 24

Spatial Transformations

An image f(x, y) can be spatially transformed to an image g(x′, y′) byspecifying:

• The coordinate mapping

x′ = r(x, y)

y′ = s(x, y)

• A method of interpolation to compute the appropriate value g from f .

Determining the mapping functions r(x, y) and s(x, y) to achieve desiredresults such as the removal of geometric distortion is the major challenge.

It is desirable to have automated methods for processing a large number ofdistorted images.

DIP Lecture 17 25

Image Interpolation

Interpolation is almost always needed in the process of geometrictransformation.

Assume that a distorted image f(x, y) is given as an image array of size Ncolumns by M rows. Assume that we want g(x′, y′) to be defined on a gridof points G.

The points of f(x, y) are unlikely to map to the grid G.

DIP Lecture 17 26

Image Interpolation

We can find the value f(x′, y′) at each point [x′k, y′k] = [r(xk, yk), s(xk, yk)],

but these points do not fall on the desired grid G.

We can find the value at the desired grid points by interpolation.

What interpolation method should we use?

DIP Lecture 17 27

Bilinear Interpolation

Suppose that we want to find the value g(q) at a point q that is interior to afour-sided figure with vertices {p1, p2, p3, p4}. Assume that these points arein order of progression around the figure and that p1 is the point farthestto the left.

1. Find the point (xa, yq) between p1

and p4. Compute g(xa, yq) by linearinterpolation between f(p1) and f(p4).

2. Find the point (xb, yq) between p2

and p3. Compute g(xb, yq) by linearinterpolation between f(p2) and f(p3).

3. Linearly interpolate between g(xa, yq)and g(xb, yq) to find g(xq, yq).

DIP Lecture 17 28

IDL BILINEAR RoutineThe BILINEAR function uses a bilinear interpolation algorithm to computethe value of a data array at each of a set of subscript values.

R=BILINEAR(P,X,Y)

P is a data array and X and Y are coordinate arrays for the points at whichthe interpolated result is desired.

EXAMPLE:

Given an array

P =

0. 1. 4.9. 16. 25.

36. 49. 64.

find values for the coordinate points {(0.5, 0.5), (0.5, 1.3), (1.3, 1.5), (1.6, 1.5)}Construct two coordinate arrays

X =[

0.5 1.30.5 1.6

]Y =

[0.5 1.51.3 1.5

]

DIP Lecture 17 29

IDL BILINEAR Routine (cont)

The interpolated values are given by R=BILINEAR(P,X,Y).

R =[

6.5 36.121.5 39.7

]

These values can be checked using the bilinear interpolation formula.

Note that the coordinate values correspond to points that fall between gridpoints with unit grid steps.

An interpolation rule is incorporated into the mapping routines that we willdiscuss. Bilinear interpolations is one of the options.

DIP Lecture 17 30

2D Polynomial Warping

IDL provides the POLY 2D function performs polynomial warping of images.This function performs a geometrical transformation in which the resultingarray is defined by:

g[x, y] = f [x′, y′] = f [r[x, y], s[x, y]]

The functions r(x, y) and s(x, y) are polynomials in x and y of degree N ,whose coefficients are given by coefficient arrays P and Q (here we followIDL notation):

x′ = r(x, y) =N∑

i=0

N∑j=0

Pi,jxjyi

y′ = s(x, y) =N∑

i=0

N∑j=0

Qi,jxjyi

DIP Lecture 17 31

Polynomial Warping

The coefficients can be found by solving a set of simultaneous equationsthat involve matching points in each image. To solve for the (N + 1)2

coefficients we need at least (N + 1)2 independent pairs of points.

It is typical to select a larger set of matched point pairs and to choosethe coefficients to minimize the mean-squared error in the match. This isconceptually connected to linear regression.

The solution can be expressed in matrix form to facilitate the mathematicsand programming.

DIP Lecture 17 32

Polynomial Warping

Let A and B be two image arrays, and let m matching points be definedby the vectors

xa = [xa(0), xa(1), . . . , xa(m− 1)]

ya = [ya(0), ya(1), . . . , ya(m− 1)]

xb = [xb(0), xb(1), . . . , xb(m− 1)]

yb = [yb(0), yb(1), . . . , yb(m− 1)]

Let us construct an array W in which row k is xjay

ia where i = k/(N + 1)

and j = k mod (N + 1). There are (N + 1)2 rows.

Let M be a corresponding array of the Pi,j and Qi,j coefficients.

DIP Lecture 17 33

Polynomial Warping

M =[

P00 P01 P02 . . . P0N P10 . . . Pij . . . PNN

Q00 Q01 Q02 . . . Q0N Q10 . . . Qij . . . QNN

]

W =

x0ay

0a

x1ay

0a

x2ay

0a

...xN

a y0a

x0ay

1a

...xj

ayia

...xN

a yNa

DIP Lecture 17 34

Polynomial Warping

The warping equations can then be expressed as[xb

yb

]= MW

These equations can be solved by computing the pseudo-inverse:[xb

yb

]WT = M(WWT )

[xb

yb

][WT (WWT )−1] = M

or, in terms of the pseudo-inverse,

M =[

xb

yb

]W †

DIP Lecture 17 35

Polynomial Warping

Once we know the coefficient array M we can map the array B onto thecoordinate plane of A.

Let xa and ya now represent all of the pixel coordinates in the A domain.

Use the coefficients from M to compute xb and yb. Note that you may notwant to actually construct the array W to conserve memory.

Remove the array values outside the B image boundary.

Find the value at the xb,yb coordinates by interpolation from the pixelvalues on the B grid. These form the image values in the A plane.

Draw the image in the A window.

DIP Lecture 17 36

2D Polynomial Warping

The coefficients can be computed using matching control points from thetwo images with the PolyWarp procedure.

POLYWARP, Xa, Ya, Xb, Yb, N, P,Q

The image can then be warped by use of POLY 2D)

A=POLY 2D(B,P,Q)

This achieves a similar, but not identical result, to actually doing the matrixcomputations.

DIP Lecture 17 37

ExampleImage A Image B[100, 200] [180, 117.0][150, 300] [270, 173.4][150, 200] [210, 116.4][150, 400] [330, 239.4][150, 50] [120, 30.0][200, 150] [210, 84.0][200, 50] [150, 22.8][250, 350] [360, 204.6][300, 300] [360, 178.8][350,400] [450, 235.8][350, 50] [240, 27.6][350, 200] [330, 112.2][350, 150] [300, 81.6]

Image A Image B

The image on the right was constructed using POLYWARP and POLY 2D.

DIP Lecture 17 38