Image Transform

Panorama Image (Keller+Lind Hall)

Image Warping (Coordinate Transform)

2 1( ) = ( )I v I u

I1 I2

u

v

Image Warping (Coordinate Transform)

2 1( ) = ( )I v I u

I1 I2

uv

: Pixel transport

Image Warping (Coordinate Transform)

2 1( ) = ( )I v I u

I1 I2

uv

: Pixel transport

Cf) Image Filtering (Pixel Transform)

gI I2 1= ( )

I1 I2

: Pixel transform

Uniform Scaling

I1

u

I2

v

2 1( ) = ( )I v I u

Uniform Scaling

I1

u

I2

x x x

y y y

1 1 1

v s u

v s u?

v

2 1( ) = ( )I v I u

Uniform Scaling

I1

u

I2

s sx y=

v

x x x

y y y

1 1 1

v s u

v s u

2 1( ) = ( )I v I u

Aspect Ratio Change

I1I2

u v

s sx y

x x x

y y y

1 1 1

v s u

v s u

2 1( ) = ( )I v I u

Translation

I1

uv

I2

t x

t y

x x x

y y y

1

1

1 1 1

v t u

v t u?

2 1( ) = ( )I v I u

Translation

I1

uv

I2

t x

t y

x x x

y y y

1

1

1 1 1

v t u

v t u

2 1( ) = ( )I v I u

Rotation

I1

u

v I2

x x

y y

cos sin

sin cos

11 1

v u

v u

θ θ

θ θ

θ

?

2 1( ) = ( )I v I u

Rotation

I1

u

v I2

θ

x x

y y

cos sin

sin cos

11 1

v u

v u

θ θ

θ θ

2 1( ) = ( )I v I u

x x x

y y y

cos sin

sin cos

1 1 1

v t u

v t u

θ θ

θ θ

Euclidean Transform SE(3)

Rotation around the center of image

I1 I2

?

Euclidean Transform SE(3)

x x x

y y y

cos sin

sin cos

1 1 1

v t u

v t u

θ θ

θ θ

Rotation around the center of image

I1 I2

Euclidean Transform SE(3)

Rotation around the center of image

I1 I2Invariant properties

• Length • Angle • Area

Degree of freedom

3 (2 translation+1 rotation)

x x x

y y y

cos sin

sin cos

1 1 1

v t u

v t u

θ θ

θ θ

Euclidean Transform SE(3)

Rotation around the center of image

I1 I2Invariant properties

• Length • Angle • Area

Degree of freedom

3 (2 translation+1 rotation)

x x x

y y y

cos sin

sin cos

1 1 1

v t u

v t u

θ θ

θ θ1 1 1

v R t u

0

Euclidean Transform SE(3)

I2

Rotate about the image center

?

vu

t

1 1 1

v R t u

0+v Ru t

Euclidean Transform SE(3)

Rotate about the image center

vu

? t

1 1 1

v R t u

0+v Ru t

Euclidean Transform SE(3)

I2

Rotate about the image center

p

vu

p

1 1 1

v R t u

0+v Ru t

Euclidean Transform SE(3)

I2

Rotate about the image center

= -u u pp

vu

p = -v v p

v Ru

1 1 1

v R t u

0+v Ru t

Euclidean Transform SE(3)

I2

Rotate about the image center

= -u u pp

vu

p = -v v p

v Ru

1 1 1

v R t u

0+v Ru t

v -p R u-p

v Ru-Rp+p

Euclidean Transform SE(3)

I2

Rotate about the image center

= -u u pp

vu

p = -v v p

v Ru

1 1 1

v R t u

0+v Ru t

v -p R u-p

v Ru-Rp+p

t -Rp+p

Similarity Transform

I1 I2

x x x x

y y y y

cos sin

sin cos1

11 1 1 1

R t

0

v t u u

v t u u

θ θ

θ θ?

Similarity Transform

I1 I2

x x x x

y y y y

cos sin

sin cos1

11 1 1 1

R t

0

v t u u

v t u u

θ θ

θ θ

Similarity Transform

I1 I2Invariant properties

• Length ratio • Angle

x x x x

y y y y

cos sin

sin cos1

11 1 1 1

R t

0

v t u u

v t u u

θ θ

θ θ

Degree of freedom

4 (2 translation+1 rotation+1 scale)

Affine Transform

I1 I2

x x

y y1

1 1

R t

0

v u

v u

Euclidean transform

Affine Transform

I1 I2

x x

y y1

1 1

R t

0

v u

v ux 11 12 13 x x

y 21 22 23 y y1

0 0 11 1 1

A t

0

v a a a u u

v a a a u u

Euclidean transform

Affine Transform

I1 I2

x x

y y1

1 1

R t

0

v u

v ux 11 12 13 x x

y 21 22 23 y y1

0 0 11 1 1

A t

0

v a a a u u

v a a a u u

Euclidean transform

Invariant properties

• Parallelism • Ratio of area • Ratio of length

Degree of freedom

6

Perspective Transform (Homography)

I1 I2

x 11 12 13 x x

y 21 22 23 y y

31 32 11 1 1

H

v h h h u u

v h h h u u

h h

Perspective Transform (Homography)

I1 I2

x 11 12 13 x x

y 21 22 23 y y

31 32 11 1 1

H

v h h h u u

v h h h u u

h h

: General form of plane to plane linear mapping

Ground plane

Camera plane

u

Perspective Transform (Homography)

u K R t Xu

Ground plane Camera plane

X

Ground plane

Camera plane

u

Perspective Transform (Homography)

u K R t X

0

1

X =

X

Y

uGround plane Camera plane

Ground plane

Camera plane

u

Perspective Transform (Homography)

u K R t X

0

1

X =

X

Y

uGround plane Camera plane

1 2 30

1

u K r r r t

X

Y

Ground plane

Camera plane

u

Perspective Transform (Homography)

u K R t X

0

1

X =

X

Y

uGround plane Camera plane

1 2 30

1

u K r r r t

X

Y

1 2

1

u K r r t

X

Y

Ground plane

Camera plane

u

Perspective Transform (Homography)

u K R t X

0

1

X =

X

Y

uGround plane Camera plane

1 2 30

1

u K r r r t

X

Y

1 2

1

u K r r t

X

Y

x

y

11

H

u X

u Y

Perspective Transform (Homography)

I1 I2Invariant properties

• Cross ratio

• Concurrency

• Colinearity

x 11 12 13 x x

y 21 22 23 y y

31 32 11 1 1

H

v h h h u u

v h h h u u

h h

Degree of freedom

8 (9 variables 1 scale)

Hierarchy of Transformations

Euclidean (3 dof) Similarity (4 dof) Affine (6 dof) Projective (8 dof)

• Length • Angle • Area

• Length ratio • Angle

• Parallelism • Ratio of area • Ratio of length

• Cross ratio • Concurrency • Colinearity

x

y

cos sin

sin cos

1

t

t

θ θ

θ θ

x

y

cos sin

sin cos

1

t

t

θ θ

θ θ

11 12 13

21 22 23

0 0 1

a a a

a a a11 12 13

21 22 23

31 32 1

h h h

h h h

h h

u1

u2

u3

u4

Fun with Homography

v1 v2

v3v4

H

The image can be rectified as if it is seen from top view.

Fun with Homography u = [u1'; u2'; u3'; u4']; v = [v1'; v2'; v3'; v4']; % Need at least non-colinear four points H = ComputeHomography(v, u); im_warped = ImageWarping(im, H);

u1

u2

u3

u4

RectificationViaHomography.m

Fun with Homography u = [u1'; u2'; u3'; u4']; v = [v1'; v2'; v3'; v4']; % Need at least non-colinear four points H = ComputeHomography(v, u); im_warped = ImageWarping(im, H);

u1

u2

u3

u4

RectificationViaHomography.m

u_x = H(1,1)*v_x + H(1,2)*v_y + H(1,3); u_y = H(2,1)*v_x + H(2,2)*v_y + H(2,3); u_z = H(3,1)*v_x + H(3,2)*v_y + H(3,3); u_x = u_x./u_z; u_y = u_y./u_z; im_warped(:,:,1) = reshape(interp2(im(:,:,1), u_x(:), u_y(:)), [h, w]); im_warped(:,:,2) = reshape(interp2(im(:,:,2), u_x(:), u_y(:)), [h, w]); im_warped(:,:,3) = reshape(interp2(im(:,:,3), u_x(:), u_y(:)), [h, w]); im_warped = uint8(im_warped);

ImageWarping.m

x x

y y

1 1

H

v u

v u

u_x = H(1,1)*v_x + H(1,2)*v_y + H(1,3); u_y = H(2,1)*v_x + H(2,2)*v_y + H(2,3);

Cf) ImageWarpingEuclidean.m

Fun with Homography

H v1

v2

v3v4

u1

u2

u3

u4

Fun with Homography

u1

u2

u3

u4

Fun with Homography

Fun with Homography

Image Warping

Image Inpainting

Virtual Advertisement

Image Transform via Plane

Keller entrance left Keller entrance right

Plane

Image Transform via 3D Plane

X

YX =

0

1

u

u K R t X

Plane

Image Transform via 3D Plane

X

YX =

0

1

1 2

1 2

=

1

=

1

u K r r t

v K r r t

X

Y

X

Y

u

u K R t X

Plane

Image Transform via 3D Plane

X

YX =

0

1

1 2

1 2

=

1

=

1

u K r r t

v K r r t

X

Y

X

Y

u

u K R t X

v

Plane

Image Transform via 3D Plane

X

YX =

0

1

1 2

1 2

=

1

=

1

u K r r t

v K r r t

X

Y

X

Y

u

u K R t X

v

How are two image coordinates (u,v) related?

Plane

Image Transform via 3D Plane

X

YX =

0

1

1 2

1 2

=

1

=

1

u K r r t

v K r r t

X

Y

X

Y

u

u K R t X

v

How are two image coordinates (u,v) related?

-1-1 -1 -1

1 2 1 2

-1 -11 2 1 2

= =

1

=

=

r r t K u r r t K v

v K r r t r r t K u

v Hu

X

Y

Image Transform via 3D Plane

Keller entrance left Keller entrance right

Image Transform via 3D Plane

Keller entrance left Right image to left

Image Transform via 3D Plane

Image Transform via 3D Plane

Left image to right Right image to left

Image Transform via 3D Plane

360 Panorama

https://www.youtube.com/watch?v=H6SsB3JYqQg

Camera

Image Transform by Pure 3D Rotation

X

Camera

Image Transform by Pure 3D Rotation

X1 = u KX

Camera

Image Transform by Pure 3D Rotation

X

2 = v KRX

1 = u KX

Camera

Image Transform by Pure 3D Rotation

X

2 = v KRX

1 = u KX

-1 T -11 2= = X K u R K v

Camera

Image Transform by Pure 3D Rotation

X

2 = v KRX

1 = u KX

-1 T -11 2= = X K u R K v

-1=v KRK u

Camera

Image Transform by Pure 3D Rotation

X

2 = v KRX

1 = u KX

-1 T -11 2= = X K u R K v

-1=v KRK u

-1=H KRK

Camera

Image Transform by Pure 3D Rotation

X

2 = v KRX

1 = u KX

-1 T -11 2= = X K u R K v

-1=v KRK u

-1=H KRK-1=R K HK

Lind Hall Left Lind Hall Right

u Hv=

Euclidean Transform (Translation)

Homography

Homography

Why misalignment?

Image Panorama (Cylindrical Projection)

Image Panorama (Cylindrical Projection)

Image Panorama (Cylindrical Projection)

h

Image Panorama (Cylindrical Projection)

-1 TX = K R u

u

1R 2R 11R3R 4R 5R 6R 7R 8R 9R 10R

-1=R K HK

Image Panorama (Cylindrical Projection)

u

-1=R K HK

h

H

cos

-2

sin

u =

f

Hh

f

-1 TX = K R u

Image Panorama (Cylindrical Projection)

u

-1=R K HK

h

cos

-2

sin

u =

f

Hh

f

f

H

-1 TX = K R u

Image Panorama (Cylindrical Projection)

u

-1=R K HK

h

cos

-2

sin

u =

f

Hh

f

f

H

h-1 TX = K R u

Image Panorama (Cylindrical Projection)

u

-1=R K HK

h

cos

-2

sin

u =

f

Hh

f

f

H

h

-1 T= K R u

-1 TX = K R u

Image Panorama (Cylindrical Projection)

h