CSE486, Penn StateRobert Collins

Lecture 17:

Mosaicing and Stabilization

CSE486, Penn StateRobert Collins

Recall: Planar Projection

x

y

Point on planeRotation + Translation

Perspectiveprojection

Pixel coordsu

v

Internalparams

Homography

CSE486, Penn StateRobert Collins

Destination image

Recall: Projective (un)Warping

from Hartley & Zisserman

Source Image

H

CSE486, Penn StateRobert Collins

Recall : Planar Projection

H2H1

H

CSE486, Penn StateRobert Collins

Applications: Stabilization

Given a sequence of video frames, warp theminto a common image coordinate system.

This “stabilizes” the video to appear as if thecamera is not moving.

frame k k+1 k+2k-1k-2

destinationframe

CSE486, Penn StateRobert Collins

Stabilization Example

VIVID project

CSE486, Penn StateRobert Collins

Stabilization by Chaining

What if the reference image does not overlap with all the source images? As long as there are pairwise overlaps, wecan chain (compose) pairwise homographies.

frame k k+1 k+2

destination

k+3 k+4

Not recommended for long sequences, as alignment errors accumulate over time.

H43H3

2H21H1

0

H40 = H1

0 * H21 * H3

2 * H43

CSE486, Penn StateRobert Collins

Applications: Mosaicing

from Hartley & Zisserman

Destination imageSource 1 Source 2

Mosaic

H1 H2

CSE486, Penn StateRobert Collins

Note on Planar Mosaicing

Assumes scene is roughly planar.

What if scene isn’t planar? Alignment will not be good if significant 3D relief

“Ghosting”

CSE486, Penn StateRobert Collins

Ghosting Example

Source image Reference imageH

CSE486, Penn StateRobert Collins

Ghosting Example (cont)

Mosaic

CSE486, Penn StateRobert Collins

Mosaics from Rotating Cameras

However, there is a mitigating factorin regards to ghosting…

Images taken from a rotating camera are related by a 2D homography…

regardless of scene structure!

CSE486, Penn StateRobert Collins

Rotating Camera (top-down view)

CSE486, Penn StateRobert Collins

Rotating Camera (top-down view)

CSE486, Penn StateRobert Collins

Rotating Camera (top-down view)

CSE486, Penn StateRobert Collins

Rotating Camera (top-down view)Rays in camera coord system are invariant!

CSE486, Penn StateRobert Collins

Rotating Camera (top-down view)Rays in camera coord system are invariant!

CSE486, Penn StateRobert Collins

Rotating Camera (top-down view)Rays in camera coord system are invariant!

CSE486, Penn StateRobert Collins

Special Case : Rotating Camera

Relative Rotation of camera Translation is 0

This is important!

Relative R,TProjectionInternalparams

CSE486, Penn StateRobert Collins

Special Case : Rotating Camera

Relative R,TProjectionInternalparams

CSE486, Penn StateRobert Collins Relations among Images

Taken by Rotating Camera

Image 1

Image 2

Same ray!

-1

CSE486, Penn StateRobert Collins

Mosaicing ExampleOriginal Images (from a pan/tilt camera)

Panoramic (Mosaic) View

CSE486, Penn StateRobert Collins

One more detail: Blending!

CSE486, Penn StateRobert Collins

Approaches to Blending

1) Straight averaging

2) Feathering

3) Equalize intensity statistics (gain, offset)

P = (P1 + P2) / 2

P = (w1*P1 + w2*P2 ) / (w1+w2)

How to combine colors in area of overlap?

With wi being distance from image border

CSE486, Penn StateRobert Collins

Before and After Blending

CSE486, Penn StateRobert Collins

360 Degree Panoramas?

Problem: Can’t just choose a reference imageto map all other images to.

Solution: Use cylindrical or spherical mosaicsurface rather than a plane.

CSE486, Penn StateRobert Collins

Panorama Input images

Sarnoff

CSE486, Penn StateRobert Collins

Spherical Panorama Result

Sarnoff

CSE486, Penn StateRobert Collins

Panorama Unwarped

Sarnoff

CSE486, Penn StateRobert Collins

Quicktime VR

This example from www.panoguide.com/gallery/

Also big list at http://www.multimedialibrary.com/diana/qtvr_sites.asp

CSE486, Penn StateRobert Collins

Quicktime VR

CSE486, Penn StateRobert Collins

Quicktime VR

This example from www.ems.psu.edu/~fraser/qtvr/