Streaming Multigridfor Gradient-Domain Operations

on Large Images

Johns Hopkins University Misha Kazhdan

Microsoft ResearchHugues Hoppeand

Stitching Image PanoramasDifferent exposures Seams in the panorama

Courtesy of Uyttendaele

Gradient Domain Fix:Set seam-crossing gradients to zero.

High Dynamic Range CompressionGradient Domain Fix:Amplify small gradients and dampen large ones.

Medium ExposureShort ExposureCourtesy of Industrial Light & Magic

Long Exposure

No single image exposes all the detail

Tone-Mapped Image

Solver

Image0

Image1

Imagen

ImageGradientDomain

Operator

Grad0

Gradn

Grad1 Grad

Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.

Solver

Image0

Image1

Imagen

ImageGradientDomain

Operator

Grad0

Gradn

Grad1 Grad

Solver

Image0

Image1

Imagen

ImageGradientDomain

Operator

Grad0

Gradn

Grad1 Grad

Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.

Removing Lighting EffectsRange CompressionImage CompositingImage StitchingImage Authoring

[Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05][Fattal ’02, Weyrich ’07][Perez ’03, Agarwala ’04, Jia ’06][Levin ’04, Agarwala ’07][McCann ’08]

Solver

Image0

Image1

Imagen

ImageGradientDomain

Operator

Grad0

Gradn

Grad1 Grad

Solver

Image0

Image1

Imagen

ImageGradientDomain

Operator

Grad0

Gradn

Grad1 Grad

Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.

Image0

Image1

Imagen

ImageGradientDomain

Operator

Grad0

Gradn

Grad1 SolverGrad

Removing Lighting EffectsRange CompressionImage CompositingImage StitchingImage Authoring

[Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05][Fattal ’02, Weyrich ’07][Perez ’03, Agarwala ’04, Jia ’06][Levin ’04, Agarwala ’07][McCann ’08]

Outline

• Introduction• Solving for the Image– The Poisson Equation– Solving the Linear System

• Choosing the System• Solving for Big Images• Results and Discussion

The Poisson EquationGradient-domain processing requires fitting an image to local-difference constraints:

1

gu

fu

g uf=div(g)

)(div)(div gu

1

averaged^

Gauss-Seidel SolversGradient-domain processing requires fitting an image to local-difference constraints.

Iterative Algorithm:

For each pixel (i,j)1. Assume the rest of the pixel values are correct2. Update u[i][j] to satisfy the Laplacian:

u[i][j] (Average of Neighbors) + f[i][j]

gu

fu

averaged^

Gauss-Seidel Solvers

This quickly solves for the high frequencies, but the low frequencies converge slowly.

Desired Solution

10 Iterations 100 Iterations10 IterationResidual

100 IterationResidual

2 Iterations2 IterationResidual

Multigrid SolversV-Cycle Algorithm:The process is recursed for the low-res problem.

DesiredBest-Guess

Iterated

Residual

1. Perform a few updates on the high-res image2. Compute the residual constraints

Desired

3. Down-sample, solve, and up-sample

SolvedUp-Sampled

Best-Guess

4. Sum the low-res and high-res results

Iterated

5. Refine the high-res image

+

-

Outline

• Introduction• Solving for the Image• Choosing the System• Solving for Big Images• Results and Discussion

Choosing the SystemThe choice depends on the interpretation of pixels as elements of a continuous function.

Pixel values 0th-order interpretation1st-order interpretation2nd-order interpretation3rd-order interpretation

Elements

Choosing the SystemImplementing Up-/Down-Sampling:The nesting of the elements determines the transition between the levels of the solver.

0th-order elements 1st-order elements 2nd-order elements 3rd-order elements1 1

1/2

1

1/21/4

3/4 3/4

1/4 1/81/2

3/41/2

1/8

Choosing the SystemDefining the Laplacian:The neighborhood of the Laplacian is defined by the element overlap.

Undefined

0th-order elements 1st-order elements 2nd-order elements 3rd-order elements

1-Ring Average 2-Ring Average 3-Ring Average

Second-Order ElementsWhich order elementconverges fastest to anaccurate solution?

Original Image Image Gradients

1st-order error

2nd-order error

3rd-order error0 5

1st-Order2nd-Order3rd Order

256

1

1/256

Average Error

GSUpdates

Second-Order Elements

Spectral Analysis of Convergence:The eigenvalues of the multigrid operator determine how efficiently the residual is dampened.

0 320.0

0.2

0.4

Two-Grid Spectra (k=5)

Eigenvalue Index

Eigenvalue

1st-order2nd-order3rd-order

G.S Update

G.S Update

G.S Update

Two-Grid Operator

Down-Sample

Up-Sample

Outline

• Introduction• Solving for the Image• Choosing the System• Solving for Big Images– Naïve Out-of-Core Multigrid– Temporal Blocking– Interleaved Multilevel Streaming

• Results and Discussion

Naïve Out-of-Core MultigridComponents of the solver:

1. Gauss-Seidel updates2. Down-sampling3. Up-sampling

The multigrid solvercan be streamed byadvancing a fixed-size window.

Local data access

On Disk

In Memory

Naïve Out-of-Core MultigridThe large linear system can be solved by streaming each step of the V-cycle.

Low-Res Solve

Down-Sample

Down-Sample

Up-Sample

Up-Sample

G.S.Update

Solution

G.S. Update

G.S. Update

k k

kk

G.S. Update

Constraints

Disk Streaming

k Gauss-Seidel Updates

2(k+2) Streaming Passes

Residual

Residual

Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.neighbor

Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.

000

0 1

0

0 1

0

0 1

1 2

0

0 1

1 2

0

0 1

1 2

2

neighbor

Win

dow

k=2 Gauss-Seidel Updates

2nd-order elements

Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.

SolutionConstraints

neighbor

Low-Res Solve

Down-Sample

Down-Sample

Up-Sample

Up-Sample

G.S.Update

G.S. Update

G.S. Update

G.S. Update

Residual

Residual

Disk Streaming

Interleaved Multilevel StreamingProcessing at one level is buffered for the next.

piped into

Processing at one level is buffered for the next.

1/4

3/4 3/4

1/4

¾¾¼

¼

Interleaved Multilevel Streaming

000

0 1

0

0 1

0

0 1

1 2

0

0 1

1 2

0

0 1

1 2

0

2

0 1

1 2

0

0 1

1 2

2

0

0 1

1 2

0

2

0 1

1 2

00

0 11 2

0 1

1 2

20000 1

¾¾¼

¼¾¾¼

¼

Resid

ual

High-Resolution Low-Resolution

2nd-order elements

piped into

Interleaved Multilevel StreamingProcessing at one level is buffered for the next.

SolutionConstraints

Low-Res Solve

Down-Sample

Down-Sample

Up-Sample

Up-Sample

G.S.Update

G.S. Update

G.S. Update

G.S. Update

Residual

Residual

Streaming Pass 1 Streaming Pass 2

Globality of the Poisson equation

Two streaming passes is optimal

Disk StreamingMemory Streaming

Disk Streaming

piped into

For gradient-domain operators that are local, processing only takes two streaming passes.

+1 streaming pass for each additional V-cycle.

Streaming through the Gradient-Domain

Pass 3uu

Update

Updatef

Solve

Update…

u0

u1

g0

g1 g

un gn

Op…

Update

…

Memory StreamingDisk Streaming

Pass 1 Pass 2

Solve

Update Update

UpdateUpdate

Outline

• Introduction• Solving for the Image• Choosing the System• Solving for Big Images• Results and Discussion

Stitching Big Images

Courtesy of Agarwala

Agarwala ’07:A high-res solution is only required near the seams, so solve over an adapted quadtree.

Stitching Results

Courtesy of Agarwala and Curless

Beynac: 6,0461,920 Rainier: 10,9602,096

Edinburgh: 16,9502,956

Redrock: 19,5884,457

Dataset Pixels ImagesError [0,256) Memory (MB) Time (seconds)

Quadtree Streaming Quadtree Streaming Quadtree Streaming

Beynac 12x106 3 0.01 0.01 16 190 8 17

Rainier 23x106 5 0.02 0.01 17 110 14 33

Edinburgh 50x106 25 0.01 123 203 122 79

Redrock 87x106 9 0.01 112 133 118 118

Tone-Mapping Results

Bring out subtle details by amplifying small gradients and dampening large ones.

Large Image ResultsSt James:• Stitched from 643 photographs• Contains 3.3 billion (88,309 x 37,842) pixels

Tone-MappedMemory: 224 MBTime: 45:10Max Error [0,256): 0.5 Courtesy of Uyttendaele

StitchedMemory: 408 MBTime: 1:27:50Max Error [0,256): 0.15

Large Image Results

Courtesy of Uyttendaele

St James:• Peak Memory: 408 MB• Solver Time: 1:27:50

Total disk I/O:• Down-Sampling

– Read in the gradient constraints: 40 GB– Write out solution and Laplacian: 53 GB

• Up-Sampling– Read in solution and Laplacian: 53 GB– Write out the image: 20 GB

53GB in-core1:19:02 I/O Time

Summary

Streaming multigrid for big image processing:– 2nd order elements– Temporal blocking– Interleaved streaming

Issues addressed in the paper:– Integration with finite-differences– Setting the average color–Non power-of-two images

Accurate solution in two streaming passes

Issues for Future Work

1. Non-trivial boundaries and constraints?e.g. cut-and-paste, matting

2. Adaptive weighting?e.g. local tone adjustment

3. Extensions to higher dimensions?for incompressible fluid simulation

4. Less I/O in the streaming passes?on-the-fly compression

5. Really big images?parallelized and distributed solvers

AcknowledgmentsDatasets:

Aseem AgarwalaBrian CurlessMatt Uyttendaele

Discussions and Ideas:Ketan Dalal Rick SzeliskiBill Bolosky Ann WolvertonCAVGRAPH Reviewers SIGGRAPH Reviewers

Code & Images @ http://www.cs.jhu.edu/~misha/Code/SMG

Thank You!

General Temporal BlockingTo perform k updates:

Updating the 3rd row from front,The row two behind that,The row two behind that,

000

0 1

0

0 1

0

0 1

1 2

0

0 1

1 2

0

0 1

1 2

2 33

k=3 Gauss-Seidel Updates

00

0 1

1 2

2 3

0 1

1 2

2 3

k

Win

dow

…

General Temporal BlockingTo perform k updates:

Updating the 3rd row from front,The row two behind that,The row two behind that,…

k=3 Gauss-Seidel Updates

k

with d-th order elements^

3

00

0 1

1 2

2 3

0 1

1 2

2 3

Win

dow

(d+1)-stdd

Finite Differences to Finite Elements

The derivative of a d-th order element is the difference of elements of order d-1.

Finite Differences

1st-order Element

0th-order Elements

Coefficients of Derivative ElementsDifferentiation

Second-Order ElementsWhich order elementconverges fastest to anaccurate solution?

Original Image Image Gradients

1st-order error

2nd-order error

3rd-order error0 5 10 15 20

1st-Order2nd-Order3rd-Order

GS Updates

256

1

1/256

Average Error256

Average Error

0 1000

1st-Order2nd-Order3rd-Order

Operations

256Average Error256

1

1/256

Average Error

1/2561/256