A Gentle Introduction to Bilateral Filtering and its ...yu/Teaching/CISC849_S15/handouts/bil… ·...

A Gentle Introductionto Bilateral Filteringand its Applications


Sylvain Paris – MIT CSAIL

Pierre Kornprobst – INRIA Odyssée

Jack Tumblin – Northwestern University

Frédo Durand – MIT CSAIL

• The bilateral filter is becoming in computational photography.

• Many applications with high quality results.

Photographic Style Transfer[Bae 06]Photographic Style Transfer[Bae 06]

input

Photographic Style Transfer[Bae 06]Photographic Style Transfer[Bae 06]

output

Tone Mapping[Durand 02]Tone Mapping[Durand 02]

HDR input

Tone Mapping[Durand 02]Tone Mapping[Durand 02]

output

input

Cartoon Rendition[Winnemöller 06]Cartoon Rendition[Winnemöller 06]

Cartoon Rendition[Winnemöller 06]Cartoon Rendition[Winnemöller 06]

output

6 papers atSIGGRAPH’07

6 papers atSIGGRAPH’07

Goal: Image SmoothingGoal: Image Smoothing

Split an image into:

• large-scale features, structure

• small-scale features, texture

input smoothed(structure, large scale)

residual(texture, small scale)

Gaussian Convolution

BLUR HALOS

Naïve Approach: Gaussian BlurNaïve Approach: Gaussian Blur

Impact of Blur and HalosImpact of Blur and Halos• If the decomposition introduces blur and

halos, the final result is corrupted.

Sample manipulation:increasing texture

(residual × 3)



edge-preserving: Bilateral Filter

Bilateral Filter: no Blur, no HalosBilateral Filter: no Blur, no Halos

increasing texturewith Gaussian convolution

H A L O S

increasing texturewith bilateral filter

N O H A L O S

Notation and DefinitionsNotation and Definitions

• Image = 2D array of pixels

• Pixel = intensity (scalar) or color (3D vector)

• Ip = value of image I at position: p = ( px , py )

• F [ I ] = output of filter F applied to image I

x

y

Strategy for Smoothing ImagesStrategy for Smoothing Images

• Images are not smooth because adjacent pixels are different.

• Smoothing = making adjacent pixelslook more similar.

• Smoothing strategypixel → average of its neighbors

Box AverageBox Average

average

input

square neighborhood

output

sum overall pixels q

normalizedbox function

intensity atpixel q

result atpixel p

Equation of Box AverageEquation of Box Average

∑∈

−=S

IBIBAq

qp qp )(][ σ

0

Square Box Generates Defects Square Box Generates Defects

• Axis-aligned streaks

• Blocky results

input

output

unrelatedpixels

unrelatedpixels

relatedpixels

Box ProfileBox Profile

pixelposition

pixelweight

Strategy to Solve these ProblemsStrategy to Solve these Problems

• Use an isotropic (i.e. circular) window.

• Use a window with a smooth falloff.

box window Gaussian window

Gaussian BlurGaussian Blur

average

input

per-pixel multiplication

output*

box average

Gaussian blur

normalizedGaussian function

Equation of Gaussian BlurEquation of Gaussian Blur

( )∑∈

−=S

IGIGBq

qp qp ||||][ σ

Same idea: weighted average of pixels.

0

1

unrelatedpixels

unrelatedpixels

uncertainpixels

uncertainpixels

relatedpixels

Gaussian ProfileGaussian Profile

pixelposition

pixelweight

⎟⎟⎠

⎞⎜⎜⎝

⎛−= 2

2

2exp

21)(

σπσσxxG

size of the window

Spatial ParameterSpatial Parameter

( )∑∈

−=S

IGIGBq

qp qp ||||][ σ

small σ large σ

input

limited smoothing strong smoothing

How to set σHow to set σ

• Depends on the application.

• Common strategy: proportional to image size– e.g. 2% of the image diagonal

– property: independent of image resolution

Properties of Gaussian BlurProperties of Gaussian Blur

• Weights independent of spatial location

– linear convolution

– well-known operation

– efficient computation (recursive algorithm, FFT…)

Properties of Gaussian BlurProperties of Gaussian Blur

• Does smooth images

• But smoothes too much:edges are blurred.– Only spatial distance matters

– No edge term

input

output

( )∑∈

−=S

IGIGBq

qp qp ||||][ σspace

Blur Comes from Averaging across EdgesBlur Comes from Averaging across Edges

*

*

*

input output

Same Gaussian kernel everywhere.

Bilateral FilterNo Averaging across EdgesBilateral FilterNo Averaging across Edges

*

*

*

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

space weight

not new

range weight

I

new

normalizationfactor

new

Bilateral Filter Definition:an Additional Edge TermBilateral Filter Definition:an Additional Edge Term

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

Same idea: weighted average of pixels.

Illustration a 1D ImageIllustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot

pixelintensity

pixel position

space

Gaussian Blur and Bilateral FilterGaussian Blur and Bilateral Filter

space rangenormalization

Gaussian blur

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

Bilateral filter[Aurich 95, Smith 97, Tomasi 98]

space

spacerange

p

p

q

q

( )∑∈

−=S

IGIGBq

qp qp ||||][ σ

qqpp

Bilateral Filter on a Height FieldBilateral Filter on a Height Field

output input

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

pp

reproducedfrom [Durand 02]

Space and Range ParametersSpace and Range Parameters

• space σs : spatial extent of the kernel, size of the considered neighborhood.

• range σr : “minimum” amplitude of an edge

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

Influence of PixelsInfluence of Pixels

pp

Only pixels close in space and in range are considered.

space

range

σs = 2

σs = 6

σs = 18

σr = 0.1 σr = 0.25σr = ∞

(Gaussian blur)

input

Exploring the Parameter SpaceExploring the Parameter Space

σs = 2

σs = 6

σs = 18

σr = 0.1 σr = 0.25σr = ∞

(Gaussian blur)

input

Varying the Range ParameterVarying the Range Parameter

σr = 0.1

σr = 0.25

σr = ∞(Gaussian blur)

σs = 2

σs = 6

σs = 18

σr = 0.1 σr = 0.25σr = ∞

(Gaussian blur)

input

Varying the Space ParameterVarying the Space Parameter

σs = 2

σs = 6

σs = 18

How to Set the ParametersHow to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude– e.g., mean or median of image gradients

• independent of resolution and exposure

A FewMore Advanced

Remarks

A FewMore Advanced

Remarks

Bilateral Filter Crosses Thin LinesBilateral Filter Crosses Thin Lines• Bilateral filter averages across

features thinner than ~2σs

• Desirable for smoothing: more pixels = more robust• Different from diffusion that stops at thin lines

close-up kernel

Iterating the Bilateral FilterIterating the Bilateral Filter

• Generate more piecewise-flat images

• Often not needed in computational photo.

][ )()1( nn IBFI =+

1 iteration

2 iterations

4 iterations

Bilateral Filtering Color ImagesBilateral Filtering Color Images

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

( ) ( )∑∈

−−=S

GGW

IBFq

qqpp

p CCCqp ||||||||1][rs σσ

For gray-level images

For color images

intensity difference

color difference

The bilateral filter isThe bilateral filter isextremely easy to adapt to your need.extremely easy to adapt to your need.

scalar

3D vector (RGB, Lab)

input

output

Hard to ComputeHard to Compute

• Nonlinear

• Complex, spatially varying kernels– Cannot be precomputed, no FFT…

• Brute-force implementation is slow > 10min

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ



edge-preserving: Bilateral Filter

Decomposition into Large-scale and Small-scale LayersDecomposition into Large-scale and Small-scale Layers

Weighted Average of PixelsWeighted Average of Pixels

space rangenormalization

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

space

range

p

q

• Depends on spatial distanceand intensity difference– Pixels across edges have almost influence



Efficient Implementationsof the Bilateral Filter

Sylvain Paris – MIT CSAIL

OutlineOutline

• Brute-force Implementation

• Separable Kernel [Pham and Van Vliet 05]

• Box Kernel [Weiss 06]

• 3D Kernel [Paris and Durand 06]

Brute-force ImplementationBrute-force Implementation

For each pixel pFor each pixel q

Compute

8 megapixel photo: 64,000,000,000,000 iterations!

V E R Y S L O W !V E R Y S L O W !More than 10 minutes per imageMore than 10 minutes per image

( ) ( )∑∈

−−=S

IIIGGW

IBFq

qqpp

p qp ||||||1][rs σσ

( ) ( ) qqpqp IIIGG ||||||rs

−− σσ

ComplexityComplexity

• Complexity = “how many operations are needed, how this number varies”

• S = space domain = set of pixel positions

• | S | = cardinality of S = number of pixels– In the order of 1 to 10 millions

• Brute-force implementation: )|(| 2SΟ

Better Brute-force ImplementationBetter Brute-force Implementation

Idea: Far away pixels are negligible

For each pixel pa. For each pixel q such that || p – q || < cte × σs

looking at all pixels looking at neighbors only

DiscussionDiscussion

• Complexity:

• Fast for small kernels: σs ~ 1 or 2 pixels

• BUT: slow for larger kernels

)|(| 2sσ×SΟ

neighborhood area

OutlineOutline





Separable KernelSeparable Kernel

• Strategy: filter the rows then the columns

• Two “cheap” 1D filters instead of an “expensive” 2D filter

[Pham and Van Vliet 05]


• Complexity: – Fast for small kernels (<10 pixels)

• Approximation: BF kernel not separable– Satisfying at strong edges and uniform areas

– Can introduce visible streaks on textured regions

)|(| sσ×SΟ

brute-forceimplementation

separable kernelmostly OK,

some visible artifacts(streaks)

OutlineOutline





Box KernelBox Kernel

• Bilateral filter with a square box window

• The bilateral filter can be computed only from the list of pixels in a square neighborhood.

[Weiss 06]

( ) ( )∑∈

−−=S

IIIGBW

IYq

qqpp

p qp ||||||1][rs σσ

[Yarovlasky 85]

box window

( )∑∈

−=s

r||1][

σ

σB

IIIGW

IYq

qqpp

p

restrict the sum

independent of position q

Box KernelBox Kernel• Idea: fast histograms of square windows

[Weiss 06]

input:full histogram is known

update:add one line, remove one line

Tracking one window

Box KernelBox Kernel• Idea: fast histograms of square windows

[Weiss 06]

input:full histograms are known

update:add one line, remove one line,

add two pixels, remove two pixels

Tracking two windows at the same time


• Complexity: – always fast

• Only single-channel images

• Exploit vector instructions of CPU

• Visually satisfying results (no artifacts)– 3 passes to remove artifacts due to

box windows (Mach bands)

)log|(| sσ×SΟ

1 iteration

3 iterations

box kernelvisually different,

yet no artifacts

OutlineOutline





3D Kernel3D Kernel

• Idea: represent image data such that the weights depend only on the distance between points

[Paris and Durand 06]

pixelintensity

pixel position

1D image

PlotI = f ( x )

far in range

close in space

1st Step: Re-arranging Symbols1st Step: Re-arranging Symbols

( ) ( )

( ) ( )∑

∑

∈

∈

−−=

−−=

S

S

IIGGW

IIIGGW

IBF

qqpp

qqqp

pp

qp

qp

||||||

||||||1][

rs

rs

σσ

σσ

( ) ( )

( ) ( ) 1||||||

||||||][

rs

rs

∑

∑

∈

∈

−−=

−−=

S

S

IIGGW

IIIGGIBFW

qqpp

qqqppp

qp

qp

σσ

σσ

Multiply first equation by Wp

1st Step: Summary1st Step: Summary

• Similar equations

• No normalization factor anymore

• Don’t forget to divide at the end

( ) ( )

( ) ( ) 1||||||

||||||][

rs

rs

∑

∑

∈

∈

−−=

−−=

S

S

IIGGW

IIIGGIBFW

qqpp

qqqppp

qp

qp

σσ

σσ

2nd Step: Higher-dimensional Space2nd Step: Higher-dimensional Space

pp

space

range

• “Product of two Gaussians” = higher dim. Gaussian


pp

space

range

• 0 almost everywhere, I at “plot location”


pp


• Weighted average at each point = Gaussian blur


pp


• Weighted average at each point = Gaussian blur

• Result is at “plot location”

higher dimensional functions

Gaussian blur

division

slicing

Higherdimensional

Homogeneousintensity

Higherdimensional

Homogeneousintensity

New num. scheme:• simple operations• complex space

higher dimensional functions

Gaussian convolution

division

slicing

D O W N S A M P L E

U P S A M P L E

Heavilydownsampled

Heavilydownsampled

Strategy:downsampledconvolution

Conceptual view,not exactly the actual algorithm

Actual AlgorithmActual Algorithm

• Never compute full resolution– On-the-fly downsampling

– On-the-fly upsampling

• 3D sampling rate = ),,( rss σσσ

Pseudo-code: StartPseudo-code: Start

• Input– image I

– Gaussian parameters σs and σr

• Output: BF [ I ]

• Data structure: 3D arrays wi and w (init. to 0)

Pseudo-code: On-the-fly DownsamplingPseudo-code: On-the-fly Downsampling

• For each pixel

– Downsample:

– Update:

SYX ∈),(

⎟⎟⎠

⎞⎜⎜⎝

⎛⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡⎥⎦

⎤⎢⎣

⎡=

rss

),(,,),,(σσσ

YXIYXzyx

1),,(),(),,(

=+=+

zyxwYXIzyxwi

D O W N S A M P L E

U P S A M P L E

[ ] = closest int.

Pseudo-code: ConvolvingPseudo-code: Convolving• For each axis , , and

– For each 3D point

• Apply a Gaussian mask ( 1 , 4 , 6 , 4 , 1 ) to wi and we.g., for the x axis:

wi’(x) = wi(x-2) + 4.wi(x-1) + 6.wi(x) + 4.wi(x+1) + wi(x+2)

xr yr zr

),,( zyx

D O W N S A M P L E

U P S A M P L E

Pseudo-code: On-the-fly UpsamplingPseudo-code: On-the-fly Upsampling

• For each pixel

– Linearly interpolate the values in the 3D arrays

SYX ∈),(

( )( )),(,,,einterpolat

),(,,,einterpolat),(][YXIYXwYXIYXwiYXIBF =

D O W N S A M P L E

U P S A M P L E


• Complexity:

• Fast for medium and large kernels– Can be ported on GPU [Chen 07]: always very fast

• Can be extended to color images but slower

• Visually similar to brute-force computation

⎟⎟⎠

⎞⎜⎜⎝

⎛+

r2s

||||||σσRSSΟ

numberof pixels

numberof 3D cells

| R | : number of gray levels

3D kernelvisually similar

Running TimesRunning Times

separable kernel

3D kernel box kernel

brut

e fo

rce

How to Choose an Implementation?How to Choose an Implementation?

Depends a lot on the application. A few guidelines:

• Brute-force: tiny kernels or if accuracy is paramount

• Box Kernel: for short running times on CPU with any kernel size, e.g. editing package

• 3D kernel: – if GPU available

– if only CPU available: large kernels, color images, cross BF(e.g., good for computational photography)

A Gentle Introduction to Bilateral Filtering and its ...yu/Teaching/CISC849_S15/handouts/bil… ·...

Documents

Transcript of A Gentle Introduction to Bilateral Filtering and its ...yu/Teaching/CISC849_S15/handouts/bil… ·...