Fast Bilateral Filtering
description
Transcript of Fast Bilateral Filtering
![Page 2: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/2.jpg)
• Manipulating the texture in a photo isa central operation in many tasks.
• Let’s look at a few examples…
![Page 3: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/3.jpg)
Photographic Style Transfer [Bae 06]
input
![Page 4: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/4.jpg)
Photographic Style Transfer [Bae 06]
output
![Page 5: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/5.jpg)
Tone Mapping[Durand 02]
HDR input
![Page 6: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/6.jpg)
Tone Mapping[Durand 02]
output
![Page 7: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/7.jpg)
input
Cartoon Rendition[Winnemöller 06]
![Page 8: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/8.jpg)
Cartoon Rendition[Winnemöller 06]
output
![Page 9: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/9.jpg)
input smoothed(structure, large scale)
residual(texture, small scale)
Gaussian Convolution
BLUR HALOS
Naïve Approach: Gaussian Blur
![Page 10: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/10.jpg)
Impact of Blur and Halos
• If the decomposition introduces blur and halos, the final result is corrupted.
Sample manipulation:increasing texture
(residual 3)
![Page 11: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/11.jpg)
input smoothed(structure, large scale)
residual(texture, small scale)
edge-preserving: Bilateral Filter
Bilateral Filter: no Blur, no Halos
![Page 12: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/12.jpg)
input
![Page 13: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/13.jpg)
increasing texturewith Gaussian convolution
H A L O S
![Page 14: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/14.jpg)
increasing texturewith bilateral filter
N O H A L O S
![Page 15: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/15.jpg)
Traditional Denoising versus Computational Photography
Edge-preserving filtering introduced for denoising.
• Denoising: decompose into signal + noise – Throw away noise– Small kernels
• Computational photography: decompose into base + detail– Detail is valuable– Large kernels Bilateral filter [Aurich 95, Smith 97, Tomasi 98]
![Page 16: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/16.jpg)
Objective of bilateral filtering
Smooth texture
Preserve edges
![Page 17: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/17.jpg)
Illustration a 1D Image
• 1D image = line of pixels
• Better visualized as a plot
pixelintensity
pixel position
![Page 18: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/18.jpg)
Definition
space rangenormalization
space
Gaussian blur
Bilateral filter[Aurich 95, Smith 97, Tomasi 98]
• only spatial distance, intensity ignored
• spatial and range distances• weights sum to 1
space
spacerange
p
p
q
q
![Page 19: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/19.jpg)
Example on a Real Image
• Kernels can have complex, spatially varying shapes.
input output
![Page 20: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/20.jpg)
Bilateral Filter is Expensive
• Brute-force computation is slow (several minutes)– Two nested for loops:
for each pixel, look at all pixels– Non-linear, depends on image content
no FFT, no pre-computation…
• Fast approximations exist [Durand 02, Weiss 06]
– Significant loss of accuracy– No formal understanding of accuracy versus speed
![Page 21: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/21.jpg)
Today• We will reformulate the bilateral filter
– Link with linear filtering
– Fast and accurate algorithm
![Page 22: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/22.jpg)
Questions ?
![Page 23: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/23.jpg)
Outline
• Reformulation of the BF
• Fast algorithm to compute the BF
• Practical implementation
• Application and extension– Photographic style transfer– Bilateral grid
![Page 24: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/24.jpg)
Bilateral Filter on 1D Signal
BF
![Page 25: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/25.jpg)
p
Our Strategy
Reformulate the bilateral filter– More complex space:
Homogeneous intensity Higher-dimensional space
– Simpler expression: mainly a convolution Leads to a fast algorithm
weightsappliedto pixels
![Page 26: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/26.jpg)
p
Link with Linear Filtering1. Handling the Division
sum ofweights
Handling the division with a projective space.
![Page 27: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/27.jpg)
Formalization: Handling the Division
• Normalizing factor as homogeneous coordinate• Multiply both sides by
![Page 28: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/28.jpg)
Formalization: Handling the Division
• Similar to homogeneous coordinates in projective space
• Division delayed until the end
with Wq=1
![Page 29: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/29.jpg)
Questions ?
![Page 30: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/30.jpg)
space range
p
Link with Linear Filtering2. Introducing a Convolution
q
2D Gaussian
![Page 31: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/31.jpg)
p
Link with Linear Filtering2. Introducing a Convolution
q
space x range
Corresponds to a 3D Gaussian on a 2D image.Result appeared previously in [Barash 02].
![Page 32: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/32.jpg)
Link with Linear Filtering2. Introducing a Convolution
space-range Gaussian
sum all values
sum all values multiplied by kernel convolution
![Page 33: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/33.jpg)
space-range Gaussian
result of the convolution
Link with Linear Filtering2. Introducing a Convolution
![Page 34: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/34.jpg)
Link with Linear Filtering2. Introducing a Convolution
space-range Gaussian
result of the convolution
![Page 35: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/35.jpg)
higher dimensional functions
Gaussian convolution
division
slicing
w i w
![Page 36: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/36.jpg)
Reformulation: Summary
1. Convolution in higher dimension• expensive but well understood (linear, FFT, etc)
2. Division and slicing• nonlinear but simple and pixel-wise
Exact reformulation
![Page 37: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/37.jpg)
Questions ?
![Page 38: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/38.jpg)
Outline
• Reformulation of the BF
• Fast algorithm to compute the BF
• Practical implementation
• Application and extension– Photographic style transfer– Bilateral grid
![Page 39: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/39.jpg)
higher dimensional functions
Gaussian convolution
division
slicing
Higherdimensional
Homogeneousintensity
w i w
Recap:• simple operations• complex space
![Page 40: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/40.jpg)
higher dimensional functions
Gaussian convolution
division
slicing
w i w
D O W N S A M P L E
U P S A M P L E
Heavilydownsampled
Strategy:downsampledconvolution
![Page 41: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/41.jpg)
Sampling Theorem
• Sampling a signal at a least twice its smallest wavelength is enough.
Not enough
![Page 42: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/42.jpg)
Sampling Theorem
• Sampling a signal at a least twice its smallest wavelength is enough.
Not enough
![Page 43: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/43.jpg)
Sampling Theorem
• Sampling a signal at a least twice its smallest wavelength is enough.
Enough
![Page 44: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/44.jpg)
higher dimensional functions
Gaussian convolution
division
slicing
Low-pass filter Almost only
low freq. High freq. negligible
w i w
Signal processinganalysis
![Page 45: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/45.jpg)
higher dimensional functions
Gaussian convolution
division
slicing
w i w
D O W N S A M P L E
U P S A M P L E
Almost noinformation
loss
“Safe” downsampling
![Page 46: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/46.jpg)
Fast Convolution by Downsampling• Downsampling cuts frequencies
above Nyquist limit (half the sampling rate)– Less data to process– But introduces error
• Evaluation of the approximation
• Efficient implementation
![Page 47: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/47.jpg)
Accuracy versus Running Time1 second instead of several minutes• Finer sampling increases accuracy.• More precise than previous work.
finer sampling
Accuracy as function of Running Time Digital photograph
1200 1600
Brute-force bilateral filter takes over 10 minutes.
![Page 48: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/48.jpg)
Visual Results
input exact BF our result prev. work1200 1600
• Comparison with previous work [Durand 02]– running time = 1s for both techniques
0
0.1differencewith exact
computation(intensities in [0:1])
![Page 49: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/49.jpg)
More on Accuracy and Running Times
![Page 50: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/50.jpg)
Kernel Size• Larger kernels are faster
because we downsample more. Useful for photography.
Running Time as a function of Kernel SizeDigital
photograph1200 1600
Brute-force bilateral filter takes over 10 minutes.
![Page 51: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/51.jpg)
Questions ?
![Page 52: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/52.jpg)
Outline
• Reformulation of the BF
• Fast algorithm to compute the BF
• Practical implementation
• Application and extension– Photographic style transfer– Bilateral grid
![Page 53: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/53.jpg)
Efficient Implementation
• Never build the full resolution 3D space– Bin pixels on the fly– Interpolate on the fly
• Separable Gaussian kernel
• 5-tap approximation
![Page 54: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/54.jpg)
Sampling Rate
• 1 sample every sigma
– Kernel parameters are all equal to 1 sample
– 5-tap approximation is sufficient
1 – 4 – 6 – 4 – 1
![Page 55: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/55.jpg)
Initialize the 3D grid to 0.
Useful later to save space.
![Page 56: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/56.jpg)
Initialize the 3D grid to 0.
Useful later to save space.
![Page 57: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/57.jpg)
Look at each pixel.
Create the data.
Computethe grid location.
Update the grid.
![Page 58: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/58.jpg)
Look at each pixel.
Create the data.
Computethe grid location.
Update the grid.
![Page 59: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/59.jpg)
Look at each pixel.
Create the data.
Computethe grid location.
Update the grid.
![Page 60: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/60.jpg)
Convolve with 1 – 4 – 6 – 4 – 1 along each axis.3 for loops needed, one for each axis.
In 3D, 15 samples (3 times 5) considered instead of 125 (53) for a full convolution.Same result!
![Page 61: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/61.jpg)
Look at each pixel.
![Page 62: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/62.jpg)
Look at each pixel.
![Page 63: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/63.jpg)
Comments• Every sample is processed even if empty
– The grid is coarse and fairly dense in 3D.• e.g. parameters (16,0.1): 256 pixels for 10 bins
convolution spans 5 binsat least 50% occupancy
– Simple data structure, simple sweep fast
– More sophisticated approaches needed for higher dimensional cases[Adams et al. 09,10] [Gastal and Oliveira 11,12]
![Page 64: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/64.jpg)
Complexity
• There is no nested loops over the whole set of samples– At most: “for each sample, for 5 samples”
Creation + slicing“for each pixel”
Convolution“for each grid sample”
![Page 65: Fast Bilateral Filtering](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ad9550346895dc8a679/html5/thumbnails/65.jpg)
References•Short version: A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. Sylvain Paris and Frédo Durand. ECCV 2006
•Long version: A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. Sylvain Paris and Frédo Durand. International Journal of Computer Vision, 2009
•Different presentation, more applications: Real-time Edge-Aware Image Processing with the Bilateral Grid. Jiawen Chen, Sylvain Paris, and Frédo Durand. SIGGRAPH 2007
•Survey: Bilateral Filtering: Theory and Applications. Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand. Foundations and Trends in Computer Graphics and Vision, 2009
•Data, code, etc: http://people.csail.mit.edu/sparis/bf/