Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... ·...
Transcript of Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... ·...
![Page 1: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/1.jpg)
Course Evaluations
h"p://www.siggraph.org/courses_evalua4on
4 Random Individuals will win an ATI Radeontm HD2900XT
![Page 2: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/2.jpg)
A Gentle Introduction to Bilateral Filtering and its Applications
• From Gaussian blur to bilateral filter – S. Paris
• Applications – F. Durand
• Link with other filtering techniques – P. Kornprobst
• Implementation – S. Paris
• Variants – J. Tumblin
• Advanced applications – J. Tumblin
• Limitations and solutions – P. Kornprobst
BREAK
![Page 3: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/3.jpg)
A Gentle Introduction to Bilateral Filtering and its Applications
Recap
Sylvain Paris – Adobe
![Page 4: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/4.jpg)
input smoothed (structure, large scale)
residual (texture, small scale)
edge-preserving: Bilateral Filter
Decomposition into Large-scale and Small-scale Layers
![Page 5: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/5.jpg)
Weighted Average of Pixels
space range normalization
space
range
p
q
• Depends on spatial distance and intensity difference – Pixels across edges have almost influence
![Page 6: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/6.jpg)
A Gentle Introduction to Bilateral Filtering and its Applications
Efficient Implementations of the Bilateral Filter
Sylvain Paris – Adobe
![Page 7: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/7.jpg)
Outline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
![Page 8: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/8.jpg)
Brute-force Implementation
For each pixel p For each pixel q
Compute
8 megapixel photo: 64,000,000,000,000 iterations!
![Page 9: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/9.jpg)
Complexity
• Complexity = “how many operations are needed, how this number varies”
• S = space domain = set of pixel positions
• | S | = cardinality of S = number of pixels – In the order of 1 to 10 millions
• Brute-force implementation:
![Page 10: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/10.jpg)
Better Brute-force Implementation
Idea: Far away pixels are negligible
For each pixel p
a. For each pixel q such that || p – q || < cte × σs
looking at all pixels looking at neighbors only
![Page 11: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/11.jpg)
Discussion
• Complexity:
• Fast for small kernels: σs ~ 1 or 2 pixels
• BUT: slow for larger kernels
neighborhood area
![Page 12: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/12.jpg)
Outline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
![Page 13: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/13.jpg)
Separable Kernel
• Strategy: filter the rows then the columns
• Two “cheap” 1D filters instead of an “expensive” 2D filter
[Pham and Van Vliet 05]
![Page 14: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/14.jpg)
Discussion
• Complexity: – Fast for small kernels (<10 pixels)
• Approximation: BF kernel not separable – Satisfying at strong edges and uniform areas
– Can introduce visible streaks on textured regions
![Page 15: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/15.jpg)
input
![Page 16: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/16.jpg)
brute-force implementation
![Page 17: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/17.jpg)
separable kernel mostly OK,
some visible artifacts (streaks)
![Page 18: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/18.jpg)
Outline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
![Page 19: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/19.jpg)
Box Kernel
• Bilateral filter with a square box window
• The bilateral filter can be computed only from the list of pixels in a square neighborhood.
[Weiss 06]
[Yarovlasky 85]
box window restrict the sum
independent of position q
![Page 20: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/20.jpg)
Box Kernel • Idea: fast histograms of square windows
[Weiss 06]
input: full histogram is known
update: add one line, remove one line
Tracking one window
![Page 21: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/21.jpg)
Box Kernel • Idea: fast histograms of square windows
[Weiss 06]
input: full histograms are known
update: add one line, remove one line,
add two pixels, remove two pixels
Tracking two windows at the same time
![Page 22: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/22.jpg)
Discussion
• Complexity: – always fast
• Only single-channel images
• Exploit vector instructions of CPU
• Visually satisfying results (no artifacts) – 3 passes to remove artifacts due to
box windows (Mach bands)
1 iteration
3 iterations
![Page 23: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/23.jpg)
Bilateral Filtering in O(1) [Porikli CVPR’08]
• Uses integral histograms to remove the log
• Uses Taylor expansion and power images
• Memory intensive (1 histogram per pixel)
![Page 24: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/24.jpg)
input
![Page 25: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/25.jpg)
brute-force implementation
![Page 26: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/26.jpg)
box kernel visually different,
yet no artifacts
![Page 27: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/27.jpg)
Outline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
![Page 28: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/28.jpg)
3D Kernel
• Idea: represent image data such that the weights depend only on the distance between points
[Paris and Durand 06]
pixel intensity
pixel position
1D image
Plot I = f ( x )
far in range
close in space
![Page 29: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/29.jpg)
1st Step: Re-arranging Symbols
Multiply first equation by Wp
![Page 30: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/30.jpg)
1st Step: Summary
• Similar equations
• No normalization factor anymore
• Don’t forget to divide at the end
![Page 31: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/31.jpg)
2nd Step: Higher-dimensional Space
space
range
• “Product of two Gaussians” = higher dim. Gaussian
![Page 32: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/32.jpg)
2nd Step: Higher-dimensional Space
space
range
• 0 almost everywhere, I at “plot location”
![Page 33: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/33.jpg)
2nd Step: Higher-dimensional Space • 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
![Page 34: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/34.jpg)
2nd Step: Higher-dimensional Space • 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
• Result is at “plot location”
![Page 35: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/35.jpg)
higher dimensional functions
Gaussian blur
division
slicing
New num. scheme: • simple operations • complex space
Higher dimensional
Homogeneous intensity
![Page 36: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/36.jpg)
higher dimensional functions
Gaussian convolution
division
slicing
D O W N S A M P L E
U P S A M P L E
Strategy: downsampled convolution
Conceptual view, not exactly the actual algorithm
Heavily downsampled
![Page 37: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/37.jpg)
Actual Algorithm
• Never compute full resolution – On-the-fly downsampling
– On-the-fly upsampling
• 3D sampling rate =
![Page 38: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/38.jpg)
Pseudo-code: Start
• Input – image I
– Gaussian parameters σs and σr
• Output: BF [ I ]
• Data structure: 3D arrays wi and w (init. to 0)
![Page 39: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/39.jpg)
Pseudo-code: On-the-fly Downsampling
• For each pixel
– Downsample:
– Update:
D O W N S A M P L E
U P S A M P L E
[ ] = closest int.
![Page 40: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/40.jpg)
Pseudo-code: Convolving • For each axis , , and
– For each 3D point
• Apply a Gaussian mask ( 1 , 4 , 6 , 4 , 1 ) to wi and w e.g., for the x axis:
wi’(x) = wi(x-2) + 4.wi(x-1) + 6.wi(x) + 4.wi(x+1) + wi(x+2)
D O W N S A M P L E
U P S A M P L E
![Page 41: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/41.jpg)
Pseudo-code: On-the-fly Upsampling
• For each pixel (X,Y) in S
– Linearly interpolate the values in the 3D arrays
BF[I](X,Y) =
D O W N S A M P L E
U P S A M P L E
interpolate(wi, X, Y, I(X,Y))
interpolate(w, X, Y, I(X,Y))
![Page 42: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/42.jpg)
Discussion
• Complexity:
• Fast for medium and large kernels – Can be ported on GPU [Chen 07]: always very fast
• Can be extended to color images but slower
• Visually similar to brute-force computation
number of pixels
number of 3D cells
| R | : number of gray levels
![Page 43: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/43.jpg)
input
![Page 44: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/44.jpg)
brute-force implementation
![Page 45: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/45.jpg)
3D kernel visually similar
![Page 46: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/46.jpg)
Running Times
box kernel
![Page 47: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/47.jpg)
How to Choose an Implementation?
Depends a lot on the application. A few guidelines:
• Brute-force: tiny kernels or if accuracy is paramount
• Box Kernel: for short running times on CPU with any kernel size, e.g. editing package
• 3D Kernel: – if GPU available
– if only CPU available: large kernels, color images, cross BF (e.g., good for computational photography)
• Bilteral Pyramid [Fattal 07]: for multi-scale
![Page 48: Course Evaluations - People | MIT CSAILpeople.csail.mit.edu/sparis/bf_course/slides08/06... · 2009. 6. 25. · A Gentle Introduction to Bilateral Filtering and its Applications •](https://reader033.fdocuments.in/reader033/viewer/2022052104/603fc68fcfe1ad038637c23c/html5/thumbnails/48.jpg)
Questions ?