Edge-aware and bilateral filtering - Computer...
Transcript of Edge-aware and bilateral filtering - Computer...
![Page 1: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/1.jpg)
Edge-aware and bilateral filtering
15-463, 15-663, 15-862Computational Photography
Fall 2019, Lecture 8http://graphics.cs.cmu.edu/courses/15-463
![Page 2: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/2.jpg)
Course announcements
• Homework 2 is out.- Due September 27th. - Requires camera and tripod.- Start early! Substantially larger programming and imaging components than in Homework 1.- Generous bonus component, up to 50% extra credit.- No really: start early!
![Page 3: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Leftover from lecture 7.
• Back to tonemapping.
• Edge-aware filtering and bilateral filtering.
• Non-local means.
• Flash/no-flash photography.
• Joint bilateral filtering.
![Page 4: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/4.jpg)
Slide credits
Many of these slides were inspired or adapted from:
• James Hays (Georgia Tech).• Fredo Durand (MIT).• Gordon Wetzstein (Stanford).• Sylvain Paris (MIT).• Sam Hasinoff (Google).
![Page 5: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/5.jpg)
Back to tonemapping
![Page 6: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/6.jpg)
Dealing with color
If we tonemap all channels the same, colors are washed out
Can you think of a way to deal with this?
![Page 7: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/7.jpg)
Intensity-only tonemapping
tonemapintensity
How would you implement this?
leave color the same
![Page 8: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/8.jpg)
Comparison
Color now OK, but some details are washed out due to loss of contrast
Can you think of a way to deal with this?
![Page 9: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/9.jpg)
Low-frequency intensity-only tonemapping
tonemap low-frequency intensity component
How would you implement this?
leave color the same
leave high-frequency intensity component
the same
![Page 10: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/10.jpg)
Comparison
We got nice color and contrast, but now we’ve run into the halo plague
Can you think of a way to deal with this?
![Page 11: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/11.jpg)
Tonemapping with bilateral filtering
![Page 12: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/12.jpg)
Comparison
We fixed the halos without losing contrast
![Page 13: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/13.jpg)
![Page 14: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/14.jpg)
Edge-aware filtering and bilateral filtering
![Page 15: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/15.jpg)
Motivational example
Let’s say I want to reduce the amount of detail in this picture. What can I do?
original
![Page 16: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/16.jpg)
Motivational example
What is the problem here?
original Gaussian filtering
![Page 17: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/17.jpg)
Motivational example
original Gaussian filtering
How to smooth out the details in the image without losing the important edges?
![Page 18: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/18.jpg)
Motivational example
original Gaussian filtering bilateral filtering
![Page 19: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/19.jpg)
The problem with Gaussian filtering
input
Gaussian kernel
*
*
*
output
Why is the output so blurry?
![Page 20: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/20.jpg)
The problem with Gaussian filtering
input
Gaussian kernel
*
*
*
output
Blur kernel averages across edges
![Page 21: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/21.jpg)
The bilateral filtering solution
input
bilateral filter kernel
*
*
*
output
Do not blur if there is an edge! How does it do that?
![Page 22: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/22.jpg)
Bilateral filtering
![Page 23: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/23.jpg)
Bilateral filtering
Spatial weighting
1) it’s nearbyAssign a pixel a large weight if:
![Page 24: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/24.jpg)
Bilateral filtering
Intensity range weighting
2) it looks like meand
Spatial weighting
1) it’s nearbyAssign a pixel a large weight if:
![Page 25: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/25.jpg)
Bilateral filtering
Normalization factor Intensity range weighting
2) it looks like meand
Spatial weighting
1) it’s nearbyAssign a pixel a large weight if:
![Page 26: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/26.jpg)
Bilateral filtering vs Gaussian filtering
Which is which?
![Page 27: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/27.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filtering
![Page 28: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/28.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filteringSpatial weighting: favor nearby pixels
![Page 29: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/29.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filteringSpatial weighting: favor nearby pixels
Intensity range weighting: favor similar pixels
![Page 30: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/30.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filteringSpatial weighting: favor nearby pixels
Normalization factor
Intensity range weighting: favor similar pixels
![Page 31: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/31.jpg)
Bilateral filtering vs Gaussian filtering
Gaussian filtering
Bilateral filtering
Smooths everything nearby (even edges)Only depends on spatial distance
Smooths ‘close’ pixels in space and intensityDepends on spatial and intensity distance
![Page 32: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/32.jpg)
Gaussian filtering visualization
Output Gaussian Filter Input
![Page 33: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/33.jpg)
Bilateral filtering visualization
Output Bilateral Filter Input
Spatial range Intensity range
![Page 34: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/34.jpg)
Exploring the bilateral filter parameter space
input
ss = 2
ss = 6
ss = 18
sr = 0.1 sr = 0.25sr =
(Gaussian blur)
![Page 35: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/35.jpg)
Does the bilateral filter respect all edges?
input
bilateral filter kernel
*
*
*
output
![Page 36: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/36.jpg)
Does the bilateral filter respect all edges?
input
bilateral filter kernel
*
*
output
Bilateral filter crosses (and blurs) thin edges.
![Page 37: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/37.jpg)
Denoising
noisy input bilateral filtering median filtering
![Page 38: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/38.jpg)
Contrast enhancement
input sharpening based on bilateral filtering
sharpening based on Gaussian filtering
How would you use Gaussian or bilateral filtering for sharpening?
![Page 39: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/39.jpg)
Photo retouching
![Page 40: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/40.jpg)
Photo retouching
original digital pore removal (aka bilateral filtering)
![Page 41: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/41.jpg)
Before
![Page 42: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/42.jpg)
After
![Page 43: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/43.jpg)
Close-up comparison
original digital pore removal (aka bilateral filtering)
![Page 44: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/44.jpg)
Cartoonization
input cartoon rendition
![Page 45: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/45.jpg)
Cartoonization
How would you create this effect?
![Page 46: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/46.jpg)
Cartoonization
edges from bilaterally filtered image bilaterally filtered image
+ =
cartoon rendition
Note: image cartoonization and abstraction are very active research areas.
![Page 47: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/47.jpg)
Is the bilateral filter:
Shift-invariant?
Linear?
![Page 48: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/48.jpg)
Is the bilateral filter:
Shift-invariant?
Linear?
• No.
• No.
Does this have any bad implications?
![Page 49: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/49.jpg)
The bilateral grid
Data structure for fast edge-aware image processing.
![Page 50: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/50.jpg)
Modern edge-aware filtering: local Laplacian pyramids
![Page 51: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/51.jpg)
Modern edge-aware filtering: local Laplacian pyramids
![Page 52: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/52.jpg)
Modern edge-aware filtering: local Laplacian pyramids
input
texture decrease
texture increase
large texture increase
![Page 53: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/53.jpg)
Tonemapping with edge-aware filtering
![Page 54: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/54.jpg)
Tonemapping with edge-aware filtering
bilateral filterlocal Laplacian pyramids
![Page 55: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/55.jpg)
Non-local means
![Page 56: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/56.jpg)
Redundancy in natural images
![Page 57: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/57.jpg)
Non-local means
𝑤(𝑝, 𝑞)
𝑤(𝑝, 𝑟)𝑤(𝑝, 𝑠)
ො𝑥 𝑖 =1
𝐶𝑖
𝑗
𝑦(𝑗) 𝑒−𝑆𝑆𝐷 y 𝑁𝑖 −y 𝑁𝑗
2𝜎2
𝑤 𝑖, 𝑗
No need to stop at neighborhood. Instead search everywhere in the image.
![Page 58: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/58.jpg)
Non-local means vs bilateral filtering
Non-local means filtering
Bilateral filtering
Spatial weighting: favor nearby pixels
Intensity range weighting: favor similar pixels (patches in case of non-local means)
![Page 59: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/59.jpg)
Everything put together
Gaussian filtering
Bilateral filtering
Smooths everything nearby (even edges)Only depends on spatial distance
Smooths ‘close’ pixels in space and intensityDepends on spatial and intensity distance
Non-local means
Smooths similar patches no matter how far awayOnly depends on intensity distance
![Page 60: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/60.jpg)
Denoising example
bilateral filteringGaussian filteringnoisy input non-local means
![Page 61: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/61.jpg)
Very general forms of “structural” filtering
We will see more in later lectures.
![Page 62: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/62.jpg)
Is non-local means:
Shift-invariant?
Linear?
![Page 63: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/63.jpg)
Is non-local means:
Shift-invariant?
Linear?
• No.
• No.
Non-local means is not a convolution, and is generally very very challenging to implement efficiently.
Efficient algorithms for non-local means are an active research area.
![Page 64: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/64.jpg)
Flash/no-flash photography
![Page 65: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/65.jpg)
Red Eye
![Page 66: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/66.jpg)
Unflattering Lighting
![Page 67: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/67.jpg)
Motion Blur
![Page 68: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/68.jpg)
Noise
![Page 69: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/69.jpg)
A lot of Noise
![Page 70: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/70.jpg)
Ruined Ambiance
![Page 71: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/71.jpg)
No-FlashFlash
+ Low Noise+ Sharp- Artificial Light- Jarring Look
- High Noise- Lacks Detail+ Ambient Light+ Natural Look
![Page 72: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/72.jpg)
Image acquisition
Lock Focus& Aperture
1
time
![Page 73: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/73.jpg)
Image acquisition
1/30 sISO 3200
No-Flash ImageLock Focus& Aperture
21
time
![Page 74: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/74.jpg)
Image acquisition
1/30 sISO 3200
1/125 sISO 200
No-Flash ImageLock Focus& Aperture
Flash Image
2 31
time
![Page 75: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/75.jpg)
Denoising Result
![Page 76: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/76.jpg)
• Show a larger result here
No-Flash
![Page 77: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/77.jpg)
Denoising Result
![Page 78: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/78.jpg)
Key idea
Denoise the no-flash image while maintaining the edge structure of the flash image• How would you do this using the image editing techniques we’ve learned about?
![Page 79: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/79.jpg)
Joint bilateral filtering
![Page 80: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/80.jpg)
Denoising with bilateral filtering
noisy input bilateral filtering median filtering
![Page 81: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/81.jpg)
Denoising with bilateral filtering
• However, results still have noise or blur (or both)
ambient
flashBilateral
filter
spatial kernel
intensity kernel
![Page 82: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/82.jpg)
Denoising with joint bilateral filtering
• In the flash image there are many more details
• Use the flash image F to find edges
![Page 83: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/83.jpg)
Denoising with joint bilateral filtering
Bilateral filter
Joint Bilateral filter
The difference
![Page 84: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/84.jpg)
Not all edges in the flash image are real
Can you think of any types of edges that may exist in the flash image but not the ambient one?
![Page 85: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/85.jpg)
Not all edges in the flash image are real
shadows
specularities
• May cause over- or under-blur in joint bilateral filter
• We need to eliminate their effect
![Page 86: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/86.jpg)
Detecting shadows
• Observation: the pixels in the flash shadow should be similar to the ambient image.
• Not identical:
1. Noise.
2. Inter-reflected flash.
• Compute a shadow mask.
• Take pixel p if
• is manually adjusted
• Mask is smoothed and dilated
![Page 87: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/87.jpg)
Detecting specularities
• Take pixels where sensor input is close to maximum (very bright).
• Over fixed threshold
• Create a specularity mask.
• Also smoothed.
• M – the combination of shadow and specularity masks:
Where Mp=1, we use ABase. For other pixels we use ANR.
![Page 88: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/88.jpg)
Detail transfer
• Denoising cannot add details missing in the ambient image
• Exist in flash image because of high SNR
• We use a quotient image:
• Multiply with ANR to add the details
• Masked in the same way
Reduces the effect of
noise in F
Why does this quotient image make sense for detail?
Bilateral filtered
![Page 89: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/89.jpg)
Detail transfer
• Denoising cannot add details missing in the ambient image
• Exist in flash image because of high SNR
• We use a quotient image: Reduces the effect of
noise in F
![Page 90: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/90.jpg)
Full pipeline
![Page 91: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/91.jpg)
Demonstration
ambient-only joint bilateral and detail transfer
![Page 92: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/92.jpg)
Flash
![Page 93: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/93.jpg)
No-Flash
![Page 94: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/94.jpg)
No-Flash
![Page 95: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/95.jpg)
Result
![Page 96: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/96.jpg)
Flash
![Page 97: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/97.jpg)
No-Flash
![Page 98: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/98.jpg)
No-Flash
![Page 99: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/99.jpg)
Result
![Page 100: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/100.jpg)
Flash
![Page 101: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/101.jpg)
No-Flash
![Page 102: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/102.jpg)
Flash
![Page 103: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/103.jpg)
No-Flash
![Page 104: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/104.jpg)
Result
![Page 105: Edge-aware and bilateral filtering - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture8.pdf · texture decrease texture increase large texture increase.](https://reader035.fdocuments.in/reader035/viewer/2022081600/605e7b935099bd30205f398b/html5/thumbnails/105.jpg)
ReferencesBasic reading:• Durand and Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” SIGGRAPH 2002.
The paper on tonemapping using bilateral filtering.• Paris et al., “A Gentle Introduction to the Bilateral Filter and Its Applications,” SIGGRAPH 2007-08, CVPR 2008,
https://people.csail.mit.edu/sparis/bf_course/Short course on the bilateral filter, including discussion of fast implementations.
• Petschnigg et al., “Digital photography with flash and no-flash image pairs,” SIGGRAPH 2004.• Eisemann and Durand, “Flash Photography Enhancement via Intrinsic Relighting,” SIGGRAPH 2004.
The first two papers exploring the idea of photography with flash and no-flash pairs, both using variants of the joint bilateral filter.
Additional reading:• Chen et al., “Real-time edge-aware image processing with the bilateral grid,” SIGGRAPH 2007.• Paris and Durand, “A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach,” IJCV 2009.
Two papers on acceleration techniques for the bilateral filer.• Paris et al., “Local Laplacian Filters: Edge-aware Image Processing with a Laplacian Pyramid,” SIGGRAPH 2011 and CACM 2015.
The paper on local Laplacian pyramids.• Buades et al., “Nonlocal Image and Movie Denoising,” IJCV 2008.
The journal version of the original non-local means paper.• Barnes et al., “PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing,” SIGGRAPH 2009.
A paper on a very efficient implementation of non-local means, including a few amazing applications focusing on creative manipulation of images.