04/12/2023 Domain Transform

DOMAIN TRANSFORM FOR EDGE-AWARE IMAGE AND VIDEO PROCESSINGEduardo S. L. Gastal and Manuel M. Oliveira

Instituto de Inform´atica – UFRGS

SIGGRAPH 2011

04/12/2023 Domain Transform

Edge-aware filter• e.g. Bilateral filter

• Weighting F of spacial distance and (color) range distance

Spacial distance is fixed.Fast

Range distance and kernel value has to be calculated Bottleneck !

How about L1 distance ?

How about replacing 2D filter with 1D vertical and horizontal filter ?

How to speed up?No way in most cases !

pq0

||PQ||1 = |xp-xq| + |yp-yq| , ||PQ||2= √((xp-xq)2+(yp-yq)2)

04/12/2023 Domain Transform

Abstract• 1D filtering on transformed multiple-dimension domain

e.g. a case of 2D filter on a color image (XY, RGB)

This paper provides high quality edge-preserving smoothing after

• (x, RGB) ct(x). Apply 1D horizontal filter on ct(x)

• (y, RGB) ct(y). Apply 1D vertical filter on ct(y)

• Real-time at arbitrary scale

• Many applications of this real-time edge preserving filter • Depth-of-field effects, stylization, recoloring, colorization, detail

enhancement, and tone mapping.

04/12/2023 Domain Transform

Contribution• Real-time high-quality edge aware filtering of image/video

based on a dimensionality reduction • Smoothing on ct(X,RGB) and ct(Y, RGB) ≡ edge preserving on 2D

color image

• Perform anisotropic edge-preserving on curves of 2D image manifold using 1D linear filters• Isotropic kernel on ct(x) ≡ anisotropic kernel on x

• 2D edge-preserving filter as a sequence of 1D linear filter

• The first real-time edge-preserving smoothing• Any scale kernel• Process time independent of the filter parameters• Can control kernel shape• Handle color image correctly

1D

04/12/2023 Domain Transform

RELATED WORK

04/12/2023 Domain Transform

It’s have been well known that …

2D Smoothing ≡ 1D Edge-preserving

However, replace 1D Bilateral with 2D Gaussian …. High Cost !

04/12/2023 Domain Transform

2D Smoothing ≡ 1D Edge-preservingBilateral grid

[Chen et al. SIG07]

Gaussian KD-tree

[Adams et al. SIG09]

higher dimensional functions

Gaussian convolution

division

slicing

w i w

limited for small kernels, coarse problem not real-time

down-sample

04/12/2023 Domain Transform

Anisotropic Diffusion Filter (AD)

• [Perona and Malik 1990] suggested ppt

• Diffusion is smoothing : pixels diffuse ∆I • Anisotropic kernel preserves edge : strong edges reduce kernel size

Unknown iterations

c(p, t) is large when p is not a part of an edge c(p, t) is small when p is a part of an edge

04/12/2023 Domain Transform

Edge-avoid Wavelets (EAW)• [Raanan Fatta SIG2009]

Constrain the size of the smoothing kernel to 2^n

translation-invariant trans. Edge-avoiding trans.

Edge-avoiding scaling func.

inversely related to the pixel difference.

04/12/2023 Domain Transform

DOMAIN TRANSFORM

04/12/2023 Domain Transform

Domain transform ct(u)

Smoothing on transformed domain Ωw ≡ Edge-preserving on Ω

Ωw is ct(u)

04/12/2023 Domain Transform

I’

|I’|

04/12/2023 Domain Transform

|I’|

ct(u) is accumulation of each L1 distance of (x, I(x))

04/12/2023 Domain Transform

Domain Transform ct(u)

RG B

ct(u) : (u,I(u)) 1 : 1

ct(u) : (u,I(u)) 1 : 1

ct(u) is accumulation of each L1 distance of (u, I(u))

04/12/2023 Domain Transform

Domain Transform ct(u)

RG B

ct(u) : (u,I(u)) 1 : 1

ct(u) : (u,I(u)) 1 : 1

ct(u) is accumulation of each L1 distance of (u, I(u))

04/12/2023 Domain Transform

Ωw = ct(u)pq

04/12/2023 Domain Transform

Ωw = ct(u)

edge edgeedgeedge

p

q

04/12/2023

Smoothing Ωw ≡ Edge-preserving Ω

Ωw is ct(u)

WHY ?

Smoothing Ωw ≡ Edge-preserving Ω

04/12/2023

Smoothing Ω can not preserve edge …

Smoothing Ωw ≡ Edge-preserving Ω

04/12/2023

Smoothing Ωw ≡ Smoothing neighbors in L1

Sample Ωw uniformly ≡ sample more densely in high |I ’| (edge) Sample Ωw uniformly ≡ along curve in Ω ≡ adaptive kernel Ω |(spatial, color)|-1

L1

sample Ωw uniformly

sampling along the curve

Smoothing on Ω

Smoothing Ωw ≡ Edge-preserving Ω

04/12/2023 Filter Attributes

Kernel size of Bilateral filter – σs, σr • σs

2 = variance of spatial kernel. σs radius of spatial kernel

• σr2

= variance of range kernel. σr radius of range kernel

large σs

σr

x

I

pkernel in Ωw

What do σs and σr mean in Ωw ?

04/12/2023 Filter Attributes

Kernel size of Bilateral filter – σs, σr • σs

2 = variance of spatial kernel. σs radius of spatial kernel

• σr2

= variance of range kernel. σr radius of range kernel

small σs

σr

x

I

p

What do σs and σr mean in Ωw ?

kernel in Ωw

04/12/2023 Domain Transform

Kernel size of Bilateral filter – σs, σr • σs

2 = variance of spatial kernel. σs radius of spatial kernel

• σr2

= variance of range kernel. σr radius of range kernel

small σs

σr

x

I

pscaled kernel in Ωw

|pq| L1 = |px-qx| + |pr-qr|

q non-scaled kernel in Ωw

04/12/2023 Filter Attributes

Kernel size of Bilateral filter – σs, σr • σs

2 = variance of spatial kernel. σs radius of spatial kernel

• σr2

= variance of range kernel. σr radius of range kernel

large σs

σr

x

I

p

|pq| L1 = |px-qx| + |pr-qr|

scaled kernel in Ωw

non-scaled kernel in Ωw

04/12/2023 Filter Attributes

Scaling Factors – σs, σr • σs

2 = variance of spatial kernel. σs radius of spatial kernel

• σr2 = variance of range kernel. σr radius of range kernel

• σH2 = variance of transformed domain kernel. σH radius of transformed

domain kernel

04/12/2023 Filter Attributes

Relationship to σs, σr and I

no longer edge-preserving

unbounded smoothing as input

as input

σs

σr

x

I

04/12/2023

FILTERING IN THE TRANSFORMED DOMAINNormalized Convolution (NC)

Interpolated Convolution (IC)

Recursive Filtering (RF)

04/12/2023

Filtering in Ωw

• Sample uniformly in Ωw is unnecessary

• It’s easy to determine who is neighbor in Ωw

I

I Ω

Ωw

p

ct(p)

ab

c

ab

c

ct(p)-ct(a) < r a is b’s neighborct(c)-ct(p) > r c is not b’s neighbor

04/12/2023

Normalized Convolution (NC)

𝑡 (�̂� )=𝑐𝑡 (𝑝 )

𝑡 (�̂� )

moving-average [Dougherty 94]

[Knutsson and Westin 93]

r

04/12/2023

Normalized Convolution (NC)

𝑡 (�̂� )=𝑐𝑡 (𝑝 )

moving-average [Dougherty 94]

[Knutsson and Westin 93]

why cost time can be independent on filter attributes ( or ) by moving average?

Bilateral : large costs much time NC : Tlarge σs is smilar to Tsmall σs

Σ I

Ω

√3𝜎𝑠0 √3𝜎𝑠1

04/12/2023

Interpolated Convolution (IC)[Piroddi and Petrou 04]

continuous, not discrete !

04/12/2023

NC vs. IC

𝑡 (�̂� ) 𝑡 (�̂� )

Normalized convolution (NC) Interpolated convolution (IC) NC, IC

04/12/2023

Recursive Filtering (RF)

a ∈ [0, 1] is a feedback coefficient

An infinite impulse response (IIR) with exponential decay

d a↗ ⇨ d 0 .. stop propagation and preserve edge

[Smith 07]

04/12/2023

FILTERING 2D SIGNALSReplacing 2D filter with multiple iterations of 1D vertical and horizontal filter

04/12/2023 Filtering 2D signals

Filtering 2D Signals

• Is it possible to apply domain transform on (x,y) ? • ct(p) is the accumulated L1 distance along (p, I(p))

• Only exists in surface with zero Gaussian curvature [O’Neill 06]•

• Implementation• Several iterations

• Horizontal pass along each image row• Vertical pass along each image column

04/12/2023 Filtering 2D signals

How to Determine #Iterations ? • Stripes are only present along the last filtered dimension

• The length of the stripes size of the filter in the last pass

1st iteration 2nd iteration

stripes

Halve σH at each iteration. Stop while σH is small enough

04/12/2023 Filtering 2D signals σH = σs = 40, σr = 0.77

σH = σs = 50, σr = 0.5

2 itr.

04/12/2023

COMPARISON TO OTHER APPROACHES

04/12/2023

Impulse ResponseCase : the left impulse without strong edge; the right impulse with strong edge

NC:Normalized convolution; IC:Interpolated convolution; RF:Recursive convolution; BF:Bilateral ; AD:Anisotropic Diffusion; WLS:Weighted Least Squares

The NC and IC filters have Gaussian-like response, similar to AD and BF.

Strong edges : IC ~ AD. NC has a higher response near strong edges : pixels near

edges have less neighbors in the same population, and will weight their contribution strongly.

Impulse : RF ~ WLS

#iteration = 3

Ωw

Ω

strong edge

04/12/2023

Smooth Quality

04/12/2023

Performance

Filtering on CPU

• 2.8 GHz Quad Core PC • 8GB memory• NC, RF by C++• IC in MATLAB

• 1M RGB * #3 = 0.16~0.06 sec.• 10 M RGB * #3 = 1.6 ~ 0.6 sec.

• Linearly with image size • Independent on σs or σr

• Faster than EAW, PLBF, CTBF

Filtering on GPU

• NVIDIA GeForce GTX280• CUDA

• 1M RGB * #3 = 0.007 sec.• Domain tx : 0.7 mx• Each iteration : 2 ms

04/12/2023

Comparison

• Bilateral Grid [Chen et al. 07]• Faster than domain tx. • Luminance only• Due to down-sampling process, not possible for small

spatial and range kernel

• NVIDIA GeForce GTX280

This paper PLBF WLS

1M RGB * #3 = 0.007 sec

0.5 M RGB = 0.1 sec

1 M gray = 1sec

04/12/2023

REA-TIME APPLICATIONS OF EDGE-PRESERVING FILTER (EPF) Detail Manipulation Tone Mapping Stylization Joint Filtering Colorization Recoloring

04/12/2023

Detail Manipulation

• EPF decomposes image into level-of-detail : J0, J1, .. Jk

• I = J0 , Di = Ji – Ji+1

input I using D0 by IC (σs=20, σr=.08) [Farbman et al. 08]

EAW by [Fattal 09]

04/12/2023

Tone Mapping• Edge-aware tone mapping avoids haloing and artifacts due to compression• The compressed luminance channel Lc in HDR [Farbman et al. 08]

This paper (RF) WLS

12 msec on log-luminance by RF ! J1 : σs= 20, σr=0.33 J2 : σs= 50, σr=0.67 J3 : σs=100, σr=1.34

I = J0 , Di = Ji – Ji+1 μ is the mean of Blocal min B = 0local max B = 1

04/12/2023

Stylization 1/2

• Abstract low-contrast region and preserving high-contrast features

edge + this edge-aware filtered image

NC filtering

04/12/2023

Stylization 2/2

• Assign each output pixel a scaled version of the value of the normalization factor Kp in NC

0.11 Kp

Kp : #points in the moving windowsOn the edge : #point dark↘

On the smooth area : #point light↗

04/12/2023

Joint Filtering• Alpha can be combined with other map to create some

localized or selective stylization

alpha map

04/12/2023

Joint Filtering• Smooth image A based on the

edges information of image B simulate depth-of-field (DoF)

depth-of-field effect

𝑐𝑡 (𝑢)=∫0

𝑢

1+𝜎 𝑠

𝜎𝑟

|𝐵 ′ (𝑥)|𝑑𝑥

B is an alpha matte

04/12/2023 Domain Transform

Joint FilteringCanny edge identifies edges

04/12/2023

Colorization• Propagate user color S by blurring them using the edge

information

using this RF (σs=100, σr=.03) [Levin et al. 04]User color scribbles

𝐶𝑜𝑙𝑜𝑟𝑝=~𝑆(𝑝 )/~𝑁 (𝑝)

S : user scribbles N : normalization function : blurred S : blurred N

04/12/2023

Recoloring

• Soft segmentation – region Ri defined by color Ci

𝐶𝑜𝑙𝑜𝑟𝑝=~𝑁𝑅𝑖 (𝑝 )/∑

𝑗

~𝑁𝑅𝑗 (𝑝)

04/12/2023

Conclusions and Future Work• High-quality edge-preserving filtering of image and videos

in real time• First real-time edge-preserving filter on color image at arbitrary

scale • 1D smoothing filter on transformed domain ≡ Edge-preserving

filter • High quality 2D edge-preserving filtering by iterating 1D-filtering• independent of the filter parameters(σs, σr) (due to moving average)

• Future work• Edge-preserving applications

• Limitation• Not rotationally invariant

04/12/2023

Q & A