IMAGE AND VIDEO ABSTRACTION BY MULTI-SCALE ANISOTROPIC KUWAHARA FILTERING

NPAR 2011

JAN ERIC KYPRIANIDIS

HASSO-PLATTNER-INSTITUT, GERMANY

ABSTRACT

Multi-scale Anisotropic Kuwahara Filter - a coarse-to-fine edge preserving smoothing filter

1. Strong image abstraction 2. Avoid artifacts in large low-contrast region

original image anisotropic Kuwahara filter proposed method

OUTLINE

1. Introduction

2. Related work

3. Method

1. Pyramid structure : coarse fine

2. Anisotropic Kuwahara filter

3. Merging function : (upper level, this level)

4. Results

5. Conclusions

Image abstraction &Edge preserving smoothing filters

Image abstraction

Edge preserving smoothing filter • Bilateral• Kuwahara

Segmentation

(mean-shift)NPR

Temporal coherence in

video

INTRODUCTION

Image abstraction is useful for NPR or temporal coherence in video

PROBLEM OF SEGMENTATION

Regions segmented by mean-shift [DeCarlo & Santella 02; Wen et 1l. 06] have rough boundaries and require elaborate post-processing

Mean-shift results in rough-boundaryoriginal image

PROBLEM OF EDGE PRESERVING SMOOTHIHG FILTER

Bilateral filter or Kuwahara filter cause overblurring

• Remove detail in low-contrast region

Bilateral filter results in overblurring in low-contrast regions.

original image

PROBLEM OF ANISOTROPIC KUWAHARA FILTER

Preserve features, directions, and is robust against high-contrast noise

• However .. • level of abstraction depending on filter radius • Artifacts in large low-contrast regions

Bilateraloriginal image Anisotropic Kuwahara

(direction)

ADVANCE OFMULTI-SCALE ANISOTROPIC KUWAHARA FILTERING

• Avoid artifacts and smooth results • By adding thresholding to the weighting term

• Strong abstraction and avoidance of artifacts in large low-contrast regions• Coarse-to-fine from multi-scale image pyramids

• Real-time processing on GPU

1.down sample to create image pyramid first

2. from coarse-to-fine, apply anisotropic

Kawahara filter and merge the previousDown

sampleUp

sample

RELATED WORK

1. Image pyramids

2. Bilateral filter

3. Kuwahara filters

4. Anisotropic Kuwahara filter

5. Mean curvature flow + shocking filtering

6. Diffusion and shock filtering

7. Image abstraction on gradient domain

edge preserving smoothing filter

IMAGE PYRAMIDS

Gaussian filter + down-sample

Gaussian filter + down-sample

Gaussian filter + down-sample

down-sampling without smoothing aliasing

BILATERAL FILTER[Smith 97, Tomasi 98]

edge preserving + smoothing filter

Weights

• Gaussian on space distance• Gaussian on range distance• sum to 1

space range

Sylvain Paris and Frédo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV’ 06.

Input Result

smoothing preserve edges

KUWAHARA FILTER

[Kuwahara et al. 76]

Select the ave. value of sub-region whose var. is min.

where q in the sub-region Ri of p with min. variance

𝑘 (𝑝)=min .𝑣𝑎𝑟 ( 𝑓 ) 𝑖𝑛𝑅 𝑖

𝑓 (𝑞)

Anisotropic Kuwahara [09]

[07]

[76]

A tiled & aliased image filtered by Kuwahara filter

origin

Kuwahara filter

Due to 1. Rectangular sub-regions2. Unstable if noise exits3. Subregions have the same variance.

GENERALIZED KUWAHARA FILTER

[Papari et al. 07]

New val. is sum of mean of each sub-region weighted by the inverse stdv.

Anisotropic Kuwahara [09]

[07]

[76]

si : variance of sub-region imi : mean of sub-region i

Fail to capture directional features & clustering artifacts

Anisotropic KuwaharaGeneralized Kuwahara

non directional

[Kyprianidis et al. PG09]

New val. is sum of mean of each sub-region weighted by the inverse stdv.

Anisotropic Kuwahara [09]

[07]

[76]

si : variance of sub-region imi : mean of sub-region i

GENERALIZED KUWAHARA FILTER

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth image tangents

• Set the ellipse kernel

2nd eigen vector v2 of local gradients

smoothed by Gaussian filter

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth image tangents

• Set the ellipse kernel

2nd eigen vector v2 of local gradients

smoothed by Gaussian filter

(fx, fy) = mean(local gradients (gx,gy))

1. Image tangent = 2nd eigen vector of Jij structure tensor

Jij

PCA of image gradients

2nd eigen vector

Image tangent = 2nd eigen vector of local gradients

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth image tangents

• Set the ellipse kernel

2nd eigen vector v2 of local gradients

smoothed by Gaussian filter

𝐽 𝑖𝑗𝒗=𝜆 𝒗

1. Image tangent = 2nd eigen vector of Jij

2. Smooth the image tangents by Gaussian filter

(fx, fy) = mean(local gradients (gx,gy)Jij

structure tensor

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth the tangent of image

• Set the ellipse kernel

scale & rotate as ellipse kernel

rotate K0 to generate kernel of each subregion

K0

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth the tangent of image

• Set the ellipse kernel

K0

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth the tangent of image

• Set the ellipse kernel

rotate K0 to generate kernel of each region

Ki

ANISOTROPIC KUWAHARA FILTER

[Kyprianidis et al. PG09]

• Smooth the tangent of image

• Set the ellipse kernelscale & rotate as ellipse kernel

1. Scale 2. Rotation

α=1

Jij

𝐽 𝑖𝑗𝒗=𝜆 𝒗𝛀

MEAN CURVATURE FLOW (MCF)+ SHOCKING FILTERING

[Kang and Lee. PG08]

• Mean curvature flow simplifies shape of boundaries

• Users must protect important features• Shock filter sharpens the discontinuities and flattens each

homogenous regions

IMAGE ABSTRACTION ON GRADIENT DOMAIN

[Orzan et al. NPAR’07]

origin

thickness = importance

reconstruction

IMAGE ABSTRACTION ON GRADIENT DOMAIN

1. Importance

2. Gradient reconstruction

• Solve Poisson eq. with constrain of gradients

lifetime ∝ |gradient| ∝ thickness ∝ importance

Canny edge (small scale large scale)source

METHOD – OVERVIEW

Downsample

Upsample

fK+1

fK

New image fk = Merge (fk, upsample (Filter(fk+1, Jk+1)))

tensor Jk := covariance matrix of gradients of fk

New tensor Jk = Merge (gk, upsample (smooth(Jk+1)))

1. Down sample to create image pyramid first

2. From coarse-to-fine, apply anisotropic Kawahara filter and merge the previous

GRADIENT

Replace (Gaussian derivatives and Sobel filter) with Jähne Filter

Jähne Filter

STRUCTURE TENSOR

2nd eigen vector asimage tangent

𝐽 𝒗=𝜆𝒗

SMOOTH THE STRUCTURE OF TENSOR

x

=

vCal. v to min. err(local gradients, v)

=

𝐸 (𝑥 )=∫𝐺𝜌 (𝑥− 𝑦 )𝑔 (𝑦 )𝑔 ( 𝑦 )𝑑𝑦−𝑣 (𝑥 )𝑇 𝐽 𝜌 (𝑥 )𝑣 (𝑥)

=x

v v(x) is the 1st eigen vector of local gradients

The image tangent vector of x is vertical to v(x)

The 2nd eigen vector is the image tangent vector

Konstantinos G. Derpanis. “The Harris Corner Detector”. 2004

Taylor series

(a)

=[x+]

If a = v2 min. c(x,y)

ANISOTROPY MEASURE

A = 1 …. λ1>> λ2

A = 0 …. λ1= λ2

anisotropic ellipse kernel

isotropic circle kernel

ANISOTROPIC KUWAHARA FILTER

New val. is sum of mean of each sub-region weighted by the inverse stdv.

Add threshold to avoid artifacts and smooth results …. Why ?

Kernel of each sub-region

Lower bound of stdv. to avoid1. Zero stdv. in flat region divided by zero2. Small differences in the stdv. in large low contrast region artifacts

𝜏𝜔=0.02 ,𝑞=8

MERGE BETWEEN SCALES

Downsample

Upsample

fK+1

fK

New image fk = Merge (fk, upsample (Filter(fk+1, Jk+1)))

tensor Jk := covariance matrix of gradients of fk

New tensor Jk = Merge (gk, upsample (smooth(Jk+1)))

1. Down sample to create image pyramid first

2. From coarse-to-fine, apply anisotropic Kawahara filter and merge the previous

MULTI-SCALE ESTIMATION – MERGE TENSOR

=

Always prefer the more anisotropic tensor !ex2. Ak+1 = 0, Ak = 1 αk = 1 ~Jk = Jk

ex1. Ak+1 = 1, Ak = 0 αk = 0 ~Jk = Jk+1

fK+1

fK

MULTI-SCALE ESTIMATION – MERGE TENSOR

Always prefer the more anisotropic tensor !

fK+1

fK

MULTI-SCALE FILTERING

fK+1

fK

𝑝𝑠=0.5 ,𝑝𝑑=1 . 25 ,𝜏𝑣=0.1 ,𝜏𝜔=0.02

𝑓 𝑘+1=𝑢𝑝𝑠𝑎𝑚𝑝𝑙𝑒 ( h𝐾𝑢𝑤𝑎 𝑎𝑟𝑎 ( 𝑓 𝑘+1 ))

fK+1

fK

RESULT By C++, GLSL

NVDIA GTX580

512x512 … 42 ms

HD 720p (1280x720) .. 150 ms

(b) the fur above the nose is less abstracted than at the neck.

very consistent level of abstraction

strong abstraction where slightly less abstraction above the nose

stronger contrast,

LIMITATION

Fail to produce good-looking results

parts above the plant are blended with the ground

LIMITATION

Images with high frequency texture is hard to abstract

CONCLUSION

• Avoid artifacts and smooth results • By adding thresholding to the weighting term

• Strong abstraction and avoidance of artifacts in large low-contrast regions• Coarse-to-fine from multi-scale image pyramids

• Real-time processing on GPU

ANY QUESTION ?

If the area of a sub-region in a Kuwahara filter is very very small, is it similar to a bilateral filter ?

Kernel of 1D bilateral filter

Kernel of 2D Kuwahara filter Sub-regionc can not too small

END