study Image and video abstraction by multi scale anisotropic kuwahara
-
Upload
chiamin-hsu -
Category
Technology
-
view
900 -
download
0
Transcript of study Image and video abstraction by multi scale anisotropic kuwahara
IMAGE AND VIDEO ABSTRACTION BY MULTI-SCALE ANISOTROPIC KUWAHARA FILTERING
NPAR 2011
JAN ERIC KYPRIANIDIS
HASSO-PLATTNER-INSTITUT, GERMANY
ABSTRACT
Multi-scale Anisotropic Kuwahara Filter - a coarse-to-fine edge preserving smoothing filter
1. Strong image abstraction 2. Avoid artifacts in large low-contrast region
original image anisotropic Kuwahara filter proposed method
OUTLINE
1. Introduction
2. Related work
3. Method
1. Pyramid structure : coarse fine
2. Anisotropic Kuwahara filter
3. Merging function : (upper level, this level)
4. Results
5. Conclusions
Image abstraction &Edge preserving smoothing filters
Image abstraction
Edge preserving smoothing filter • Bilateral• Kuwahara
Segmentation
(mean-shift)NPR
Temporal coherence in
video
INTRODUCTION
Image abstraction is useful for NPR or temporal coherence in video
PROBLEM OF SEGMENTATION
Regions segmented by mean-shift [DeCarlo & Santella 02; Wen et 1l. 06] have rough boundaries and require elaborate post-processing
Mean-shift results in rough-boundaryoriginal image
PROBLEM OF EDGE PRESERVING SMOOTHIHG FILTER
Bilateral filter or Kuwahara filter cause overblurring
• Remove detail in low-contrast region
Bilateral filter results in overblurring in low-contrast regions.
original image
PROBLEM OF ANISOTROPIC KUWAHARA FILTER
Preserve features, directions, and is robust against high-contrast noise
• However .. • level of abstraction depending on filter radius • Artifacts in large low-contrast regions
Bilateraloriginal image Anisotropic Kuwahara
(direction)
ADVANCE OFMULTI-SCALE ANISOTROPIC KUWAHARA FILTERING
• Avoid artifacts and smooth results • By adding thresholding to the weighting term
• Strong abstraction and avoidance of artifacts in large low-contrast regions• Coarse-to-fine from multi-scale image pyramids
• Real-time processing on GPU
1.down sample to create image pyramid first
2. from coarse-to-fine, apply anisotropic
Kawahara filter and merge the previousDown
sampleUp
sample
RELATED WORK
1. Image pyramids
2. Bilateral filter
3. Kuwahara filters
4. Anisotropic Kuwahara filter
5. Mean curvature flow + shocking filtering
6. Diffusion and shock filtering
7. Image abstraction on gradient domain
edge preserving smoothing filter
IMAGE PYRAMIDS
Gaussian filter + down-sample
Gaussian filter + down-sample
Gaussian filter + down-sample
down-sampling without smoothing aliasing
BILATERAL FILTER[Smith 97, Tomasi 98]
edge preserving + smoothing filter
Weights
• Gaussian on space distance• Gaussian on range distance• sum to 1
space range
Sylvain Paris and Frédo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV’ 06.
Input Result
smoothing preserve edges
KUWAHARA FILTER
[Kuwahara et al. 76]
Select the ave. value of sub-region whose var. is min.
where q in the sub-region Ri of p with min. variance
𝑘 (𝑝)=min .𝑣𝑎𝑟 ( 𝑓 ) 𝑖𝑛𝑅 𝑖
𝑓 (𝑞)
Anisotropic Kuwahara [09]
[07]
[76]
A tiled & aliased image filtered by Kuwahara filter
origin
Kuwahara filter
Due to 1. Rectangular sub-regions2. Unstable if noise exits3. Subregions have the same variance.
GENERALIZED KUWAHARA FILTER
[Papari et al. 07]
New val. is sum of mean of each sub-region weighted by the inverse stdv.
Anisotropic Kuwahara [09]
[07]
[76]
si : variance of sub-region imi : mean of sub-region i
Fail to capture directional features & clustering artifacts
Anisotropic KuwaharaGeneralized Kuwahara
non directional
[Kyprianidis et al. PG09]
New val. is sum of mean of each sub-region weighted by the inverse stdv.
Anisotropic Kuwahara [09]
[07]
[76]
si : variance of sub-region imi : mean of sub-region i
GENERALIZED KUWAHARA FILTER
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth image tangents
• Set the ellipse kernel
2nd eigen vector v2 of local gradients
smoothed by Gaussian filter
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth image tangents
• Set the ellipse kernel
2nd eigen vector v2 of local gradients
smoothed by Gaussian filter
(fx, fy) = mean(local gradients (gx,gy))
1. Image tangent = 2nd eigen vector of Jij structure tensor
Jij
PCA of image gradients
2nd eigen vector
Image tangent = 2nd eigen vector of local gradients
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth image tangents
• Set the ellipse kernel
2nd eigen vector v2 of local gradients
smoothed by Gaussian filter
𝐽 𝑖𝑗𝒗=𝜆 𝒗
1. Image tangent = 2nd eigen vector of Jij
2. Smooth the image tangents by Gaussian filter
(fx, fy) = mean(local gradients (gx,gy)Jij
structure tensor
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth the tangent of image
• Set the ellipse kernel
scale & rotate as ellipse kernel
rotate K0 to generate kernel of each subregion
K0
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth the tangent of image
• Set the ellipse kernel
K0
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth the tangent of image
• Set the ellipse kernel
rotate K0 to generate kernel of each region
Ki
ANISOTROPIC KUWAHARA FILTER
[Kyprianidis et al. PG09]
• Smooth the tangent of image
• Set the ellipse kernelscale & rotate as ellipse kernel
1. Scale 2. Rotation
α=1
Jij
𝐽 𝑖𝑗𝒗=𝜆 𝒗𝛀
MEAN CURVATURE FLOW (MCF)+ SHOCKING FILTERING
[Kang and Lee. PG08]
• Mean curvature flow simplifies shape of boundaries
• Users must protect important features• Shock filter sharpens the discontinuities and flattens each
homogenous regions
IMAGE ABSTRACTION ON GRADIENT DOMAIN
[Orzan et al. NPAR’07]
origin
thickness = importance
reconstruction
IMAGE ABSTRACTION ON GRADIENT DOMAIN
1. Importance
2. Gradient reconstruction
• Solve Poisson eq. with constrain of gradients
lifetime ∝ |gradient| ∝ thickness ∝ importance
Canny edge (small scale large scale)source
METHOD – OVERVIEW
Downsample
Upsample
fK+1
fK
New image fk = Merge (fk, upsample (Filter(fk+1, Jk+1)))
tensor Jk := covariance matrix of gradients of fk
New tensor Jk = Merge (gk, upsample (smooth(Jk+1)))
1. Down sample to create image pyramid first
2. From coarse-to-fine, apply anisotropic Kawahara filter and merge the previous
GRADIENT
Replace (Gaussian derivatives and Sobel filter) with Jähne Filter
Jähne Filter
STRUCTURE TENSOR
2nd eigen vector asimage tangent
𝐽 𝒗=𝜆𝒗
SMOOTH THE STRUCTURE OF TENSOR
x
=
vCal. v to min. err(local gradients, v)
=
𝐸 (𝑥 )=∫𝐺𝜌 (𝑥− 𝑦 )𝑔 (𝑦 )𝑔 ( 𝑦 )𝑑𝑦−𝑣 (𝑥 )𝑇 𝐽 𝜌 (𝑥 )𝑣 (𝑥)
=x
v v(x) is the 1st eigen vector of local gradients
The image tangent vector of x is vertical to v(x)
The 2nd eigen vector is the image tangent vector
Konstantinos G. Derpanis. “The Harris Corner Detector”. 2004
Taylor series
(a)
=[x+]
If a = v2 min. c(x,y)
ANISOTROPY MEASURE
A = 1 …. λ1>> λ2
A = 0 …. λ1= λ2
anisotropic ellipse kernel
isotropic circle kernel
ANISOTROPIC KUWAHARA FILTER
New val. is sum of mean of each sub-region weighted by the inverse stdv.
Add threshold to avoid artifacts and smooth results …. Why ?
Kernel of each sub-region
Lower bound of stdv. to avoid1. Zero stdv. in flat region divided by zero2. Small differences in the stdv. in large low contrast region artifacts
𝜏𝜔=0.02 ,𝑞=8
MERGE BETWEEN SCALES
Downsample
Upsample
fK+1
fK
New image fk = Merge (fk, upsample (Filter(fk+1, Jk+1)))
tensor Jk := covariance matrix of gradients of fk
New tensor Jk = Merge (gk, upsample (smooth(Jk+1)))
1. Down sample to create image pyramid first
2. From coarse-to-fine, apply anisotropic Kawahara filter and merge the previous
MULTI-SCALE ESTIMATION – MERGE TENSOR
=
Always prefer the more anisotropic tensor !ex2. Ak+1 = 0, Ak = 1 αk = 1 ~Jk = Jk
ex1. Ak+1 = 1, Ak = 0 αk = 0 ~Jk = Jk+1
fK+1
fK
MULTI-SCALE ESTIMATION – MERGE TENSOR
Always prefer the more anisotropic tensor !
fK+1
fK
MULTI-SCALE FILTERING
fK+1
fK
𝑝𝑠=0.5 ,𝑝𝑑=1 . 25 ,𝜏𝑣=0.1 ,𝜏𝜔=0.02
𝑓 𝑘+1=𝑢𝑝𝑠𝑎𝑚𝑝𝑙𝑒 ( h𝐾𝑢𝑤𝑎 𝑎𝑟𝑎 ( 𝑓 𝑘+1 ))
fK+1
fK
RESULT By C++, GLSL
NVDIA GTX580
512x512 … 42 ms
HD 720p (1280x720) .. 150 ms
(b) the fur above the nose is less abstracted than at the neck.
very consistent level of abstraction
strong abstraction where slightly less abstraction above the nose
stronger contrast,
LIMITATION
Fail to produce good-looking results
parts above the plant are blended with the ground
LIMITATION
Images with high frequency texture is hard to abstract
CONCLUSION
• Avoid artifacts and smooth results • By adding thresholding to the weighting term
• Strong abstraction and avoidance of artifacts in large low-contrast regions• Coarse-to-fine from multi-scale image pyramids
• Real-time processing on GPU
ANY QUESTION ?
If the area of a sub-region in a Kuwahara filter is very very small, is it similar to a bilateral filter ?
Kernel of 1D bilateral filter
Kernel of 2D Kuwahara filter Sub-regionc can not too small
END