Discriminative Approach forTransform Based Image

Restoration

Yacov Hel-Or Doron Shaked Gil Ben-Artzi

SIAM – Imaging Science, July 2008

The Interdisciplinary CenterIsrael

HP LasBar-Ilan Univ.

Israel

- Can we clean Lena?

Motivation – Image denoisingMotivation – Image denoising

nxy

,0~ Nn

• All the above deal with degraded images.• Their reconstruction requires solving an

inverse problem

• Inpainting

• De-blurring

• De-noising

• De-mosaicing

Broader ScopeBroader Scope

Key point: Stat. Prior of Natural Images

xPxyPyxPxxx

maxargmaxargˆ Bayesian estimation:

likelihood prior

Problem: P(x) is complicated to model

form Mumford & Huang, 2000

– Defined over a huge dimensional space. – Sparsely sampled.– Known to be non Gaussian.

A prior p.d.f. of a 2x2 image patch

The Wavelet Transform Marginalizes Image PriorThe Wavelet Transform Marginalizes Image Prior

• Observation1: The Wavelet transform tends to de-correlate pixel dependencies of natural images.

W.T.

xBxB i

iBiB xPxP

• Observation2: The statistics of natural images are homogeneous.

iBibandiBi xPxP

Share the same statistics

Donoho & Johnston 94 Donoho & Johnston 94 Wavelet Shrinkage Denoising: Unitary CaseWavelet Shrinkage Denoising: Unitary Case

• Degradation Model:

• MAP estimation in the transform domain

BBBnxy

BB

xB yxPx

B

maxargˆ

,0~ NnB

• The Wavelet domain diagonalizes the system.

• The estimation of a coefficient depends solely on its own measured value

• This leads to a very useful property:

Modify coefficients via scalar mapping functions

iBkx̂i

Bky

i

BkiB

kkyx ˆ

where Bk stands for the k’th band

yy

Shrinkage Pipe-lineShrinkage Pipe-line

Image domain

Transformdomain

+

xiB

yiB

B3 B2

B1

BT1

BT1

BT2

BT2

BT3

BT3

Image domain

Bkyy k(Bky)

x

BTkk(Bky) x= BT

kk(Bky)

Result

BT1

BT1

BT2

BT2

BT3

BT3B2

B1

B3

Wavelet Shrinkage as aWavelet Shrinkage as aLocally Adaptive Patch Based MethodLocally Adaptive Patch Based Method

KxK

xiB

yiB

DCT

DCT-1 xiB

yiB

xiB

yiB

WDCT

Unitary Transform

• Can be viewed as shrinkage de-noising in a Unitary Transform (Windowed DCT).

xiB

yiBWDCT-1

KxK bands

xiB

yiB

DCT

DCT-1 xiB

yiB

KxK

Alternative Approach: Sliding WindowAlternative Approach: Sliding Window

xiB

yiB

UWDCT

Redundant Transform

• Can be viewed as shrinkage de-noising in a redundant transform (U.D. Windowed DCT).

xiB

yiBUWDCT-1

• Descriptive approach: The shape of the mapping function j depends solely on Pj and the noise variance .

How to Design the Mapping Functions?How to Design the Mapping Functions?

jBandi

iBx yw

Modeling marginal p.d.f.

of band j

noise variance () noise variance ()

jMAPobjective

MAPobjective

• Commonly Pj(yB) are approximated by GGD:

psxexP ~ for p<1

from: Simoncelli 99

from: Simoncelli 99

Hard Thresholding

Soft Thresholding

Linear Wiener Filtering

MAP estimators for GGD model with three different exponents. The noise is additive Gaussian, with variance one third that of the signal.

• Due to its simplicity Wavelet Shrinkage became extremely popular:

– Thousands of applications.

– Thousands of related papers

• What about efficiency?

– Denoising performance of the original Wavelet Shrinkage technique is far from the state-of-the-art results.

• Why?

– Wavelet coefficients are not really independent.

Recent DevelopmentsRecent Developments• Since the original approach suggested by D&J

significant improvements were achieved:

Original Shrinkage

Redundant RepresentationJoint (Local) Coefficient

Modeling

• Overcomplete transform• Scalar MFs• Simple• Not considered state-of-the-art

• Multivariate MFs

• Complicated

• Superior results

1. Mapping functions:– Naively borrowed from the unitary case.

2. Independence assumption:– In the overcomplete case, the wavelet coefficients are

inherently dependent.

3. Minimization domain:– For the unitary case MFs are optimized in the transform

domain. This is incorrect in the overcomplete case (Parseval is not valid anymore).

4. Unsubstantiated– Improvements are shown empirically.

What’s wrong with existing redundant What’s wrong with existing redundant Shrinkage?Shrinkage?

Questions we are going to addressQuestions we are going to address

• How to design optimal MFs for redundant bases.

• What is the role of redundancy.

• What is the role of the domain in which the MFs

are optimized.

• We show that the shrinkage approach is

comparable to state-of-the-art approaches where

MFs are correctly designed.

Optimal Mapping Function:Optimal Mapping Function:

Traditional approach: Descriptive

kBi

iBx

kMAPobjective

MAPobjective

x

Modeling marginal p.d.f. of band k

Optimal Mapping Function:Optimal Mapping Function:

Suggested approach: Discriminative

• Off line: Design MFs with respect to a given set of examples: {xe

i} and {yei}

• On line: Apply the obtained MFs to new noisy signals.

ex eyDenoisingAlgorithm

k

ey B1B1B1B1BkBk

Option 1Option 1: Transform domain –: Transform domain – independent bandsindependent bands

exkB

y

kBx

B1B1B1B1BT

kBT

k

ex B1B1B1B1BkBk

exB1B1B1B1BT

kBT

kkBy

kBx

k i

e

ikkeik yBxB

2

1

+

+

ey B1B1B1B1BkBk

exkB

y

kBx

B1B1B1B1BT

kBT

k

ex B1B1B1B1BkBk

exB1B1B1B1BT

kBT

kkBy

kBx

+

+

k i

e

ikkTk

eik

Tk yBBxBB

2

2

Option 2Option 2: Spatial domain –: Spatial domain – independent bandsindependent bands

ey B1B1B1B1BkBk

exkB

y

kBx

B1B1B1B1BT

kBT

k

ex B1B1B1B1BkBk

exB1B1B1B1BT

kBT

kkBy

kBx

+

+

Option 3Option 3: Spatial domain –: Spatial domain – joint bandsjoint bands

i k

e

ikkTk

ei yBBx

2

3

The Role of Optimization DomainThe Role of Optimization Domain

• Theorem 1: For unitary transforms and for any set of {k}:

• Theorem 2: For over-complete

(tight-frame) and for any set of {k}:

123

123

=

Unitary v.s. OvercompleteUnitary v.s. OvercompleteSpatial v.s. Transform DomainSpatial v.s. Transform Domain

Over-completeUnitary

Spatial domain

Transform domain

=

>

>

=

Is it Justified to optimized in the transform domain?Is it Justified to optimized in the transform domain?

1 3

)(1 Unitary

32

2

• In the transform domains we minimize an upper envelope.

• It is preferable to minimize in the spatial domain.

• Problem: How to optimize non-linear MFs ?

• Solution: Span the non-linear {k} using a linear sum of basis functions.

• Finding {k} boils down to finding the span coefficients (closed form).

Mapping functionsMapping functions

y

k(y)

Optimal Design of Non-Linear MF’sOptimal Design of Non-Linear MF’s

For more details: see Hel-Or & Shaked: IEEE-IP, Feb 2008

yby ii

kik

• Let zR be a real value in a bounded interval [a,b).

• We divide [a,b) into M segments q=[q0,q1,...,qM]

• w.l.o.g. assume z[qj-1,qj)

• Define residue r(z)=(z-qj-1)/(qj-qj-1)

a bz

q0 q1 qMqj-1 qj

r(z)

z=r(z) qj+(1-r(z)) qj-1z=[0,,0,1-r(z),r(z),0,]q = Sq(z)q

The Slice Transform The Slice Transform

• We define a vectorial extension:

• We call this the

Slice Transform (SLT) of z.

qq zSz

zqS

0,r,r-1,0 ii zz

ith row

The SLTThe SLT PropertiesProperties

• Substitution property: Substituting the boundary vector q with a different vector p forms a piecewise linear mapping.

=Sq(z)

zq0

q1

q2

q3

q4

q1 q2 q3 q4

qp

p0

p1

p2

p3

p4

zz’

z

z

z’

Back to the MFs DesignBack to the MFs Design• We approximate the non-linear {k} with piece-wise linear functions:

• Finding {pk} is a standard LS problem with a

closed form solution!

i k

ke

ikqTk

ei yBSBx

k

2

p

kq pk

yBSyB kkk

ResultsResults

Training ImagesTraining Images

Tested ImagesTested Images

Simulation setupSimulation setup

• Transform used: Undecimated DCT• Noise: Additive i.i.d. Gaussian • Number of bins in SLT: 15• Number of bands: 3x3 .. 10x10

MFs for UDCT 8x8 (i,i) bands, i=1..4, =20

OptionOption 1

OptionOption 2

OptionOption 3

Why considering joint band dependencies produces non-monotonic MFs ?

image space

noisy image

Unitary MF

Redundant MF

Comparing psnr results for 8x8 undecimated DCT, sigma=20.

barbara boat fingerprint house lena peppers256 27.5

28

28.5

29

29.5

30

30.5

31

31.5

32

32.5

33

psnr

Method 1

Method 2

Method 3

8x8 UDCT=10

8x8 UDCT=20

8x8 UDCT=10

1 2 5 10 15 20 25

30

35

40

45

50

s.t.d.

PS

NR

barbara

1 2 5 10 15 20 25

30

35

40

45

50

s.t.d.

PS

NR

boat

1 2 5 10 15 20 25

30

35

40

45

50

s.t.d.

PS

NR

fingerprint

1 2 5 10 15 20 25

30

35

40

45

50

s.t.d.

PS

NR

house

1 2 5 10 15 20 25

30

35

40

45

50

s.t.d.

PS

NR

lena

1 2 5 10 15 20 25

30

35

40

45

50

s.t.d.

PS

NR

peppers

Comparison with BLS-GSM

1 2 5 10 15 20 25

28

30

32

34

36

38

40

42

44

46

48

50

s.t.d.

PS

NR

proposed method

GSM method

Comparison with BLS-GSM

Other Degradation ModelsOther Degradation Models

JPEG Artifact RemovalJPEG Artifact Removal

JPEG Artifact RemovalJPEG Artifact Removal

Image SharpeningImage Sharpening

Image SharpeningImage Sharpening

ConclusionsConclusions

• New and simple scheme for over-complete transform based denoising.

• MFs are optimized in a discriminative manner.

• Linear formulation of non-linear minimization.

• Eliminating the need for modeling complex statistical prior in high-dim. space.

• Seamlessly applied to other degradation problems as long as scalar MFs are used for reconstruction.

Recent ResultsRecent Results

• What is the best transform to be used (for a given image or for a given set)?

Thank You