Image RenaissanceUsing Discrete Optimization

Cédric Allène Nikos Paragios

ENPC – CERTIS

ESIEE – A²SIECP - MAS

France

Image Renaissance

Outline

1)Theory: positions and patches

2)Finding positions with good similarities

3)Determining the patch sizes

4)Results and discussions

Image Renaissance

- 1 -Theory:

positions and patches

Image Renaissance1

- T

heory

: p

osit

ion

s a

nd

patc

hes

Main idea: inpainting as a jigsaw

Inpainting: Reconstructing a destroyed part of an image (specified by a

mask) The process should be undetectable

Our method: Considering the reconstruction as a jigsaw which pieces are

patches taken from the “good” part of the image

Image Renaissance

Two steps1) Finding positions/offsets in the “good” image having strong

similarities with the content of the image at the boundary of the area to rebuild

2) For each offset found, finding which pixels should be copied to reconstruct the destroyed area

1 -

Th

eory

: p

osit

ion

s a

nd

patc

hes

Image Renaissance

- 2 -Finding positions

with good similarities

Image Renaissance

Positions and offsets

Such a problem consists in finding, given an origin point at the boundary of the region to inpaint, a number of candidate positions in the image domain that have similar content with the one at the boundary of the area to rebuild

A few remarks: Missing content is not necessarily in the neighborhood of the

destroyed region… Testing each offset in the image for each position at the boundary of

the destroyed area would be too expensive in computation time…

Our solution: using a probabilistic method, the particle filters

2 –

Fin

din

g p

osit

ion

s w

ith

good

sim

ilari

ties

Image Renaissance

Particle filters theory Considering a fixed number of samples, we have to repeat this

steps: Weight the samples with a Probability Density Function (pdf) Generate a new set of samples (of equal number) from the existing

ones with the biggest weights applying small perturbations (importance of perturbation decreases with the iterations to obtain a convergence)

2 –

Fin

din

g p

osit

ion

s w

ith

good

sim

ilari

ties

Image Renaissance

Particle filters in positions research

A particle has an origin point (at the boundary of the area to inpaint, which is invariant), an offset and a size (width and height)

The perturbation is a displacement of the offset and a modification of the size

The weight is evaluated by a Sum of Square Differences (SSD) between the sized neighborhood around the origin point and the offset point balanced with the size

2 –

Fin

din

g p

osit

ion

s w

ith

good

sim

ilari

ties

Image Renaissance

- 3 -Determining

the patch sizes

Image Renaissance

How to find the optimal partition

For each pixel in the area to inpaint we have multiple offsets (and, as a consequence, multiple pixel values possible), so we have to find the best label partition to rebuild the image:

In that aim we use a combinatorial optimization: the α-expansion algorithm (using graph cuts)

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

Image Renaissance

Term 1 to minimize:constraints to data

: region of the image to be reconstructed

: region of the image to be reconstructed and its close neighborhood (which contains data)

: distance (similarity) between the label and the data of the image at position

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

Image Renaissance

Term 2 to minimize:constraints between patches

: region of the image to be reconstructed

: 4-neighborhood of

: distance (similarity) between the label L and the label at position

: Euclidian distance from to the boundary of the area to rebuild (to reconstruct firstly the borders of the region)

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

Image Renaissance

Term 3 to minimize:smoothing term

: region of the image to be reconstructed

: 4-neighborhood of

:

Potts model:penalty for neighbors with different

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

Image Renaissance

Global energy function on partition

to minimize

ω

Minimized through successive α-expansion (using graph cuts)

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

Image Renaissance

Alpha-expansion principle

With a given partition, the algorithm expands the label α over all the other labels minimizing the energy function

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

"green"-expansion

Image Renaissance

Alpha-expansion algorithm

For each offset selected (explaining the interest to reduce their number), we:

Construct a graph representing the energy on the area to inpaint Execute the graph-cut algorithm on the graph Modify the partition

This is repeated until stability (but in practice, one iteration on each offset gives a result near of the final result)

Each iteration will so enhance the result by minimizing the global energy

Y. Boykov, O. Veksler and R. Zabih.Fast Approximate Energy Minimization via Graph Cuts.

PAMI 2001.

3 –

Dete

rmin

ing

th

e p

atc

h s

izes

Image Renaissance

- 4 -Results and discussions

Image Renaissance

Results

4 –

Resu

lts a

nd

dis

cu

ssio

n

Image Renaissance

Results

4 –

Resu

lts a

nd

dis

cu

ssio

n

Image Renaissance

Conclusion

Graph-based combinatorial approach for inpainting

Advantages: Efficient for textured images Quite fast: in general a couple of minutes (but the final result is

nearly reached after the first series of α-expansion)

Drawbacks: Polynomial cost depending on the number of offsets selected (so

particular attention must be paid on this step)

4 –

Resu

lts a

nd

dis

cu

ssio

n

Image Renaissance

Thank you for your attention!

Any question?