Fast Image Replacement Using Multi-Resolution Approach

Chih-Wei Fang and Jenn-Jier James Lien Robotics Lab

Department of Computer Science and Information Engineering

National Cheng Kung University

Robotics Lab.

Motivation

Photographs sometimes include unwanted objects. Lose data when the network transmits media (ex.

video)

System: Two modules

Texture Analysis (or Training) Module

Texture Synthesis Module

Texture Analysis Module

Input image for training (Size: W x H)

Is PeriodicYesNo

Non-periodic pattern

side-by-side

Periodic pattern

side-by-sideProcessing for Γ-shaped pattern

Principal Component Analysis (PCA) ： Weight vector (W1…WN) for each Γ-shaped pattern

Vector Quantization (VQ) ： Speeding up to search the best patches.

Mp > Mn

M = Mp or Mn

Processing for periodic pattern

There are Mp patches.

Processing for non-periodic pattern

There are Mn patches.

…………

……………………

…………

…………

1. Sampling

Processing for non-periodic pattern

A pixel-by-pixel shifting method is used to divide the input texture into M patches. M = (W–Wp+1)× (H–Hp+1)

W is the width of the input image (or texture) H is the height of the input image Wp is the width of the patch. Hp is the height of the patch.

Processing for periodic pattern

The pattern is divided into M patches, where M=W×H. Periodic patterns have continuous veins between two equivalent patterns

when positioned side by side. When the patch reaches the boundary of the input image, in order to form a

complete patch, additional pixels are supplied from the opposite border.

Processing for the Γ-shaped pattern

Taking the whole pieces of the patch as training data may lead to an overestimation of the underlying structure of the patch and will certainly increase the length of the training time.

In addition, searching the matching patches by considering their whole contents generally produce unsatisfactory results since the whole contents tend to be quite different from the initial random values and it may cause the rim effect to become distinct.

ω pixels

Each patch has K elements. During training, PCA is used to transform the original K×M matrix to an N×K matrix, where M>>K>N. And there are N eigenvectors. This can reduce most of the time complexity operations, and hence dramatically increases the performance.

Recombining the appearance of the features while maintaining the coherence of the characteristic content

2. Principal Component Analysis (PCA)

KMKK

M

M

PPP

PPP

PPP

21

22221

11211

ΝKΝΝ

K

K

ΕΕΕ

ΕΕΕ

ΕΕΕ

Ψ

21

22221

11211

PCA

After the PCA process and a sort based on the eigenvalues with corresponding eigenvectors: The first several eigenvectors control the global geometrical structure. The middle eigenvectors control the local features. The last few eigenvectors are controlled by some noises.

These noises cause the photograph to appear truer, but have no influence on the geometric structure.

Therefore a good matching structure need only compare the first several eigenvectors.

N

K

ΝKΝΝ

K

K

W

W

W

P

P

P

ΕΕΕ

ΕΕΕ

ΕΕΕ

2

12

1

21

22221

11211

3. Vector Quantization (VQ)

Searching for the best matching pattern from the eigenspace Ψ during the synthesis process is a computationally expensive task.

In order to speed up the synthesis process, the training data is initially projected onto the eigenspace Ψ in order to retrieve the weight vectors.

The data is then separated into C clusters by means of VQ. Using this approach, the computational time can be reduced from O(M) to O( ). M

3. Similarity measure (based on SSD): Using the weight vector to find the closest cluster and then find the best matching patch from this cluster in the input image. Then this matching patch is called to fill the region of the search window.

Texture Synthesis Module

1. Initialization: Each pixel is assigned a random value in the WoxHo-pixel output image area.

Shifting this search window over this image in the scan-line order from the top-left order to the bottom-right order.

………

2. Γ-shaped pattern projectiong: Each WpxHp-pixel search window (or patch) is taken the Γ-shaped pattern and then projected onto the eigenspace Ψ to obtain one N-dimensional weight vector.

……… ………

4. Result: Find the best patches to fill in the WoxHo-pixel output image.

Texture Synthesis Module ： Demo

Image Replacement

Down-sampling with Texture Analysis

Up-sampling with Texture SynthesisI0

I1

I2

I3

I4 β4

β3

β2

β1

β0

C0C3C4 C2 C1

1. Preprocessing Using Multi-resolution Approach

The input image I0 and the inverse matte β0 to do l times down-sampling ↓ , then to get each level of input image Ii and inverse matte βi, i=0~l

Bi denotes the background region and the foreground region Fi is then utilized to search the background Bi

liβIΒ iii ~0, liβIF iii ~0,

2. Training Process Based on Background Region

If the same Γ-shaped pattern of the search window (or patch) is applied, the fragment between two different textures will be mapped by only one texture rather than being mapped by the mixing of two different textures. Hence, the search window is modified.

ω

K pixels…

3. Completion Process

The reconstructed image Ci First, the system reconstructs the level i (i=l) image Ii

The replaced part is reconstructed (or synthesized), called Ii’ Only the reconstructed part of Ii’ is preserved at position Fi

The other parts are replaced by the original background Bi

The system applies the up-sampling ↑ technique from the reconstructed image Ci at level i and proceeds to only replace

the level i-1 image Ii-1 at position Fi-1

iiiii βIβIC

1111 )( iiiii βCβII

Image Replacement ： Demo

Experimental ResultsTable 1: Average training time in Section 3 for input images of various sizes. Units of time: milliseconds (ms).

Size of input image (pixels)

64x64 96x96 128x128

PCA 250 437 875

Projecting to eigenspace 359 703 1656

VQ 203 391 953

Total time (ms) 812 1531 3484

Table 2: Average synthesizing time in Section 4 for output images of various sizes.

Size of output image (pixels)

200x200 300x300 400x400 600x600

Synthesis time (ms) 11 37 59 140

Table 3: The replacement information of each level in Figure 9 and Figure 12 for multi-resolution approach. “~0” means close to 0 ms.

Level Width (pixels) Height (pixels)Number of whole

patchesSynthesis time (ms)

0 392 364 114034 63

1 196 182 24272 16

2 96 91 4024 ~0

3 49 46 110 ~0

4 25 23 0 ~0

Table 4: The time in Figure 9 and Figure 12 for various processes. The total time of the training and synthesis processes include the processes of the multi-resolution approach, PCA, projection (the patches are projected onto eigenspace Ψ to obtain the weight vectors), VQ, and synthesis.

Method Time (ms)

Data type Gray value RGB space

Training process

Multi-resolution 32 31

PCA 891 7719

Projecting to eigenspace 1828 4906

VQ 421 375

Synthesis process

Synthesis 78 108

Total time 3250 13139

Conclusions

We developed a system including two modules the texture analysis and synthesis modules. This system is able to be applied

to the two different purposes: the synthesis of a large image, and the replacement of local removed region.

According to the training non-periodic or periodic pattern we use different sampling methods to obtain different amount of patches in

order to reduce the emergences of the seams of the output synthesized image.

Reduce most of the time complexity operations the analysis module can reduce dimensions of the training data and cluster

these data, so the synthesis module can synthesize a large output image very fast and keep geometrical structures and veins continuous.

Conclusions

Without needing to assign initial random values The same process can also be used to replace the removed regions. Here,

the multi-resolution approach is applied to the image replacement without needing to assign initial random values or approximate values.

Multi-resolution go through the image replacement process The down-sampling step is used for the analysis process as compiling the

training data, and the up-sampling step is used for the reconstructing process as assigning initial values.

Handle the large removed region So this approach enables the system to handle the large removed region and

obtain more realistic image (or textures) quickly.

Experimental ResultsTexture Synthesis (1)

Experimental ResultsTexture Synthesis (2)

Experimental ResultsTexture Synthesis (3)

Experimental ResultsTexture Synthesis (4)

Experimental ResultsTexture Synthesis (5)

Experimental ResultsImage Replacement

(1)

Experimental ResultsImage Replacement

(2)