Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2004 Fast and High...

Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2004

Fast and High Quality Overlap Fast and High Quality Overlap Repair for Patch-Based Texture Repair for Patch-Based Texture

SynthesisSynthesis

Andrew NealenAndrew NealenMarc AlexaMarc Alexa

Discrete Geometric Modeling Group (DGM)Discrete Geometric Modeling Group (DGM)Technische Universität DarmstadtTechnische Universität Darmstadt

Our Setting: 2D Texture Synthesis

nxm Input Texture

NxM Output Texture

► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.

Patch-Based Texture Synthesis

Some Existing Methods

► A Very Popular 2D Texture Synthesis Method• Image Quilting [Efros and Freeman 2001]• Graphcut Texures [Kwatra et. al 2003]• Wang Tiles [Cohen et. al 2003]

__ ==22

overlap error

__ ==22

overlap error

__ ==22

overlap error

__ ==22

overlap error

► Introduced at EGSR 2003 [Nealen and Alexa]

• Adaptive Patch Sampling, like Hierarchical Pattern Mapping [Soler et. al 2002]

• Per-Pixel Overlap Re-synthesis

Hybrid Texture Synthesis (HTS)

Hybrid Texture SynthesisMethod

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Goal:From nxm, synthesize

similar, but not identical

Result (N x M)Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Patch-Search in the Input + Copy to Result + Mark Invalid Pixels

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Per-Pixel Re-synthesis Steps (for each Patch)

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Input (n x m)

Intermediate Result

Result (N x M)

Hybrid Texture SynthesisGeneralization: Pro and Con

► Pro: General Method for Overlap Repair• Complementary to other Methods, such as

Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results

• Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc.

► Con: Computationally Expensive• Exhaustive search for each invalid pixel in the

overlap, based on mostly irregular valid neighborhood

• Has O(rN log N) complexity -> Doesn‘t scale well.

Fast Overlap RepairBasic Idea

► Inspiration• Ashikhmin: Synthesizing Natural Textures

[2001] termed Coherence Search• Tong et. al‘s extension: k-Coherence Search

[2002]

► Basic Idea: Intelligently Reduce Search Space• Only search within a set of coherent pixels• Introduce Trade-off between quality and speed

► Applying Coherence Search• For each pixel in the output, store its location in

the input in a source map (same size as the output texture)

Input Texture

Intermediate Result + Source Map

Fast Overlap Repair Coherence Search

► Applying Coherence Search• When searching for a new pixel, only consider

input pixels which are coherent with neighboring output pixels

Input Texture

Source Map Lookup

► Applying Coherence Search• When searching for a new pixel, only consider

input pixels which are coherent with neighboring output pixels

Input Texture

Source Map Lookup

Consequence in this example: Only two possible candidates

► Applying Coherence Search• Simply comparing to the coherent pixels results in

seams similar to Image Quilting (MEBC)

Example:

64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive

Example:

Fast Overlap Repair k-Coherence Search

Input Texture

Source Map Lookup

► Better: Applying k-Coherence Search

► Better: Applying k-Coherence Search• Extend the set by the k-nearest neighbors (knn) of

each coherent pixel (in feature space) and remove duplicates

Source Map Lookup

Input Texture

► Precomputation of knn Data Structure• Performed once for each nxm input texture and

stored for repeated use

• User defines size of box-shaped neighborhood np

• For each of the nxm input pixels─ Construct feature vector by ordered concatenation of the npx np

RGB-triples in the box-shaped neighborhood

• Dimension reduction (75-90%) by applying PCA• Compute indices of k-nearest neighbors to each

► Source Map Maintenance• Each valid pixel in the overlap region is a linear

blend (feathering) of at least two original pixel values, i.e. from at least two different sources

• To avoid the maintenance of multiple source maps, simply store the source of the pixel with greatest contribution in a single source map

Blue: invalid overlap pixels

Results varying k

k = 11

Exhaustive

Results varying k

k = 11

Exhaustive

Results varying k

k = 11

Exhaustive

Results varying k

k = 11

Exhaustive

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results timings

InputExhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

scales 64×64 δmax = 0.02 Δmax = 0.05

rock 128×128 δmax = 0.02 Δmax = 0.05

stonewall 200×200

δmax = 0.02 Δmax = 0.03

Pre: 0 sec.

Synth: 283 sec.

Pre: 0 sec.

Synth: 533 sec.

Pre: 0 sec.

Synth: 985 sec.

Pre: 6+3 sec.

Synth: 226 sec.

Pre: 45+62 sec.

Synth: 226 sec.

Pre: 247+28 s

Synth: 178 sec.

Pre: 6+4 sec.

Synth: 427 sec.

Pre: 45+74 sec.

Synth: 415 sec.

Pre: 247+37 s

Synth: 350 sec.

Results

Exhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

Results

Exhaustive

n = 7x7

k-Coherence

n = 3x3 | k = 5

k-Coherence

n = 5x5 | k = 11

ResultsSynthesis Comparisons

Efros/Leung Wei/Levoy

IQ PBS HTS

Conclusions and Future Work

► Improve Error Metric• Still using the L2 norm due to its simplicity

• Develop a metric which takes feature mismatch into account

• Texton map approach [Zhang et al. 2003]• Feature Map [Wu and Yu 2004] performs even

better, and for near-regular textures, see [Liu et. al 2004] (both to appear at SIGGRAPH 2004)

Questions ?

► Contact Information

Andrew [email protected]

Marc [email protected]

http://www.dgm.informatik.tu-darmstadt.de

Matlab code:http://www.dgm.informatik.tu-darmstadt.de/research/texsynth.html

Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2004 Fast and High...

Documents

Transcript of Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2004 Fast and High...