An Introduction to Texture Synthesis
description
Transcript of An Introduction to Texture Synthesis
An Introduction to Texture SynthesisAn Introduction to Texture Synthesis
2008/05/29
Desirable PropertiesDesirable Properties
Result looks like the Result looks like the inputinput
EfficientEfficient
GeneralGeneral
Easy to useEasy to use
ExtensibleExtensible
The ChallengeThe Challenge
Need to model the whole Need to model the whole spectrum: from spectrum: from repeated to stochastic repeated to stochastic texturetexture
repeated
stochastic
Both?
[Shannon,’48] proposed a way to [Shannon,’48] proposed a way to generate English-looking text using generate English-looking text using N-grams:N-grams:•Assume a generalized Markov model
•Use a large text to compute prob. distributions of each letter given N-1 previous letters
•Starting from a seed repeatedly sample this Markov chain to generate new letters
•Also works for whole words
Assuming Markov property, compute P(Assuming Markov property, compute P(pp|N(|N(pp))))• Building explicit probability tables infeasible
• Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p
To synthesize To synthesize pp, just pick one match at random, just pick one match at random
pp
non-parametricsampling
Input image Synthesizing a pixel
Basic AlgorithmBasic Algorithm
Exhaustively search neighborhoodsExhaustively search neighborhoods
NeighborhoodNeighborhood
Neighborhood size determines the quality & costNeighborhood size determines the quality & cost
33 55 77
99 1111 4141
423 s 528 s 739 s
1020 s 1445 s 24350 s
Efros & Leung ’99Efros & Leung ’99
The algorithmThe algorithm•Very simple
•Surprisingly good results
•Synthesis is easier than analysis!
•…but very slow
Optimizations and ImprovementsOptimizations and Improvements•Multi-resolution Pyramid
• Jump Map
•Chaos Mosaic
Multi-resolution PyramidMulti-resolution Pyramid
High resolution Low resolution
Multi-resolution AlgorithmMulti-resolution Algorithm
Multi-resolution PyramidMulti-resolution Pyramid
Better image quality & faster computation Better image quality & faster computation
1 level55
3 levels55
1 level1111
What is a Jump Map?What is a Jump Map?
Same size as inputSame size as input
What is a Jump Map?What is a Jump Map?
Same size as inputSame size as input
Set of jumps per Set of jumps per pixelpixel
What is a Jump Map?What is a Jump Map?
Same size as inputSame size as input
Set of jumps per Set of jumps per pixelpixel
Jumps are weighted Jumps are weighted according to according to similaritysimilarity• Need not sum to 1
0.4
0.3
0.2
Output
Synthesis with Jump MapsSynthesis with Jump Maps
Input
Chaos Mosaic [Xu, Guo & Shum, ‘00]Chaos Mosaic [Xu, Guo & Shum, ‘00]
Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges
input
idea result
Used in Lapped Textures [Praun et.al,’00]
Chaos Mosaic [Xu, Guo & Shum, ‘00]Chaos Mosaic [Xu, Guo & Shum, ‘00]
Of course, doesn’t work for structured textures
input
result
Image QuiltingImage Quilting
Idea:Idea:• let’s combine random block placement of
Chaos Mosaic with spatial constraints of Efros & Leung
pp
Efros & Leung ’99 extendedEfros & Leung ’99 extended
Observation:Observation: neighbor pixels are highly correlated neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = block unit of synthesis = block• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
• Not the same as multi-scale!
Synthesizing a block
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
min. error boundary
Minimal error boundaryMinimal error boundary
overlapping blocks vertical boundary
__ ==22
overlap error
),,min( 1,1,11,1 jijijiijij EEEeE
AlgorithmAlgorithm
•Pick size of block and size of overlap
•Synthesize blocks in raster order
•Search input texture for block that satisfies overlap constraints (above and left)
•Easy to optimize using NN search
•Paste new block into resulting texture
•use dynamic programming to compute minimal error boundary cut
Failures
(Chernobyl
Harvest)
Image Quilting for Texture Transfer
Image Quilting for Texture Transfer
++ ==
Texture TransferTexture Transfer
Take the texture from Take the texture from one object and “paint” it one object and “paint” it onto another objectonto another object
• This requires separating texture and shape
• That’s HARD, but we can cheat
• Assume we can capture shape by boundary and rough shading
Then, just add another constraint when Then, just add another constraint when sampling: similarity to underlying image sampling: similarity to underlying image at that spotat that spot
++ ==
++ ==
parmesan
rice
++ ==
Sourcetexture
Target image
Sourcecorrespondenc
eimage
Targetcorrespondence image
++ ==
input image
Portilla & Simoncelli
Wei & Levoy Image Quilting
Xu, Guo & Shum
Portilla & Simoncelli
Wei & Levoy Image Quilting
Xu, Guo & Shum
input image
Portilla & Simoncelli
Wei & Levoy Image Quilting
input image
Homage to Shannon!
Xu, Guo & Shum