1 Image Completion using Global Optimization Presented by Tingfan Wu.
-
date post
19-Dec-2015 -
Category
Documents
-
view
217 -
download
2
Transcript of 1 Image Completion using Global Optimization Presented by Tingfan Wu.
1
Image Completion Image Completion using Global using Global OptimizationOptimizationPresented by Tingfan WuPresented by Tingfan Wu
2
The Image Inpainting ProblThe Image Inpainting Problemem
3
OutlineOutline
IntroductionIntroduction History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.
Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP
ResultsResults Structural PropagationStructural Propagation
4
Method TypeMethod TypeInpainting
Bertalmio et al. SIGGRAPH 2000
Statistical Example BasedTexture Synth./Patch
GreedyCriminisi et al. 2003
Global Optimization
Priority BP[This paper]
Structural Propagation[Sun et. al, SIGGRAPH 05]
PDEBertalmio et al. 2001
PriorityTexture Syn
th.
Need User Guidance
5
Exampled Based MethodExampled Based Method—Jigsaw Puzzle—Jigsaw Puzzle
Patch
es
Not A
vaila
ble
6
Method TypeMethod TypeInpainting
Bertalmio et al. SIGGRAPH 2000
Statistical Example BasedTexture Synth./Patch
GreedyCriminisi et al. 2003
Global Optimization
Priority BP[This paper]
Structural Propagation[Sun et. al, SIGGRAPH 05]
PDEBertalmio et al. 2001
PriorityTexture Syn
th.
Need User Guidance
7
Greedy v.s Global OptmizatiGreedy v.s Global Optmizationon
Greedy Method
Cannot go back
Global Optimization
Ooops
Refine Globally
8
OutlineOutline
IntroductionIntroduction History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.
Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP
ResultsResults Structural PropagationStructural Propagation
9
Random Fields / Belief Random Fields / Belief NetworkNetwork
Good Project Writer?(High Project grade)
Good Test Taker?(High test score)
Smart Student?(High GPA)
Good Employee(No Observation yet)Edge:
Dependency
RFRF:: Random Variables on GraphRandom Variables on Graph Node : Random Var. (Hidden State) Node : Random Var. (Hidden State) Belief : from Neighbors, and Observation Belief : from Neighbors, and Observation
Random Variable(Observation)
10
Story about MRFStory about MRF
(Bayesian) Belief Network (DAG)(Bayesian) Belief Network (DAG) Markov Random Fields (Undirected, Markov Random Fields (Undirected,
Loopy) Loopy) Special Case:Special Case:
1D - Hidden Markov Model (HMM)1D - Hidden Markov Model (HMM)
Office Helper Wizard Hidden Markov Model (HMM)Hidden Markov Model (HMM)
11
Inpainting as MRF optimizaInpainting as MRF optimizationtion
Node : Grid on target region, overlapped patchesNode : Grid on target region, overlapped patches Edge : A Edge : A nodenode depends only on its depends only on its neighborsneighbors Optimal labeling (hidden state) that minimizing Optimal labeling (hidden state) that minimizing
mismatch energymismatch energy
12
MRF Potential FunctionsMRF Potential Functions
Mismatch (Energy) between ..Mismatch (Energy) between .. VVp p (X(Xp p ) : Source Image vs. New Label) : Source Image vs. New Label VVpqpq(X(Xpp, X, Xqq) : Adjacent Labels) : Adjacent Labels SSum of um of SSquare quare DDistances (SSD) in Overlapping Regioistances (SSD) in Overlapping Regio
nn
13
Global OptimizatoinGlobal Optimizatoin
min
14
OutlineOutline
IntroductionIntroduction History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.
Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP
ResultsResults Structural PropagationStructural Propagation
15
Belief Propagation(1/3)Belief Propagation(1/3)
•Undirected and Loopy•Propagate forward and backward
Good Project Writer?(High Project grade)
Good Test Taker?(High test score)
Smart Student?(High GPA)
Good Employee(No Observation yet)
16
Belief Propagation(2/3)Belief Propagation(2/3) Message ForwardingMessage Forwarding Iterative algorithm until convergeIterative algorithm until converge
Xq
Xp
Candidates at Node Q
Candidates at Node P
Neighbors (P)
O(|Candidate|2)
17
Belief Propagation(3/3)Belief Propagation(3/3)
18
Priority BPPriority BP BP too slow: BP too slow:
Huge #candidates Huge #candidates Time Timemsgmsg = O(|Candidat = O(|Candidates|es|22) )
Huge #Pairs Huge #Pairs Cannot cache pairwise SSDs.Cannot cache pairwise SSDs. ObservationsObservations
Non-Informative messages in unfilled regiNon-Informative messages in unfilled regions ons
Solution to some nodes is obvious (fewer cSolution to some nodes is obvious (fewer candidates.)andidates.)
19
Human WisdomHuman Wisdom
Nobody start from here
Start from non-ambiguous part
AndSearch for
Brown feather+green grass
Candidates
20
Priority BPPriority BP ObservationsObservations
needless messages in unfilled regions needless messages in unfilled regions Solution to some nodes is obvious (fewer Solution to some nodes is obvious (fewer
candidates.)candidates.) Solution: Enhanced BP:Solution: Enhanced BP:
Easy nodes goes first (priority message Easy nodes goes first (priority message scheduling)scheduling)
Keep only highly possible candidates Keep only highly possible candidates (maintain a Active Set)(maintain a Active Set)
21
Priority & PruningPriority & Pruning
?? ?
? ? ?
?? High Priorityprune a lot
Pruning may miss correct label
Low Priority
Candidates sorted by relative belief
Discard Blue Points
22
#Candidates after #Candidates after PruningPruning
Active Set (Darker means smaller)
Histogram of #candidates Similar candidates
23
A closer look at Priority A closer look at Priority BPBP Priority CalculationPriority Calculation
Priority : 1/(#significant candidate)Priority : 1/(#significant candidate) Pruning (on the fly )Pruning (on the fly )
Discard Low Confidence CandidatesDiscard Low Confidence Candidates Similar patches Similar patches One representative (by One representative (by
clustering)clustering) ResultResult
More Confident More Confident More Pruning More Pruning Confident node helps increase neighbor’s Confident node helps increase neighbor’s
confidence.confidence. Warning:Warning:
PBP and Pruning must be used PBP and Pruning must be used togethertogether
25
OutlineOutline IntroductionIntroduction
History of InpaintingHistory of Inpainting Camps – Greedy & Global Opt.Camps – Greedy & Global Opt.
Model and AlgorithmModel and Algorithm Markov Random Fields (MRF) & InpaintingMarkov Random Fields (MRF) & Inpainting Belief Propagation (BP)Belief Propagation (BP) Priority BPPriority BP
ResultsResults ConclusionConclusion Structural PropagationStructural Propagation
26
Darker pixels higher priorityAutomatically start from salient parts.
Results-Inpainting(1/3)Results-Inpainting(1/3)
27
Results-Inpainting(2/3)Results-Inpainting(2/3)
28
Results-Inpainting(3/3)Results-Inpainting(3/3)Up to 2minutes / image (256x170) on P4-2.4GUp to 2minutes / image (256x170) on P4-2.4G
29
More : Texture SynthesisMore : Texture SynthesisInterpolation as well as extrapolationInterpolation as well as extrapolation
30
31
ConclusionConclusion
Priority BPPriority BP {Confident node first} + {candidate pruning}{Confident node first} + {candidate pruning} Generic – applicable to other MRF problems.Generic – applicable to other MRF problems. Speed upSpeed up
MRF for InpaintingMRF for Inpainting Global optimization Global optimization
avoid visually inconsistence by greedyavoid visually inconsistence by greedy Priority BP for InpaintingPriority BP for Inpainting
Automatically start from salient point. Automatically start from salient point.
32
Sometimes …Sometimes … Image contains hard high-level Image contains hard high-level
structurestructure Hard for computersHard for computers Interactive completion guided by Interactive completion guided by
human.human.
33
Potential Func. For Structural PropagaPotential Func. For Structural Propagationtion
User input a guideline by human User input a guideline by human region.region.
Potential Function respect distance Potential Function respect distance between linesbetween lines
Jian Sun et al, SIGGRAPH 2005Jian Sun et al, SIGGRAPH 2005
34
VideoVideo
Link:Link:MicrosoftMicrosoft Research Research