Post on 20-Dec-2015
The University of
Ontario
University of Bonn July 2008
Optimization of surface functionals using graph cut algorithms
Yuri Boykov
presenting joint work with
V.Kolmogorov, O.Veksler, D.Cremers, V.Lempitsky, O.Juan, A.Delong
The University of
Ontario
Optimization of surface functionals using graph cut algorithms
optimization for image segmentation (overview) • energy models in vision (weak membrane, MRF, Mumford-Shah, etc.)• energies for contours and surfaces
surfaces and binary labelling of grids• geometric surface functionals and submodular binary energies
– optimization via graph cut algorithms – metrication errors
• global vs. local optimization• computational issues
applications, extensions
The University of
Ontario
Example of image labeling: piece-wise smooth image restoration
I
p
LHow to compute L from I ?
observed noisy image I image labeling L(restored intensities)
The University of
Ontario
Piece-wise smooth labeling(image restoration)
discrete MRF approach• weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87)
Nq)(p,
||
||),(
22 )()()()( TqLpL
TLLNqp
qpp
pp
qp
LLILLE
pL qL
ZZLp 2:
line process, Geman&Geman’84
Nqp
qpp
pp LLVILLE),(
2 ),()()( ,V
T Tdiscontinuity preserving potentials
Blake&Zisserman’83,87
The University of
Ontario
Piece-wise smooth labeling(image restoration)
continuous approach• Mumford-Shah model (Mumford&Shah 85,89)
||||)()(\
22 KdpLdpILLEK
pp
)( pL
RRpL 2:)(
The University of
Ontario
Piece-wise constant labeling(image restoration)
I
p
L
observed noisy image I image labeling L(restored intensities)
The University of
Ontario
Piece-wise constant labeling(image restoration)
Potts model • BVZ ‘98• Greig et al.’89 for 2 labels
Mumford-Shah• Chan-Vese ’02 for 2 labels
1 23 4
}:{ ipi LLp
Con
tin
uou
s:
D
iscr
ete
:
Nqp
qpp
pi
i
LLILLEi ),(
2 )()()(
||)()( 2 KdpILLEi
pi
i
The University of
Ontario
Piece-wise constant labeling(frontal-parallel stereo)
a pair of “stereo” images(left and right eyes views)
depth map(label = depth layer)
The University of
Ontario
Piece-wise constant labeling (frontal-parallel stereo)
Potts model • BVZ ‘98
Mumford-Shah ||)()( KdpLDLE pp
Nqp
qpp
pp LLLDLE),(
)()()(
Con
tin
uou
s:
D
iscr
ete
:
Data penalty function . In stereo it describes photoconsistency of pixel
p when it is assigned to each specific depth layer (label)
)(pD
The University of
Ontario
Binary labeling(binary image restoration)
2)()( pppp ILLD
}1,0{pL
original binary image I optimal binary labeling L
Nqp
qpp
pp LLLDLE),(
)()()( Greig Porteous Seheult ’89
Globally optimal solution is possible using combinatorial graph cut algorithms• pseudo-boolean optimization Hammer’65, Picard&Ratlif’75
The University of
Ontario
Binary labeling(object extraction)
1pL
0pL
}1,0{pL
The University of
Ontario
Binary labeling(object extraction)
10
}1,0{pL
C
)int(}1|{1 CLp p
)(}0|{0 CextLp p
Nqp
qppqp
pp LLwLDLE),(
)()()( Boykov&Jolly’01
The University of
Ontario
Binary labeling(object extraction)
n-links
s
t a cut
}1,0{pL
Where would penalties come from? pD
)1(pD
0)0( pD0)1( qD
)0(qD
Example 1: hard constraints
p
q
22exp
pq
pq
Iw
pqI
The University of
OntarioGraph cuts 2
Any paths would work, butshorter paths give faster algorithms
(in theory and practice)
The University of
OntarioGraph cuts 3
Finds the strongest boundary (least number of holes)
The University of
Ontario
Binary labeling(object extraction)
Globally optimal cut can be computed in polynomial time
}1,0{pL
The University of
Ontariopush-relabel vs. augmenting paths
alternatively: move flow excesses locally - opportunistic strategy assuming they all can reach the other terminals
The University of
Ontariopush-relabel vs. augmenting paths
motivation: - path sharing - parallelization opportunities (e.g. GPU cuts, region push-relabel, Delong&Boykov08)
The University of
Ontario
Binary labeling(object extraction)
s
t
}1,0{pL
22exp
pq
pq
Iw
pqI
Example 2: known color distributions for object and background
I)|Pr( sI p
)|Pr( tI p
pI
)|Pr(ln)( tItD pp
)|Pr(ln)( sIsD pp
The University of
Ontario
Binary labeling(object extraction)
I)|Pr( sI p
)|Pr( tI p
pI
}1,0{pL
22exp
pq
pq
Iw
pqI
s
t a cut
Example 2: known color distributions for object and background
The University of
Ontario
Binary labeling(object extraction)
Blake et al.’04, Rother et al.’04
}1,0{pL
Example 3: iteratively re-estimate color models e.g. using mixture of Gaussians
The University of
Ontario
Binary labeling(object extraction)
Potts model • BJ’01, BK’03-05
Nqp
qppqp
pp LLwLDLE),(
)()()(
)(),(1
qpNqp
pqp
p LLwf
)0()1( ppp DDf
}1,0{pLC
on
tin
uou
s:
Dis
crete
:
gC ||||
?
CC
dsgdpfCE)int(
)(
Geodesic Active Contours• Caselles et al. 93-95, Tenenbaum et al. 95
The University of
Ontario
Geometric properties of contour C and energy of binary labeling L(p)
Properties of
the interior
Properties of
the boundary
p
pp
C
p LDdpf )()int(
pp
pp
fD
fD
)0(
)1(
Nqp
qppq
C
g LLwdsgC),(
)(||||
?
The University of
OntarioIntegral geometry
C 2
0
a set of all lines L
CL
a subset of lines L intersecting contour C
ddnC L21||||Euclidean length of C :
the number of times line L intersects C
Cauchy-Crofton formula
probability that a “randomly drown” line intersects C
The University of
Ontario
Graph cuts and integral geometryBoykov&Kolmogorov’03
C
k
kkknC 21||||
Euclidean length
2kk
kw
GC ||||graph cut cost
for edge weights:the number of edges of family k intersecting C
Edges of any regular neighborhood system
generate families of lines
{ , , , }
Graph nodes are imbeddedin R2 in a grid-like fashion
The University of
OntarioMetrication errors
“standard” 4-neighborhoods
(Manhattan metric)
larger-neighborhoods8-neighborhoods
Euclidean metric
Riemannianmetric
The University of
OntarioRemoving metrication artifacts
4-neighborhood 8-neighborhood
The University of
Ontario
What geometric functionals can be globally optimized via graph cuts?
)(
)(,)()(CC C
x dpxfdsdsgCE vN
)int(
)(,)()(CC
x
C
dpxfdsdsgCE vN
Geometric length any convex,
symmetric metric (e.g. Riemannian)
Flux any vector field v
Regional bias any scalar function f
(“edge alignment”)
Tight characterization for geometric functionals of contour C that can be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05)
disclaimer: for pairwise interactions only
Nqp
qpp
pp LLELDLE),(
),()()(submodularity of energy implies
The University of
OntarioGlobally optimal surface in 3D
Volumetric segmentation:
metric g is based on image gradient
)int(
1)(CC
dvdsgCE
The University of
OntarioGlobally optimal surface in 3D
Vogiatzis, Torr, Cippola’05
Multiview reconstruction:
metric g is based on photoconsistency
)int(
1)(CC
dvdsgCE
The University of
OntarioGlobally optimal surface in 3D
CC
dsdsCE 1,)( vN
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
The University of
OntarioGlobally optimal surface in 3D
CC
dsdpdivCE 1)()int(
v
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
From 10 views
No initialization is needed
The University of
OntarioGlobal vs. local optimization
regional potentials )(vdivf
CC
dsdpdivCE 1)()int(
v
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
initial solution local minima global minima
The University of
OntarioComputing min s/t cuts Augmenting paths [Ford & Fulkerson, 1962]
Push-relabel [Goldberg-Tarjan, 1986]
Pseudoflows [Hochbaum, 1997]
Poor control of locality Computing global minima requires whole graph to fit
into memory (RAM)
problems of standard algorithms
The University of
OntarioComputing min s/t cuts
Better control of locality?
The University of
OntarioComputing min s/t cuts
region size =1(local relabeling)
region size=n(global relabeling)
region size =16 region size=49
region push-relabel [Delong&Boykov’08]
The University of
OntarioComputing min s/t cuts
)(rTTheoretical worst case running time
r=1 region size r (log scale) r=n
1 CPU
),4min(
)(
rn
rT
accounting for parallelization opportunities
4 CPU
region push-relabel [Delong&Boykov’08]
The University of
OntarioComputing min s/t cuts
region push-relabel [Delong&Boykov’08]
16 32 64 128 256 5120
20
40
60
80
100
120
140
160
180
200
abdomen (512x512x256) 1 CPU
2 CPU
4 CPU
8 CPU
region diameter
The University of
OntarioComputing min s/t cuts
region push-relabel [Delong&Boykov’08]
2GB 1GB 512MB 256MB1
10
100
1000
10000BK (256x256x160)
PR (512x512x256)
RPR (512x512x256)
system physical memory
tim
e in
min
ute
s(l
og
sc
ale
)
Scales well to large graphs that do not fit into available memory
The University of
Ontariousing parametric max-flow methods
optimization of ratio functionals in N-D using Dinkelbach’s method (Kolmogorov, Boykov, Rother 2007)
• in 2D can also use DP (Cox et al’96, Jermyn&Ishikawa’01)
C
C
x
dsg
ds
CE)(
,
)(
vN
)(
)(
)(
)(
C
C
dpxf
dsg
CE
The University of
Ontario
Related to isoperimetric problem => bias to circles
using parametric max-flow methods
area
lengtharea
lengthweightedcontrast -
length
areaweightedcolor -lengthweightedcontrast
areaweightedcolor
-
-
length
flux
The University of
Ontario
other labelsa
Extending to multiple labels
a-expansion [Boykov,Veksler,Zabih’98]
Basic idea:break multi-way cut computation into a sequence of binary s-t cuts
},..,1,0{ mLp
The University of
OntarioMulti-way graph cuts
Multi-object Extraction
The University of
OntarioMulti-way graph cuts
Stereo/Motion with slanted surfaces (Birchfield &Tomasi 1999)
Labels = parameterized surfaces
EM based: E step = compute surface boundaries M step = re-estimate surface parameters
The University of
OntarioMulti-way graph cuts
stereo vision
original pair of “stereo” images
depth map
ground truthBVZ 1998KZ 2002
The University of
OntarioMulti-way graph cuts
Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003)
similar to “image-quilting” (Efros & Freeman, 2001)
AB
C D
EF G
H I J
A B
G
DC
F
H I J
E
The University of
Ontario
normalized correlation,start for annealing, 24.7% err
simulated annealing, 19 hours, 20.3% err
a-expansions (BVZ 89,01)90 seconds, 5.8% err
1 10 100 1000 10000 100000
0100002000030000400005000060000700008000090000
100000
Annealing Our method
Time in seconds
Sm
oo
thn
es
s E
ne
rgy
a-expansions vs. simulated annealing