Extensions of submodularity and their application in ...
Transcript of Extensions of submodularity and their application in ...
![Page 1: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/1.jpg)
Extensions of submodularity and
their application in computer vision
Vladimir Kolmogorov
IST Austria
Oxford, 20 January 2014
![Page 2: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/2.jpg)
Linear Programming relaxation
• Popular approach: Basic LP relaxation (BLP)
- for pairwise energies: Schlesinger’s LP
- for higher-order energies: enforce consistency between variables and for all and
- many algorithms for (approximately) solving it (MSD, TRW-S, MPLP, ...)
![Page 3: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/3.jpg)
Tightness of BLP
Theorem [Cooper’08], [Werner’10].
If each term is a submodular function
then BLP relaxation is tight.
• Other such classes? Complete classification?
• Use VCSP framework
![Page 4: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/4.jpg)
Valued Constraint Satisfaction Problem (VCSP)
• Language G: a set of cost functions
• VCSP(G): class of functions that can be expressed as a sum of functions from G with overlapping sets of vars
- Goal: minimize this sum
• Complexity of G?
• Does BLP solves G?
• Feder-Vardi conjecture (for CSPs):
Every CSP language is either tractable or NP-hard
- CSP: contains functions
![Page 5: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/5.jpg)
Classifications for finite-valued CSPs
- Other languages are NP-hard [Thapper,Živný STOC’13]
Theorem [Thapper,Živný FOCS’12], [K ICALP’13]
BLP solves G iff it admits a binary symmetric fractional polymorphism
![Page 6: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/6.jpg)
Submodular functions
0
1
2
3
4
![Page 7: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/7.jpg)
New classes of functions
![Page 8: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/8.jpg)
Useful classes
• Submodular functions
- Pairwise functions: can be solved via maxflow (“graph cuts”)
- Lots of applications in computer vision
• Bisubmodular functions
- Obtaining partial optimal solutions
- Characterize extensions of “QPBO” to arbitrary pseudo-Boolean functions [K’10,12]
• k-submodular functions
- Partial optimality for functions of k-valued variables
- This talk: efficient algorithm for Potts energy [Gridchyn, K ICCV’13]
![Page 9: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/9.jpg)
Partial optimality
• Input: function
• Partial labeling is optimal if it can be extended
to a full optimal labeling
![Page 10: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/10.jpg)
Partial optimality
• Input: function
• Partial labeling is optimal if it can be extended
to a full optimal labeling
• Can be viewed as a labeling
![Page 11: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/11.jpg)
k-submodular relaxations
• Input: function
![Page 12: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/12.jpg)
k-submodular relaxations
• Input: function
• Construct extension
which is k-submodular
• Minimize
Theorem:
Minimum of partially optimal
![Page 13: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/13.jpg)
k-submodularity
• Function is k-submodular if
![Page 14: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/14.jpg)
k-submodular relaxations
• Case k = 2 ([K’10,12])
Bisubmodular relaxation
Characterizes extensions of QPBO
• Case k > 2
[Gridchyn,K ICCV’13] :
- efficient method for Potts energies
[Wahlström SODA’14] :
- used for FPT algorithms
![Page 15: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/15.jpg)
k-submodular relaxations for Potts energy
• k-submodular relaxation:
![Page 16: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/16.jpg)
k-submodular relaxations for Potts energy
• k-submodular relaxation:
: tree metric
![Page 17: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/17.jpg)
k-submodular relaxations for Potts energy
• k-submodular relaxation:
(·) : k-submodular relaxation of (·)
3 4 0 10
1.5
![Page 18: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/18.jpg)
k-submodular relaxations for Potts energy
• Minimizing : O(log k) maxflows
• Alternative approach: [Kovtun ’03,’04]
- Stronger than k-submodular relaxations (labels more)
- Can be solved by the same approach!
complexity: k => O(log k) maxflows
- Part of “Reduce, Reuse, Recycle’’ [Alahari et al.’08,’10]
- Our tests for stereo: 50-93% labeled
with 9x9 windows
- Speeds up alpha-expansion for unlabeled part
![Page 19: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/19.jpg)
Tree Metrics
• [Felzenszwalb et al.’10]: O(log k) maxflows for
![Page 20: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/20.jpg)
Tree Metrics
• [Felzenszwalb et al.’10]: O(log k) maxflows for
• [This work]: extension to more general unary terms
- new proof of correctness
![Page 21: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/21.jpg)
Special case: Total Variation
• Convex unary terms
• Reduction to parametric maxflow [Hochbaum’01],
[Chambolle’05], [Darbon,Sigelle’05]
![Page 22: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/22.jpg)
New condition: T-convexity
• Convexity for any pair of adjacent edges:
![Page 23: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/23.jpg)
Algorithm: divide-and-conquer
• Pick edge ( )
• Compute
![Page 24: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/24.jpg)
Algorithm: divide-and-conquer
• Pick edge ( )
• Compute
• Claim: has a minimizer as shown below
• Solve two subproblems recursively
![Page 25: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/25.jpg)
Achieving balanced splits
• For star graphs, all splits are unbalanced
• Solution [Felzenszwalb et al.’10]: insert a new short edge
- modify unary terms (·) accordingly
![Page 26: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/26.jpg)
Algorithm illustration
• k = 7 labels:
1,2,3,4,5,6,7
1
2 3
4
5
6
7
5
6
7
1
2 3
4
![Page 27: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/27.jpg)
Algorithm illustration
• k = 7 labels:
1,2,3,4
5,6,7
1
2 3
4
5
6
7
5
6
7
1
2 3
4
![Page 28: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/28.jpg)
Algorithm illustration
• k = 7 labels:
5,6,7 1,2
3,4
![Page 29: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/29.jpg)
Algorithm illustration
• k = 7 labels:
1,2
3,4
5,6
7
![Page 30: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/30.jpg)
Algorithm illustration
• k = 7 labels:
7
1
2
3 4
5
6
“Kovtun labeling” unlabeled part,
run alpha-expansion
• maxflows
![Page 31: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/31.jpg)
Stereo results
![Page 32: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/32.jpg)
Proof of correctness (sketch)
![Page 33: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/33.jpg)
Proof of correctness (sketch)
• For labeling and edge ( ) define
![Page 34: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/34.jpg)
Proof of correctness (sketch)
• For labeling and edge ( ) define
![Page 35: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/35.jpg)
Proof of correctness (sketch)
• Coarea formula:
![Page 36: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/36.jpg)
Proof of correctness (sketch)
• Coarea formula:
![Page 37: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/37.jpg)
Proof of correctness (sketch)
• Coarea formula:
![Page 38: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/38.jpg)
Proof of correctness (sketch)
• Coarea formula:
![Page 39: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/39.jpg)
Proof of correctness (sketch)
• Coarea formula:
![Page 40: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/40.jpg)
Proof of correctness (sketch)
• Coarea formula:
• Equivalent problem: minimize
with subject to consistency constraints
• Equivalent to independent minimizations of
- consistency holds automatically due to convexity of
![Page 41: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/41.jpg)
Extension to trees
• Coarea formula:
![Page 42: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/42.jpg)
Summary
Part I:
• New tractable class of functions
complete characterization for finite-valued CSPs
Part II:
• k-submodular relaxations for partial optimality
• For Potts model:
cast Kovtun’s approach as k-submodular function minimization
O(log k) algorithm
generalized alg. of [Felzenswalb et al’10] for tree metrics
• Future work: k-submodular relaxations for other functions?
![Page 43: Extensions of submodularity and their application in ...](https://reader031.fdocuments.in/reader031/viewer/2022012410/616a5a6a11a7b741a351926c/html5/thumbnails/43.jpg)
postdoc & PhD student
positions are available