Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M....

Solving Markov Random Fields using

Dynamic Graph Cuts &Second Order Cone Programming

Relaxations

M. Pawan Kumar, Pushmeet Kohli

Philip Torr

Talk Outline

• Dynamic Graph Cuts– Fast reestimation of cut– Useful for video– Object specific segmentation

• Estimation of non submodular MRF’s– Relaxations beyond linear!!

Example: Video Segmentation

Model Based Segmentation

Image Segmentation Pose Estimate

[Images courtesy: M. Black, L. Sigal]

Min-Marginals

ImageMAP Solution Belief - Foreground

Lowsmoothness

Highsmoothness

Moderatesmoothness

Colour Scale

Uses of Min marginals

• Estimate of true marginals (uncertainty)

• Parameter Learning.

• Get best n solutions easily.

Dynamic Graph Cuts

cheaperoperation

computationally

expensive operation

Simplerproblem

differencesbetweenA and B

A and Bsimilar

First segmentation problem MAP solution

Our Algorithm

second segmentation problem

Maximum flow

residual graph (Gr)

differencebetween

Ga and Gbupdated residual

• The Max-flow Problem- Edge capacity and flow balance constraints

Computing the st-mincut from Max-flow algorithms

• Notation- Residual capacity (edge capacity – current flow)- Augmenting path

• Simple Augmenting Path based Algorithms- Repeatedly find augmenting paths and push flow.- Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem]

Sink (1)

Source (0)

9 + α

4 + α

Adding a constant to both thet-edges of a node does not change the edges constituting the st-mincut.

Key Observation

Sink (1)

Source (0)

E (a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

E*(a1,a2 ) = E(a1,a2) + α(a2+ā2)

= E(a1,a2) + α [a2+ā2 =1]

Reparametrization

9 + α

All reparametrizations of the graph are sums of these two types.

Other type of reparametrization

Sink (1)

Source (0)

5 + α

2 + α

1 - α

Reparametrization, second type

Both maintain the solution and add a constant α to the energy.

Reparametrization

• Nice result (easy to prove)

• All other reparametrizations can be viewed in terms of these two basic operations.

• Proof in Hammer, and also in one of Vlad’s recent papers.

original graph

0/2 0/4

flow/residual capacity

Graph Re-parameterization

original graph

0/2 0/4

flow/residual capacity

Graph Re-parameterization

t residual graph

xi xj0/12

2/0 4/0st-mincut

ComputeMaxflow

Edges cut

Update t-edge Capacities

t residual graph

xi xj0/12

2/0 4/0

t residual graph

xi xj0/12

2/0 4/0

capacitychanges from

7 to 4

updated residual graph

xi xj0/12

2/0 4/0

7 to 4

edge capacityconstraint violated!(flow > capacity)

= 5 – 4 = 1

excess flow (e) = flow – new capacity

add e to both t-edges connected to node i

xi xj0/12

2/0 4/0

7 to 4

= 5 – 4 = 1

xi xj0/12

7 to 4

add e to both t-edges connected to node i

= 5 – 4 = 1

Update n-edge Capacities

residual graph

xi xj0/12

2/0 4/0

• Capacity changes from 5 to 2

Updated residual graph

xi xj0/12

2/0 4/0

• Capacity changes from 5 to 2- edge capacity constraint violated!

xi xj0/12

2/0 4/0

• Reduce flow to satisfy constraint

xi xj0/11

2/0 4/0

excess

deficiency

• Reduce flow to satisfy constraint- causes flow imbalance!

xi xj0/11

2/0 4/0

deficiency

excess

• Push excess flow to/from the terminals

• Create capacity by adding α = excess to both t-edges.

• Push excess flow to the terminals

xi xj0/11

3/0 4/1

• Push excess flow to the terminals

xi xj0/11

3/0 4/1

Complexity analysis of MRF Update Operations

MRF Energy Operation

Graph Operation Complexity

modifying a unary term

modifying a pair-wise term

adding a latent variable

delete a latent variable

Updating a t-edge capacity

Updating a n-edge capacity

adding a node

set the capacities of all edges of a node zero

*requires k edge update operations where k is degree of the node

• Finding augmenting paths is time consuming.

• Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004]

- Reuses search trees after each augmentation.

- Empirically shown to be substantially faster.

• Our Idea

– Reuse search trees from previous graph cut computation

– Saves us search tree creation tree time [O(#edges)]

– Search trees have to be modified to make them consistent with new graphs

– Constrain the search of augmenting paths

• New paths must contain at least one updated edge

Improving the Algorithm

Reusing Search Trees

c’ = measure of change in the energy

– Running time

• Dynamic algorithm (c’ + re-create search tree )

• Improved dynamic algorithm (c’)

• Video Segmentation Example

- Duplicate image frames (No time is needed)

Dynamic Graph Cut vs Active Cuts

• Our method flow recycling

• AC cut recycling

• Both methods: Tree recycling

Experimental Analysis

MRF consisting of 2x105 latent variables connected in a 4-neighborhood.

Running time of the dynamic algorithm

Part II SOCP for MRF

Aim• Accurate MAP estimation of pairwise Markov random fields

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Labelling m = {1, -1, -1, 1}

Random Variables V = {V1,..,V4}

Label Set L = {-1,1}

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1 + 2

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13

Minimum Cost Labelling = MAP estimate

Pr(m) exp(-Cost(m))

V1 V2 V3 V4

Label ‘-1’

Label ‘1’

Objectives• Applicable to all types of neighbourhood relationships• Applicable to all forms of pairwise costs• Guaranteed to converge (Convex approximation)

MotivationSubgraph Matching - Torr - 2003, Schellewald et al - 2005

Unary costs are uniform

ABCD A

MotivationSubgraph Matching - Torr - 2003, Schellewald et al - 2005

| d(mi,mj) - d(Vi,Vj) | <

YES NO

Potts Model

Pairwise Costs

Motivation

ABCD A

Subgraph Matching - Torr - 2003, Schellewald et al - 2005

Motivation

ABCD A

Subgraph Matching - Torr - 2003, Schellewald et al - 2005

MotivationMatching Pictorial Structures - Felzenszwalb et al - 2001

Part likelihood Spatial Prior

Outline

Texture

(x,y,,)

Motivation

(x,y,,)

• Unary potentials are negative log likelihoods

Valid pairwise configuration

Potts Model

Matching Pictorial Structures - Felzenszwalb et al - 2001

YES NO

Motivation

(x,y,,)

Pr(Cow)Image

• Unary potentials are negative log likelihoodsMatching Pictorial Structures - Felzenszwalb et al - 2001

Valid pairwise configuration

Potts Model

YES NO

Outline• Integer Programming Formulation

• Previous Work

• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model

• Applications– Subgraph Matching– Pictorial Structures

Integer Programming Formulation2

Label ‘-1’

Label ‘1’Unary Cost

Unary Cost Vector u = [ 5 2 ; 2 4 ]T

Labelling m = {1 , -1}

Label vector x = [ -1

V1 = 1

; 1 -1 ]T

Recall that the aim is to find the optimal x

Label ‘-1’

Label ‘1’Unary Cost

Unary Cost Vector u = [ 5 2 ; 2 4 ]T

Label vector x = [ -1 1 ; 1 -1 ]T

Sum of Unary Costs = 12

∑i ui (1 + xi)

Label ‘-1’

Label ‘1’Pairwise Cost

0Cost of V1 = -1 and V1 = -1

0Cost of V1 = -1 and V2 = -1

Cost of V1 = 0-1and V2 = 11 0

Pairwise Cost Matrix P

Label ‘-1’

Sum of Pairwise Costs14

∑ij Pij (1 + xi)(1+xj)

Label ‘0’

Labelling m = {1 , 0}

Sum of Pairwise Costs14

∑ij Pij (1 + xi +xj + xixj)

∑ij Pij (1 + xi + xj + Xij)=

X = x xT Xij = xi xj

Integer Programming FormulationConstraints

• Each variable should be assigned a unique label

∑ xi = 2 - |L|i Va

• Marginalization constraint

∑ Xij = (2 - |L|) xij Vb

Integer Programming FormulationChekuri et al. , SODA 2001

x* = argmin 12

∑ ui (1 + xi) + 14

∑ Pij (1 + xi + xj + Xij)

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi {-1,1}

X = x xT

ConvexNon-Convex

Key Point

• In modern optimization the issue is not linear vs non linear but convex vs nonconvex

• We want to find a convex and good relaxation of the integer program.

• Previous Work

Linear Programming Formulation

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi {-1,1}

X = x xT

Chekuri et al. , SODA 2001Retain Convex Part

Relax Non-convex Constraint

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X = x xT

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X becomes a variable to be optimized

Feasible Region (IP) x {-1,1}, X = x2

FeasibleRegionfor X.

Feasible Region (IP)Feasible Region (Relaxation 1)

x {-1,1}, X = x2

x [-1,1], X = x2

Feasible Region (IP)Feasible Region (Relaxation 1)Feasible Region (Relaxation 2)

x {-1,1}, X = x2

x [-1,1], X = x2

x [-1,1]

• Bounded algorithms proposed by Chekuri et al, SODA 2001

• -expansion - Komodakis and Tziritas, ICCV 2005

• TRW - Wainwright et al., NIPS 2002

• TRW-S - Kolmogorov, AISTATS 2005

• Efficient because it uses Linear Programming

• Not accurate

Semidefinite Programming Formulation

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi {-1,1}

X = x xT

Lovasz and Schrijver, SIAM Optimization, 1990Retain Convex Part

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X = x xT

Retain Convex Part

Lovasz and Schrijver, SIAM Optimization, 1990

1 x1 x2... xn

Rank = 1

Xii = 1

Positive SemidefiniteConvex

Non-Convex

1 x1 x2... xn

Xii = 1

Positive SemidefiniteConvex

Schur’s Complement

BTA-1 I

0 C - BTA-1B

I A-1B

A 0 C -BTA-1B 0

X - xxT 0

0 X - xxT

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X = x xT

Retain Convex PartLovasz and Schrijver, SIAM Optimization, 1990

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

Xii = 1 X - xxT 0

Retain Convex PartLovasz and Schrijver, SIAM Optimization, 1990

Feasible Region (IP) x {-1,1}, X = x2

Feasible Region (IP)Feasible Region (Relaxation 1)

x {-1,1}, X = x2

x [-1,1], X = x2

x {-1,1}, X = x2

x [-1,1], X = x2

x [-1,1], X x2

• Formulated by Lovasz and Schrijver, 1990

• Finds a full X matrix

• Max-cut - Goemans and Williamson, JACM 1995

• Max-k-cut - de Klerk et al, 2000

• Torr AI Stats for labeling problem (2003 TR 2002)

•Accurate, but not efficient •as Semidefinite Programming algorithms slow

Previous Work - Overview

LP SDP

ExamplesTRW-S,

-expansion

Max-k-Cut

Torr 2003

Accuracy Low High

Efficiency High Low

Is there a Middle Path ???

• Previous Work

Second Order Cone Programming

Second Order Cone || v || t OR || v ||2 st

x2 + y2 z2

Minimize fTx

Subject to || Ai x+ bi || <= ciT x + di

i = 1, … , L

Linear Objective Function

Affine mapping of Second Order Cone (SOC)

Constraints are SOC of ni dimensions

Feasible regions are intersections of conic regions

Second Order Cone Programming

|| v || t tI v

LP SOCP SDP

0 t2 - vTv

• Previous Work

First quick definition:Matrix Dot Product

A B = ∑ij Aij Bij

A11 A12

A21 A22

B11 B12

B21 B22

= A11 B11 + A12 B12 + A21 B21 + A22 B22

SDP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

Xii = 1 X - xxT 0

We will derive SOCP relaxation from the SDP relaxation

FurtherRelaxation

1-D ExampleX - xxT 0

X - x2 ≥ 0

For two semidefinite matrices, the dot product is non-negative

SOC of the form || v ||2 st, s is a scalar constant.

x {-1,1}, X = x2

x [-1,1], X = x2

x [-1,1], X x2

SOCP Relaxation

For 1D: Same as the SDP formulation

2-D Example

X11 X12

X21 X22

x1x1 x1x2

x2x1 x2x2

xxT =x1

2 x1x2

2-D Example(X - xxT)

1 - x12 X12-x1x2. 0

0 0 X12-x1x2 1 - x22

-1 x1 1

C1. 0 C1 0

1 - x12 X12-x1x2

. 00 0

0 1 X12-x1x2 1 - x22

LP Relaxation-1 x2 1

1 - x12 X12-x1x2

. 01 1

1 1 X12-x1x2 1 - x22

(x1 + x2)2 2 + 2X12

SOC of the form || v ||2 st

1 - x12 X12-x1x2

. 01 -1

-1 1 X12-x1x2 1 - x22

(x1 - x2)2 2 - 2X12

General form of SOC constraints

Consider a matrix C1 = UUT 0

(X - xxT)

||UTx ||2 X . C1

C1 . 0

Continue for C2, C3, … , Cn

Kim and Kojima, 2000

SOCP Relaxation

How many constraints for SOCP = SDP ?

Infinite. For all C 0

We specify constraints similar to the 2-D example

SOCP RelaxationMuramatsu and Suzuki, 2001

Constraints hold for the above semidefinite matrices

+ c + d

Constraints hold for the linear combination

a+c+d c-d

c-d b+c+d

d 0Includes all semidefinite matrices where

Diagonal elements Off-diagonal elements

SOCP Relaxation - A

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

Xii = 1 X - xxT 0

SOCP Relaxation - A

x* = argmin 12

∑ ui (1 + xi) + 14

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

(xi + xj)2 2 + 2Xij (xi - xj)2 2 - 2Xij

Specified only when Pij 0 i.e. sparse!!

Triangular Inequality

• At least two of xi, xj and xk have the same sign

• At least one of Xij, Xjk, Xik is equal to one

Xij + Xjk + Xik -1Xij - Xjk - Xik -1-Xij - Xjk + Xik -1-Xij + Xjk - Xik -1

• SOCP-B = SOCP-A + Triangular Inequalities

• Previous Work

Robust Truncated ModelPairwise cost of incompatible labels is truncated

Potts ModelTruncated Linear Model

Truncated Quadratic Model

• Robust to noise

• Widely used in Computer Vision - Segmentation, Stereo

Robust Truncated ModelPairwise Cost Matrix can be made sparse

P = [0.5 0.5 0.3 0.3 0.5]

Q = [0 0 -0.2 -0.2 0]

Reparameterization

Sparse Q matrix Fewer constraints

Compatibility Constraint

Q(ma, mb) < 0 for variables Va and Vb

Relaxation ∑ Qij (1 + xi + xj + Xij) < 0

SOCP-C = SOCP-B + Compatibility Constraints

SOCP Relaxation

• More accurate than LP

• More efficient than SDP

• Time complexity - O( |V|3 |L|3)

• Same as LP

• Approximate algorithms exist for LP relaxation

• We use |V| 10 and |L| 200

• Previous Work

Subgraph MatchingSubgraph Matching - Torr - 2003, Schellewald et al - 2005

Unary costs are uniform

ABCD A

Pairwise costs form a Potts model

Subgraph Matching

• 1000 pairs of graphs G1 and G2

• #vertices in G2 - between 20 and 30

• #vertices in G1 - 0.25 * #vertices in G2

• 5% noise to the position of vertices

• NP-hard problem

Subgraph MatchingMethod Time

(sec)Accuracy (%)

LP 0.85 6.64

LBP 0.2 78.6

GBP 1.5 85.2

SDP-A 35.0 93.11

SOCP-A 3.0 92.01

SOCP-B 4.5 94.79

SOCP-C 4.8 96.18

• Previous Work

Pictorial Structures

(x,y,,)

Matching Pictorial Structures - Felzenszwalb et al - 2001

Outline

Texture

Pictorial Structures

(x,y,,)

Unary costs are negative log likelihoods

Pairwise costs form a Potts model

| V | = 10 | L | = 200

Pictorial StructuresROC Curves for 450 +ve and 2400 -ve images

Conclusions• We presented an SOCP relaxation to solve MRF

• More efficient than SDP

• More accurate than LP, LBP, GBP

• #variables can be reduced for Robust Truncated Model

• Provides excellent results for subgraph matching and pictorial structures

Future Work

• Quality of solution– Additive bounds exist– Multiplicative bounds for special cases ??– What are good C’s.

• Message passing algorithm ??– Similar to TRW-S or -expansion– To handle image sized MRF

Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M....

Documents

Transcript of Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M....

Dielectric relaxations of poly(N-isopropylacrylamide ...chem1.bnu.edu.cn/zks/Papers/authorreprints(Suwenjuan).pdf · Dielectric relaxations of poly(N-isopropylacrylamide) microgels

Comparison of 100 torr and 200 torr bpsg layer deposited using sub atmospheric chemical vapour deposition (sacvd) process

CVD III - School of Chemistry · 2018. 10. 16. · Pirani Ionization gauge Pennig) ressure Range 60-1 torr 60 - 10-2 Torr 60 - 10-1 Torr 10-3 torr .5 to 10-4 Torr 0-3 - 10-9 torr

Electronic & Ionic Conduction & Correlated Dielectric Relaxations in Molecular Solids · 2017-12-23 · Electronic & Ionic Conduction & Correlated Dielectric Relaxations in Molecular

Diplomarbeit: Ladungsträger-Relaxations- und ... · Ladungsträger-Relaxations- und Rekombinationsmechanismen in Halbleiterquantenpunkten Diplomarbeit von Mark Rossi 4. …

Semialgebraic Relaxations using Moment-SOS Hierarchiescas.ee.ic.ac.uk/people/vmagron/slides/sierra_magron.pdf · 2014-09-22 · Semialgebraic Relaxations using Moment-SOS Hierarchies

Rolete Torr Med

Abstract arXiv:2007.03629v2 [cs.LG] 8 Jul 2020Strong Generalization and Efﬁciency in Neural Programs Yujia Li, Felix Gimeno, Pushmeet Kohli, Oriol Vinyals DeepMind {yujiali,fgimeno,pushmeet,vinyals}@google.com

New Convex Relaxations for MRF Inference With Unknown Graphsopenaccess.thecvf.com/content_ICCV_2019/papers/Wang_New_Con… · New Convex Relaxations for MRF Inference with Unknown

Relaxations in curfew

Approximating Pareto Curves using Semideﬁnite Relaxations · Approximating Pareto Curves using Semideﬁnite Relaxations Victor Magron1,2 Didier Henrion1,2,3 Jean-Bernard Lasserre1,2

Towards higher order semideﬁnite relaxations for … · Towards higher order semideﬁnite relaxations for Max-Cut Franz Rendl ... Barahona, Mahjoub (1986): ...

Cryo-Torr High-Vacuum Pump Installation, Operation and Maintenance Instructionsappliedcryogenics.com/manuals/Cryo-torr instructions.pdf · · 2017-04-06Cryo-Torr High-Vacuum Pump

Associative Hierarchical Random Fields · 1 Associative Hierarchical Random Fields L’ubor Ladický 1, Chris Russell , Pushmeet Kohli, Philip H. S. Torr Abstract—This paper makes

Hypergraphic LP Relaxations for Steiner Treesdeepc/PUBS/CKP-full.pdfThese exact LP relaxations for spanning trees in graphs inspire the hypergraphic relaxations for the Steiner tree

Learning to Navigate the Energy Landscape€¦ · Learning to Navigate the Energy Landscape Julien Valentin 1 ;3 Angela Dai 2 Matthias Nießner 2 Pushmeet Kohli 1 Philip Torr 3 Shahram

POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of.

arXiv:1805.09028v2 [cs.CV] 26 Oct 2018Salzmann , Pushmeet Kohli , Philip H.S. Torr , and M. Pawan Kumar Abstract. Dense conditional random elds (CRFs) have become a popular framework

Rounding Sum of Squares Relaxations

Simultaneous Segmentation and 3D Pose Estimation of Humans Philip H.S. Torr Pawan Kumar, Pushmeet Kohli, Matt Bray Oxford Brookes University Arasanathan.