Minimal Surfaces for Stereo

Chris Buehler, Steven J. Gortler, Michael F. Cohen, Leonard McMillan

MIT, HarvardMicrosoft Research, MIT

Motivation

• Optimization based stereo over greed based– No early commitment– Enforce interactions: each pixel sees unique item– Penalize interactions: non-smoothness

Stereo by Optimization

• Early algorithms: dynamic programming– (Baker ‘81, Belumeur & Mumford ‘92…)– Don’t generalize beyond 2 camera, single scanline

Stereo by Optimization

• Recent Algorithms: iterative expansion– (… Kolmogorov & Zabih ‘01)– very general– NP-Complete

• Local opt found quickly in practice

• Recent algorithms: MIN-CUT– (Roy & Cox ‘96, Ishikawa & Geiger ‘98) – Polynomial time global optimum– New interpretation to such methods

Contributions

• Stereo as a discrete minimal surface problem• Algorithms: Polynomial time globally optimal

surface– Using MIN-CUT (Sullivan ‘90)– Build from shortest path

• Applications to stereo vision– Rederive previous MIN-CUT stereo approaches– New 3-camera stereo formulation (Ayache ‘88)

Planar Graph Shortest Path

• Given: an embedded planar graph– faces,

edges, vertices

Planar Graph Shortest Path

• A non negative cost on each edge

57

Planar Graph Shortest Path

• Two boundary points on the exterior of the complex

Planar Graph Shortest Path

• Find minimal curve: (collection of edges) with given boundary

Planar Graph For stereo

Camera Left Camera Right

Selected MatchSelected Occlusion

Algorithms

• Classic: Dijkstra’s– Works even for non-planar graphs

• Wacky: use duality– But this will generalize to higher dimension

Duality

Duality

• face vertex • edge cross edge

- same cost57

Duality

• Split exterior

Source

Sink

Source

Duality

• Add source and sink

Cuts

Source

Sink

• Cuts of dual graph = partitions of dual verts• Cost = sum of dual edges spanning the partition• MIN-CUT can be found in polynomial time

Source

SinkCuts

• Claim: Primalization of MIN-CUT will be shortest path

Sink

SourceSource

SinkWhy this works

• Cuts of dual graph = partitions of dual verts

Sink

SourceSource

SinkWhy this works

• Partition of dual verts = partition of primal faces

Sink

SourceSource

Sink

Source

SinkWhy this works

• Partition of primal faces = primal path

Sink

SourceSource

Sink

Source

SinkWhy this works

• Cuts in dual correspond to paths in primal

• MIN-CUT in dual corresponds to shortest path in primal

Same idea works for surfaces!

Increasing the dimension

Planar graph: verts, edges, faces cost on edges boundary: 2 points on exterior sol: min path

Spacial compex: verts, edges, faces, cells cost on faces boundary: loop on exterior sol: min surface

Increasing the dimension

Planar graph: verts, edges, faces boundary: 2 points on exterior sol: min path

Spacial compex: verts, edges, faces, cells cost on faces boundary: loop on exterior sol: min surface

Increasing the dimension

Planar graph: verts, edges, faces boundary: 2 points on exterior sol: min path

Spacial compex: verts, edges, faces, cells cost on faces boundary: loop on exterior sol: min surface

Dual construction for min surf

• face vertex • edge cross edge

Sink

Source

• cell vertex • face cross edge

MIN-CUT primalizes to min surf

Checkpoint

• Solve for minimal paths and surfaces– MIN-CUT on dual graph

• Apply these algorithms to stereo vision

Flatland Stereo

Camera Left Camera Right

Geometric interpretation of Cox et al. 96

pixel

Flatland Stereo

Camera Left Camera Right

Geometric interpretation of Cox et al. 96

pixel

Flatland Stereo

Camera Left Camera Right

Cost: unmatched/discontinuity, β

Flatland Stereo

Camera Left Camera Right

Cost: correspondence quality

Flatland Stereo

Camera Left Camera Right

Camera Left Camera Right

MatchUnmatched

Uniqueness & monotonicity solution is directed pathFlatland Stereo

Camera Left Camera Right

MatchOcclusion, discontinuity

Note: unmatched pixels also function as discontinuities

Flatland Stereo

Flatland to Fatland

Camera Left Camera Right

Flatland to Fatland

Camera Left Camera Right

2 cameras, 3d

2 cameras, 3d

One Cuboid Among Many

Solve for minimal surface

Geometric interpretation IG98

Three Camera

Rectification (Ayache ‘88)

Three Camera

Three Camera

Three Camera

Three Camera

One cuboid

Dual graph of one cuboid

One Cuboid Among Many

Solve for minimal surface

More divisions of middle cell

More expressive decomposition

Complexity

• Vertices and edges: 20 n d– n: pixels per image– d: max disparity

• Time complexity O((nd)2 log(nd))

• About 1 min

ResultsLL image

RC

KZ01

MS

LL image

RC

KZ01

MS

Future

• Application of MS to n cameras– Monotonicity/oriented manifold enforces more

than uniqueness– see Kolmogorov & Zabih (today 11:00am)

• Other applications of MS