Reconstructing Relief Surfaces

George Vogiatzis, Philip Torr, Steven George Vogiatzis, Philip Torr, Steven Seitz and Roberto CipollaSeitz and Roberto Cipolla

BMVC 2004BMVC 2004

Stereo reconstruction problem:

InputInput Set of images of a scene I={ISet of images of a scene I={I11,…,I,…,IKK}}

Camera matrices PCamera matrices P11,…,P,…,PKK

OutputOutput Surface modelSurface model

Shape parametrisation

Disparity-map parametrisationMRF formulation – good optimisation

techniques exist (Graph-cuts, Loopy BP)MRF smoothness is viewpoint dependentDisparity is unique per pixel – only

functions represented

Shape parametrisation

Volumetric parametrisation – e.g. Level-sets, Space carving etc.Able to cope with non-functionsConvergence properties not well

understood, Local minimaMemory intensiveFor Space carving, no simple way to

impose surface smoothness

Solution ?

Cast volumetric methods in MRF frameworkCast volumetric methods in MRF framework Key assumption: Approximate scene geometry Key assumption: Approximate scene geometry

givengiven Benefits:Benefits:

General surfaces can be representedGeneral surfaces can be represented Optimisation is tractable (MRF solvers)Optimisation is tractable (MRF solvers) Occlusions are approximately modelledOcclusions are approximately modelled Smoothness is viewpoint independentSmoothness is viewpoint independent

MRFs

The labelling problem:The labelling problem:

MRFs

A set of random variables hA set of random variables h11,…,h,…,hMM

A binary neighbourhood relation A binary neighbourhood relation NN defined on the defined on the variablesvariables

Each can take a label out of a set HEach can take a label out of a set H11,…,H,…,HLL

CCii(h(hii) (Labelling cost)) (Labelling cost) CCi,ji,j(h(hii,h,hjj) for (i,j)) for (i,j)N N (Compatibility cost)(Compatibility cost)

-log P(h-log P(h11,…,h,…,hMM) = ) = CCii(h(hii) + ) + CCi,ji,j(h(hii,h,hjj) )

MRF inference

Minimise Minimise CCii(h(hii) + ) + CCi,ji,j(h(hii,h,hjj) ) Not in polynomial time in general caseNot in polynomial time in general case Special cases (e.g. no loops or 2 label MRF) Special cases (e.g. no loops or 2 label MRF)

solved exactly solved exactly General cases solved approximately via General cases solved approximately via

Graph-cuts or Loopy Belief Propagation. Graph-cuts or Loopy Belief Propagation. Approx. 10-15mins for MRF with 250,000 Approx. 10-15mins for MRF with 250,000 nodes.nodes.

Relief Surfaces Approximate base surfaceApproximate base surface

Triangulated feature matchesTriangulated feature matches Visual hull from silhouettesVisual hull from silhouettes Initialised by handInitialised by hand

Relief Surfaces labels :

Relief Surfaces labelling cost :

Low costHigh cost

Xi

ni

Xi+hini

Ci(hi)=photoconsistency(Xi+hini)

Relief Surfaces Compatibility cost :

Low cost

Xi

ni

nj Xj

Xi+hini

Xj+hjnj

Relief Surfaces Neighbour cost :

Ci,j(hi, hj)= ||(Xi+hini)-(Xj+hjnj)||

Xi

ni

Xi+hini

Xj+hjnj

High cost

Relief Surfaces

Base surface is the Base surface is the occluding volumeoccluding volume If base surface ‘contains’ true surface (e.g. visual If base surface ‘contains’ true surface (e.g. visual

hull) thenhull) then Points on the base surface Points on the base surface XXii are not visible by are not visible by

cameras they shouldn’t be [Kutulakos, Seitz cameras they shouldn’t be [Kutulakos, Seitz 2000]2000]

Approximation:Approximation: Visibility is propagated from Visibility is propagated from XXii to to XXii+h+hiinnii

Loopy Belief Propagation

Iterative message passing algorithmIterative message passing algorithm mm(t)(t)

i,ji,j (h (hjj) is the message passed from i to j at time step t) is the message passed from i to j at time step t

It is a L-dimensional vectorIt is a L-dimensional vector Represents what node i ‘believes’ about the true state of Represents what node i ‘believes’ about the true state of

node j.node j.

min min CCii(h(hii) + ) + CCi,ji,j(h(hii,h,hjj))

i jmi,j

Loopy Belief Propagation

Message passing rule:Message passing rule:

After convergence, optimal state is given byAfter convergence, optimal state is given by

mm(t+1)(t+1)i,ji,j (h (hjj)= min)= min{{ C Cijij(h(hii,h,hjj) +C) +Cii(h(hii) +) +mm(t)(t)

k,ik,i (h (hii))}}k N(i)hi

hhii**= min= min{{CCii(h(hii) +) +mm(())

k,ik,i (h (hii))}}k N(i)hi

i jmi,j

Loopy Belief Propagation

O(LO(L22) to compute a message (L is number of ) to compute a message (L is number of allowable heights)allowable heights)

Message passing schedule can be asynchronous Message passing schedule can be asynchronous which can accelerate convergence [Tappen & which can accelerate convergence [Tappen & Freeman ICCV 03]Freeman ICCV 03]

Iterative Scheme BP is memory intensive. BP is memory intensive. Can consider few possible labels at a timeCan consider few possible labels at a time After convergence we ‘zoom in’ to heights close to the optimalAfter convergence we ‘zoom in’ to heights close to the optimal

Evaluation

Artificial deformed sphereArtificial deformed sphere Textured with random paternTextured with random patern 20 images20 images 40,000 sample points on sphere base surface40,000 sample points on sphere base surface

True surface Texture-mapped Reconstruction

Evaluation

Benchmark: 2-view, disparity based Benchmark: 2-view, disparity based Loopy Loopy Belief Propagation Belief Propagation [Sun et al ECCV02][Sun et al ECCV02]

BP run on 10 pairs of nearby viewsBP run on 10 pairs of nearby views Compare Disparity Maps given by Compare Disparity Maps given by

2-view BP2-view BP Relief surfacesRelief surfaces Ground truthGround truth

Evaluation2-view BP Relief surface Ground truth

2-view BP2-view BP Relief surf.Relief surf.

MSEMSE 1.466 pixels1.466 pixels 0.499 pixels0.499 pixels

% of correct disparities% of correct disparities 75.9%75.9% 79.2%79.2%

Results

SarcophagusSarcophagus

Results

SarcophagusSarcophagus

Results

SarcophagusSarcophagus

Results

Building facadeBuilding facade

Results

Building facadeBuilding facade

Results

Base surface Relief surfaceRelief surfacewith texture

Stone carvingStone carving

Summary

MRF methods can be extended in the MRF methods can be extended in the volumetric domainvolumetric domain

AdvantagesAdvantages General surfaces can be representedGeneral surfaces can be represented Optimisation is tractable (MRF solvers)Optimisation is tractable (MRF solvers) Smoothness is viewpoint independentSmoothness is viewpoint independent

Future work

Photoconsistency beyond Lambertian Photoconsistency beyond Lambertian surface models. (Optimise both height and surface models. (Optimise both height and surface normal fields)surface normal fields)

Change in topologyChange in topology In cases where CIn cases where Cmnmn(h(hmm,h,hnn)=|| h)=|| hmm-h-hnn|||| or || hor || hmm--

hhnn||||2 2 we can compute messages in O(L) time we can compute messages in O(L) time instead of O(Linstead of O(L22) (Felzenszwalb & ) (Felzenszwalb & Huttenlocher CVPR 04).Huttenlocher CVPR 04).