Andrei Sharf Dan A. Alcantara Thomas Lewiner Chen Greif

Alla Sheffer Nina Amenta Daniel Cohen-Or

Space-time Surface Reconstruction using Incompressible Flow

4D Data acquisition - setting

• Synchronized static cameras • 2-16 views • Capture rates 10-30 fps

• Persistent self occlusions• Low frame rate and resolution• Noise

4D Data acquisition - limitations

Motivation

• Volume is incompressible across time

• Explicitly modeling of mass field:– Compute inside/outside volume– Include physical assumptions– Object is watertight manifold

2D

time mass field

Contribution

• Incompressible mass flow 4D reconstruction– Global system considers all frames simultaneously– Simple formulation on a grid

• General un-constrained deformations

time

Related work

• Marker based [Marschner et al. 2000; Guskov et al. 2003; White et al. 2007]

• Template based [Allen et al. 2002; Anguelov et al. 2004; Zhang et al. 2004; Anguelov et al. 2005, Aguiar et al. 2008; Bradley et al. 2008]

• Point correspondence and registration[Shinya 2004; Wang et al. 2005, Anuar and Guskov 2004, Pekelny and Gotsman 2008, Chang and Zwicker 2008; Li et al. 2008]

• Surface based space-time [Wand et al. 2007; Mitra et al. 2007; Suessmuth et al. 2008]

Mitra et al. 2007

Li et al. 2008

Anuar et al. 2004

Zhang et al. 2004

Guskov et al. 2003

FLOW reconstruction

• In: 3D point cloud frames• Out: watertight surface• Explicit volume modeling

– 4D solid on a grid– Characteristic function

3D reconstruction techniques fail

• Surface reconstruction of individual frames fails

2D example

known outside 0

known inside 1

unknown [0-1]

1D representation

• Domain: space time grid• Material: characteristic function xti values mass amount

at each cell• Flow: amount of material vti,j moving from cell xti to cell xt+1j

vti,i-1 vti,i+1

xt+1i-1 xt+1i xt+1i+1t+1

t

vti,itime

xti

Higher dimensions generalization

• Regular 4D grid on top of 3D scan frames • Space-time adjacency relationships:

1D 2D 3D

FLOW physical constraints

• Mass preservation: material in cell equals to material flowing into and out of the cell

jij,

ti

t

jij,

1-ti

t

v x

v xtime

FLOW physical constraints

• Spatial continuity: values spatially adjacent to be identical everywhere, except across boundaries

€

Fc xti( ) = xt

i − xti-1( )

0.8

i=1K n∑

t∑

space

FLOW Physical Constraints

• Flow momentum: flow direction should be smooth across time

€

Fm vti, j( ) = vt

i, j − vt +1j, j+(j-1)( )

2

i, j,t= 0K m -1∑

Constrained Minimization Problem

• Optimization:

• Constraints:– Incompressibility constraints– Boundary values

mc FFF 1

Challenges

• Sublinear exponentiterative reweighted least squares

• Huge matricesfine-tuned iterative solver

• Mass stabilityboundary constraints,

clamping

€

Fc xit

( ) = xit − xi-1

t( )

0.8

i=1K n∑

t∑

Sublinear exponent

Iteratively Reweighted Least Squares:

• from previous iteration• small close to discontinuities• converges with good init and few outliers

wit

wit

€

Fc x it

( ) = x it − x i-1

t( )

0. 8

i=1K n∑

t∑

=1

x it − x i-1

t( )

2−0. 8 x it − x i-1

t( )

2

i=1K n∑

t∑

€

Fc x it

( ) = x it − x i-1

t( )

0. 8

i=1K n∑

t∑

=1

x it − x i-1

t( )

2−0. 8 x it − x i-1

t( )

2

i=1K n∑

t∑

2.1

1

t

i-ti

ti xxw 2.1

1

t

i-ti

ti xxw

iteration

time

Huge data problem

Problem size• 20-200 frames x (28)3 grid resolution x 8

variables per cell in time

Reduce initial number of unknowns• Pre-assignment from visibility hull :

inside/outside labels• High resolution per-frame surface

reconstruction

Sharf et al. 07

Lagrange multipliers:

Kx,v

A BT

B 0

x,v

f

g

Kx,v

A BT

B 0

x,v

f

g

Minimization problem

Fc 1 Fm t xit v j,i

t

j t 1 xi

t v j,it -1

j

0

min F Fc 1 Fm

s.t. xit v j,i

t

j v j,i

t -1

j

x,v x,v x,v x,vT

x,v x,vT

0

Matrix engineering

• Iterative, preconditioned with many eigenvalues 1 fast convergence

• Augmenting approach: solve with CG MINRES solver with decreasing tolerance

Kx,v

A BT

B 0

x,v

f

g

Kx,v

A BT

B 0

x,v

f

g

M A BT B 0

0 I

M A BT B 0

0 I

M 1K

M 1K

A BT B

A BT B

Mass stability: clamping/back substitution

: amount of mass at cell (i,t) • : inside, outside• clamp • if then adjacent• back-substitution reduces the system size

xit 0,1

xit 0

xit 1

xit

xit 0,1

xit 0

vi,*t 0

iteration

time

Results (2D) – empty frames completion

Results – hand puppet from 2 views

Results - garments

Results – large deformation

Results 3D+time

• 20 frames at constant resolution • Solver converges in 100 iterations.• Time: 1 minute per-frame• 3.73 GHz CPU, memory requirements up to 4.5GB

Thank you!