Bundle Adjustment

v

Bob Alice

X

bobP aliceP

u

pp

X

1

2

geom

2

2

3

2

3

u

P X P X

P X X

u

P

E

x y

Black: given variables Red: unknowns

Camera & Point Jacobian

v

Bob Alice

X

bobP aliceP

u

pp

X

1

2

geom

2

2

3

2

3

u

P X P X

P X X

u

P

E

x y

Black: given variables Red: unknowns

( )

p,X

u

v

w

f w

Camera & Point Jacobian

v

Bob Alice

X

bobP aliceP

u

pp

X

1

2

geom

2

2

3

2

3

u

P X P X

P X X

u

P

E

x y

Black: given variables Red: unknowns

( )

p,X

u

v

w

f w 2

2

( )

p pp,X

p p

p p

u ww u

u

w w

v u wv v

w

w

f

2

2

( )

X Xp,X

X X

X X

u ww u

u

w w

v u wv v

ww

f

Camera & Point Jacobian

v

Bob Alice

X

bobP aliceP

u

pp

X

1

2

geom

2

2

3

2

3

u

P X P X

P X X

u

P

E

x y

Black: given variables Red: unknowns

( )

p,X

u

v

w

f w 2

2

( )

p pp,X

p p

p p

u ww u

u

w w

v u wv v

w

w

f

Camera & Point Jacobian

v

Bob Alice

X

bobP aliceP

u

pp

X

1

2

geom

2

2

3

2

3

u

P X P X

P X X

u

P

E

x y

Black: given variables Red: unknowns

( )

p,X

u

v

w

f w 2

2

( )

p pp,X

p p

p p

u ww u

u

w w

v u wv v

w

w

f

2

2

( )

X Xp,X

X X

X X

u ww u

u

w w

v u wv v

ww

f

Camera & Point Jacobian

v

Bob Alice

1X

bobP aliceP

u

pp

X

Black: given variables Red: unknowns

Camera & Point Jacobian ( ) ( )

p X

p ,X p ,XJ J J

p Xj i j i

ij ij ij

j i

f f

J1,bobpJ 2 70 1,bobXJ

1,aliceXJ1,alicepJ2 70

v

Bob Alice

1X

bobP aliceP

u

pp

X

Black: given variables Red: unknowns

Camera & Point Jacobian ( ) ( )

p X

p ,X p ,XJ J J

p Xj i j i

ij ij ij

j i

f f

J1,bobpJ 2 70 1,bobXJ

1,aliceXJ1,alicepJ2 70

2X

2,bobpJ 2 70 2,bobXJ

2,aliceXJ2,alicepJ2 70

2 30

2 30

2 30

2 30

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 3x4

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

J JTsize of : 24x24

x J J J b f(x)-1T T

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

J =

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

J JTsize of : 24x24

x J J J b f(x)-1T T

Main computational bottle neck

Cam 2 Cam 3 Pt 2 Pt 3

Pt 1

Pt 2

Pt 3

Pt 4

Camera 1

Camera 2 Camera 3

Camera trajectory

3D point

# of cameras: 3009 # of 3D points: 1,298,317

# of unknowns: 7x3009+3x1,298,317

J JTsize of : 6,167,690x6,167,690

J =

J = pJXJ

Camera 3D point

p XJ J J

J J =T

p XJ J JT T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

p XJ J JT T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

p XJ J JT T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T T T

T T T- ( )

p p p X p p

X p X X X X

J J J J J epb X

XJ J J J J ef

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T T T

T T T- ( )

p p p X p p

X p X X X X

J J J J J epb X

XJ J J J J ef

T T

T T

p p p X p

X p X X X

J J I J J ep

XJ J J J I eor

T

p

X

eA B p

B D X eor

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

T T T

T T T- ( )

p p p X p p

X p X X X X

J J J J J epb X

XJ J J J J ef

Normal equation:

T T

T T

p p p X p

X p X X X

J J I J J ep

XJ J J J I eor

T

p

X

eA B p

B D X eor

-1 -1

T

- -

p

X

eA B pI BD I BD

B D X0 I 0 I e

1

d

D

dM

-1

1

-1

-1

d

D

dM

Inversion of block diagonal matrix can be efficiently computed.

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T

p

X

eA B p

B D X e1

d

D

dM

-1

1

-1

-1

d

D

dM

Inversion of block diagonal matrix can be efficiently computed.

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T

p

X

eA B p

B D X e1

d

D

dM

-1

1

-1

-1

d

D

dM

Inversion of block diagonal matrix can be efficiently computed.

-1 -1

T

- -

p

X

eA B pI BD I BD

B D X0 I 0 I e

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T

p

X

eA B p

B D X e1

d

D

dM

-1

1

-1

-1

d

D

dM

Inversion of block diagonal matrix can be efficiently computed.

-1 -1

T

- -

p

X

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T

T

--

p X

X

e BD epA BD B 0

XB D e

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T

p

X

eA B p

B D X e

-1 -1

T

- -

p

X

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T

T

--

p X

X

e BD epA BD B 0

XB D e

-1-1 T -1

-1 T

- -

-

p X

X

p A BD B e BD e

X D e B p

-1 T-A BD BNote: is Schur complement of D

1

d

D

dM

-1

1

-1

-1

d

D

dM

Inversion of block diagonal matrix can be efficiently computed.

T T

T

TT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Xf

Normal equation:

T

p

X

eA B p

B D X e

-1 -1

T

- -

p

X

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T

T

--

p X

X

e BD epA BD B 0

XB D e

-1-1 T -1

-1 T

- -

-

p X

X

p A BD B e BD e

X D e B p

-1 T-A BD BNote: is Schur complement of D

1

d

D

dM

-1

1

-1

-1

d

D

dM

Inversion of block diagonal matrix can be efficiently computed.

Small size matrix