Bundle Adjustment - University of Minnesotahspark/CSci5980/Lec17_Supp...v Bob Alice X 1 P bob P...

Bundle Adjustment

Bob Alice

bobP aliceP

P X P X

Black: given variables Red: unknowns

Camera & Point Jacobian

Bob Alice

bobP aliceP

P X P X

Bob Alice

bobP aliceP

P X P X

p pp,X

u ww u

v u wv v

Bob Alice

bobP aliceP

P X P X

p pp,X

u ww u

v u wv v

X Xp,X

u ww u

v u wv v

Camera & Point Jacobian ( ) ( )

p ,X p ,XJ J J

p Xj i j i

ij ij ij

Bob Alice

bobP aliceP

p ,X p ,XJ J J

p Xj i j i

ij ij ij

J1,bobpJ 2 70 1,bobXJ

1,aliceXJ1,alicepJ2 70

Bob Alice

bobP aliceP

p ,X p ,XJ J J

p Xj i j i

ij ij ij

J1,bobpJ 2 70 1,bobXJ

2,bobpJ 2 70 2,bobXJ

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 3x4

Cam 2 Cam 3 Pt 2 Pt 3

Camera 1

Camera 2 Camera 3

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

Camera 1

Camera 2 Camera 3

J = pJXJ

Camera 3D point

p XJ J J

J J =T

p XJ J JT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

p XJ J JT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

p XJ J JT T

p p p X

X p X X

J J J J A BJ J

B DJ J J J

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

T T T- ( )

p p p X p p

X p X X X X

J J J J J epb X

XJ J J J J ef

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

p p p X p

X p X X X

J J I J J ep

XJ J J J I eor

eA B p

B D X eor

T T T- ( )

p p p X p p

X p X X X X

J J J J J epb X

XJ J J J J ef

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

Inversion of block diagonal matrix can be efficiently computed.

T T T- ( )

p p p X p p

X p X X X X

J J J J J epb X

XJ J J J J ef

p p p X p

X p X X X

J J I J J ep

XJ J J J I e

eA B p

B D X e

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

eA B p

B D X e

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

eA B p

B D X e

eA B pI BD I BD

B D X0 I 0 I e

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

eA B p

B D X e

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T

e BD epA BD B 0

XB D e

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

-1 T-A BD BNote: is Schur complement of D

eA B p

B D X e

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T

e BD epA BD B 0

XB D e

-1-1 T -1

p A BD B e BD e

X D e B p

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

Small size matrix

eA B p

B D X e

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T -1

p A BD B e BD e

X D e B p

-1-1 T

e BD epA BD B 0

XB D e

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

# of images if visible

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

( )b pf

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

( )b pf

d = d I

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

( )b pf

T- ( )

J eb X

d = d I

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

( )b pf

T- ( )

J eb X

p p p X

X p X X

J J I J JA B

B D J J J J I

d = d I

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

( )b pf

T- ( )

J eb X

-1-1 T -1

p A BD B e BD e

X D e B p

p p p X

X p X X

J J I J JA B

B D J J J J I

d = d I

1D Camera Bundle Adjustment

Camera pose

2D points

nPoints = 5; nCameras = 3; X = rand(nPoints, 2) + 2; c = rand(nCameras,2); for i = 1 : nCameras m = X-ones(nPoints,1)*c(i,:); theta = atan2(m(:,2), m(:,1)); theta = mean(theta); C(i).c = c(i,:)'; theta = theta; C(i).R = [sin(theta) -cos(theta); cos(theta) sin(theta)]; u = C(i).R*[eye(2) -C(i).c] * [X'; ones(1,nPoints)]; C(i).m = u(1,:)./u(2,:); C(i).c = c(i,:)'+0.3*randn(2,1); theta = theta + 0.3*randn(1,1); C(i).R = [sin(theta) -cos(theta); cos(theta) sin(theta)]; end X = X + 0.3*randn(size(X)); [C X] = BA(C, X);

BundleAdjustment1D.m

Data generation

Camera pose

2D points

Data generation

Ground truth projection

Camera pose

2D points

Data generation

Add noise

Camera pose

2D points

Data generation

Add noise

1D bundle adjustment

Camera pose

2D points

function [C X] = BA(C, X) lambda = 0.5; nIters = 100; xp = []; for i = 1 : length(C) theta=atan2(C(i).R(2,2), C(i).R(2,1)); xp = [xp; C(i).c; theta]; end xx = []; for i = 1 : size(X,1) xx = [xx; X(i,:)']; end

for j = 1 : nIters Jp = []; Jx = []; D_inv = []; err = []; for iPoint = 1 : size(X,1) X1 = xx(2*(iPoint-1)+1:2*iPoint); d = zeros(2,2); for iC = 1 : length(C) c = xp(3*(iC-1)+1:3*(iC-1)+2); theta = xp(3*(iC-1)+3); R = [sin(theta) -cos(theta); cos(theta) sin(theta)]; df_dc = JacobianC1D(R, c, X1); df_dR = JacobianR1D(R, c, X1)*JacobianQ1D(theta); df_dx = JacobianX1D(R, c, X1); j1 = zeros(1,3*length(C)); j1(:,3*(iC-1)+1:3*iC) = [df_dc df_dR]; j2 = zeros(1,2*size(X,1)); j2(:,2*(iPoint-1)+1:2*iPoint) = df_dx; Jp = [Jp; j1]; Jx = [Jx; j2]; d = d + df_dx'*df_dx; u = R * [eye(2) -c] * [X1;1]; u = u/u(2); u1 = C(iC).m; e = [u1(iPoint) - u(1)]; err = [err; e]; end d = d + lambda*eye(2); D_inv = blkdiag(D_inv, inv(d)); end ep = Jp' * err; ex = Jx' * err; A = Jp'*Jp + lambda*eye(3*length(C)); B = Jp'*Jx; delta_p = inv(A-B*D_inv*B') * (ep-B*D_inv*ex); delta_x = D_inv * (ex-B'*delta_p); xp = xp + delta_p; xx = xx + delta_x; end

for iC = 1 : length(C) C(iC).c = xp(3*(iC-1)+1:3*(iC-1)+2); theta = xp(3*(iC-1)+3); C(iC).R = [sin(theta) -cos(theta); cos(theta) sin(theta)]; end for iX = 1 : size(X,1) X(iX,:) = xx(2*(iX-1)+1:2*iX); end

Model update

mean_delta_X = 0.0265 mean_delta_p = 0.2648

Why camera and point error do not converge to zero while reprojection error converges?

Model update

mean_delta_X = 0.0265 mean_delta_p = 0.2648

Why camera and point error do not converge to zero while reprojection error converges?

Because the bundle adjustment is still up to scale and orietation.

Data Capture

• Enough baseline between images • Share common 3D structure • Multiple planes

Reference image

• An image in the middle may be a good choice of reference image because you can find many matches across all images.

Reference Image Selection

Matching Ref. with other images

Reference image

,f ju,f i

,f iu ,f j

u1u = K u

Indicator of no match

In addition to Mx and My, you may want to have an indicator matrix for visibility.

Camera Pose Estimation

Choose an initial pair of images to compute camera pose • Have enough number of matches •

Camera Pose Estimation

Find Next Best Image

Reconstructed image index

Find the image that has maximum number of 3D-2D matches

3D-2D correspondence: u X

= P K R t (3)R SOwhere

14x x x x1 1 1 1 1 1 1

21y y y y1 1 1 1 1 1 1

23x x x x

24y y y y

m m m m m m m

pX Y Z -u X -u Y -u Z -u

x 11 12 13 14

31 32 33 34

p X p Y p Z pu

p X p Y p Z py 21 22 23 24

31 32 33 34

p X p Y p Z pu

p X p Y p Z p

Camera Registration

If K is given,

= P K R t (3)R SOwhere

K R t p p p p

-11 2 3= R K p p p

-1 T1 2 3 22

K p p p U V

TR UV : SVD cleanup

: Translation and scale recovery

Camera Registration

Reprojection error:

X CK -R

3+4 parameters

u ww u

v u wv v

Camera Registration

T T( ) ( ) ( )( )

p p pp I b p

f f ff

Camera Registration

Triangulation of Unreconstructed Points

This is a point that is not reconstructed yet while it matches with reconstructed images.

Bob Alice

bobP aliceP

Baseline

General camera pose

1 bob=1 1

Two 3D vectors are parallel.

bob1 1

u XP 0

bob1 1

u XP 0 : Knowns

: Unknowns

alice1

Point Triangulation

Bob Alice

bobP aliceP

bob bob bob

alice alice alice

v w yE

u ww u

v u wv v

T T( ) ( ) ( )( )

x x xx b x

f f ff

Nonlinear Point Triangulation

T T( ) ( ) ( )( )

x x xx b x

f f ff

u ww u

v u wv v

Damping factor (Levenberg-Marquardt algorithm)

T T( ) ( ) ( )( )

x x xx I b x

f f ff

Bob Alice

bobP aliceP

P X P X

p pp,X

u ww u

v u wv v

X Xp,X

u ww u

v u wv v

Bundle Adjustment

Cam 1 Pt 1 Pt 4

# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)

Camera 1

Camera 2 Camera 3

Visibility Reasoning

p p p X

X p X X

J J J J A BJ J

B DJ J J J

J J =T

T T - ( )

pJ J J b X

Normal equation:

eA B p

B D X e

eA B pI BD I BD

B D X0 I 0 I e

-1-1 T

e BD epA BD B 0

XB D e

-1-1 T -1

p A BD B e BD e

X D e B p

Jacobian Sparsity

T T( ) ( ) ( )( )

p p pp I b p

f f ff p XJ J J

dM# of points

( )b pf

T- ( )

J eb X

-1-1 T -1

p A BD B e BD e

X D e B p

p p p X

X p X X

J J I J JA B

B D J J J J I

d = d I

Bundle Adjustment - University of Minnesotahspark/CSci5980/Lec17_Supp...v Bob Alice X 1 P bob P...

Documents

Transcript of Bundle Adjustment - University of Minnesotahspark/CSci5980/Lec17_Supp...v Bob Alice X 1 P bob P...

Converting Alice&Bob Protocol Speci cations to Tamarin · Formalization of Alice&Bob Notation. We have indicated that Alice&Bob notation su ers from ambiguities in many cases and

SMART UMA Alice-to-Bob sharing.

Grab a Unit 6 Election Theory Packet! · 1st Alan/Alice Bob/Betty Carl/Cathy Carl/Cathy 2nd Bob/Betty Alan/Alice Bob/Betty Bob/Betty 3rd Carl/Cathy Carl/Cathy Alan/Alice Alan/Alice

Review: Game theory - Computer Sciencelazebnik/fall11/lec10_game_theory2.pdf · Review: Game theory Alice: Testify Alice: Refuse Bob: Testify-5,-5 -10,0 Bob: 0 -10 -1 -1 Refuse, ,

Asymmetric Message Franking - Cornell Universitytyagi/slides/amf.pdfSetting: End-to-end encrypted messaging Platform Alice Bob 3 From: Alice To: Bob Hello - Conﬁdentiality and Integrity

A tale of Alice, Bob, Eve & a stretched analogy…

screening for Bob - UChicago Economics · Alice Bob Engage Bob Don’t engage Bob −A, B 1, 1 0, 0 Treat Alice well Treat Alice poorly Alice Bob Engage Bob Don’t engage Bob 1,

Fault Tolerant Service Function Chainingali/papers/sigcomm20-ftc-slides.pdf · Alice Bob Alice ⬄Apple Bob ⬄Bing Alice ⬄Apple Bob ⬄Bing. NAT Consistent State Replication 5

(a) Alice and Bob are back together. Today Alice wants to send Bob a message that is secret and also authenticated, so that Bob "knows" the message came.

Message Authentication - University of Washington · 2020. 6. 5. · •Lets Bob validate unaltered message came from Alice •Doesn’t let Bob convince Charlie that Alice sent the

arXiv:1802.05102v2 [quant-ph] 15 Feb 2018 · Alice Bob 8.2 ± 0.4 Bob Alice 7.5 ± 0.3 Bob Bob 8.5 ± 0.4 Reference time: ( 10.1 ± 0.1) ns Table I Time measurement results. The four

INFO 290 Social Computing - UC Berkeley School of Informationcourses.ischool.berkeley.edu/i290-sc/f11/lib/exe... · Alice Bob Gigi David Cathy Elisa Frank Henry Alice Bob 10 Gigi

Information Security Technologyist.ujs.edu.cn/file/jianggao/11 Key Management.pdf · Bootstrap problem: how do Alice, Bob begin? Alice can.t send it to Bob in the clear! Assume trusted

Cryptography (continued). Enabling Alice and Bob to Communicate Securely m m m Alice Eve Bob m.

for (person us: (alice, bob, charlie)) { person match ... · for (person us: (alice, bob, charlie)) { person match { case Alice'") case Person("Bob", Bob!" ) case Person(name, age)

Bob Braden, Sandy Ginoza, Alice Hagens - RFC-Editor

WhatsApp Backdoor (/Bug?) file The Backdoor Alice Bob Alice and Bob meet in person and verify the fingerprint 2 1 3

Quantum communication from Alice to Bob

Alice and Bob: a Love Story - Grenslops 140 · Alice assumes Bob will make a beeline for the appetizers, which Alice models as Bob moving straight up the y-axis: his path is thus

Aaron Steele. Project Motivation Alice Bob 50+GB Free.