OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Triangulation MethodsSeminar work

Robotics and Medicine SS 09Institut fur Prozessrechentechnik, Automation und Robotik

(IPR)

Zlatka MihaylovaSupervisor: M.Phys. Matteo Ciucci

July 13, 2009

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

IntroductionThe human visual perception systemEpipolar geometry

Triangulation methods3D point reconstructionComputation of the Fundamental matrix F

Practical examplesActive triangulation

ConclusionAppliance in the medical roboticsClosing words

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

Stereovision

I Why are we able to percept the relative distance to all objects?

I Why is it so important to measure the distance to andbetween the objects?

I How can another point of view help in solving this problem?

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

Stereovision principle

fbx1

d

I Disparity p = b fd , where f represents the lens focal length

I p is proportional to the stereoscopic base b and inverselyproportional to d - the distance to the measured object.

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

Basics of the epipolar geometry

A B

C

e e’

c c’

�

epipolar line

baseline

epipolar line

epipolar plane

I The baseline connects camera centers A and B and intersectsthe image planes in the epipoles e and e ′.

I The epipolar plane π is defined by the camera centers and the3D object point C .

I The ambiguous projection of C .

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

Basics of the epipolar geometry

A B

C

e e’

c c’

�

epipolar line

baseline

epipolar line

epipolar plane

I The baseline connects camera centers A and B and intersectsthe image planes in the epipoles e and e ′.

I The epipolar plane π is defined by the camera centers and the3D object point C .

I The ambiguous projection of C .

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

Basics of the epipolar geometry

A B

C

e e’

c c’

�

epipolar line

baseline

epipolar line

epipolar plane

I The baseline connects camera centers A and B and intersectsthe image planes in the epipoles e and e ′.

I The epipolar plane π is defined by the camera centers and the3D object point C .

I The ambiguous projection of C .Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

The Fundamental Matrix F and the camera matrices P , P ′

I F is a 3× 3 matrix representing the mapping between a pointin the first image and epipolar line in the second image.

I For all pairs of image points c and c ′ the correspondencecondition holds:

c ′T

Fc = 0 (1)

I The camera matrices P and P ′ satisfy the conditions c = PCand c ′ = P ′C for every point correspondence

I In the case, when we deal with calibrated cameras, it iscleverer to compute the Essential Matrix E , which isspecialization of F :

F = P ′−T

EP−1 (2)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

The Fundamental Matrix F and the camera matrices P , P ′

I F is a 3× 3 matrix representing the mapping between a pointin the first image and epipolar line in the second image.

I For all pairs of image points c and c ′ the correspondencecondition holds:

c ′T

Fc = 0 (1)

I The camera matrices P and P ′ satisfy the conditions c = PCand c ′ = P ′C for every point correspondence

I In the case, when we deal with calibrated cameras, it iscleverer to compute the Essential Matrix E , which isspecialization of F :

F = P ′−T

EP−1 (2)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

The human visual perception systemEpipolar geometry

The Fundamental Matrix F and the camera matrices P , P ′

I F is a 3× 3 matrix representing the mapping between a pointin the first image and epipolar line in the second image.

I For all pairs of image points c and c ′ the correspondencecondition holds:

c ′T

Fc = 0 (1)

I The camera matrices P and P ′ satisfy the conditions c = PCand c ′ = P ′C for every point correspondence

I In the case, when we deal with calibrated cameras, it iscleverer to compute the Essential Matrix E , which isspecialization of F :

F = P ′−T

EP−1 (2)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

General approach

Algorithm:

I Take two images of the scene, separated by a baseline

I Identify the point correspondences in the images

I Apply the triangulation rules: compute F , P and P ′

I Find these two lines, which intersection defines the searchedworld point

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

General approach

Algorithm:

I Take two images of the scene, separated by a baseline

I Identify the point correspondences in the images

I Apply the triangulation rules: compute F , P and P ′

I Find these two lines, which intersection defines the searchedworld point

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

General approach

Algorithm:

I Take two images of the scene, separated by a baseline

I Identify the point correspondences in the images

I Apply the triangulation rules: compute F , P and P ′

I Find these two lines, which intersection defines the searchedworld point

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

General approach

Algorithm:

I Take two images of the scene, separated by a baseline

I Identify the point correspondences in the images

I Apply the triangulation rules: compute F , P and P ′

I Find these two lines, which intersection defines the searchedworld point

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

Identification the point correspondences in the images

The most difficult part is finding the point correspondencesautomatically! Robust pattern matching algorithm needed!

I Harris corner detector: simple but scales dependentI Successful combination of Harris and Laplacian detectors:

www.robots.ox.ac.uk/∼vgg/research/affine/det evalfiles/mikolajczyk ijcv2004.pdf

I Laplacian and Difference of Gaussian (DoG) ”points ofinterest” detectors

I Salient region detector: www.robots.ox.ac.uk/∼vgg/research/affine/det eval files/kadir04.pdf

I Maximally stable extremal regions (MSER)(http://www.robots.ox.ac.uk/∼vgg/research/affine/det eval files/matas bmvc2002.pdf - specially developed for the stereoproblem analysis)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

Algorithms for computing F

Having F computed gives us the possibility to estimate the scenepoints. There are some algorithms available:

I Eight point algorithm: F has 8 degrees of freedom, thereforewe need 8 unique point pairs to compute it. Every pair definesequation, which solution contains the nine coefficients of F

I Algebraic minimization algorithm: based on the eight pointalgorithm, but tries to minimize the algebraic error caused bynoisy measurement.

I Gold standard algorithm: dealing with the problem ofGaussian noise. This approach uses statistical methods forsolving the triangulation puzzle, namely computing F byminimizing the Likelihood function. (proposed in the book:”Multiple View Geometry in Computer Vision” - RichardHartley and Andrew Zisserman)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

Algorithms for computing F

Having F computed gives us the possibility to estimate the scenepoints. There are some algorithms available:

I Eight point algorithm: F has 8 degrees of freedom, thereforewe need 8 unique point pairs to compute it. Every pair definesequation, which solution contains the nine coefficients of F

I Algebraic minimization algorithm: based on the eight pointalgorithm, but tries to minimize the algebraic error caused bynoisy measurement.

I Gold standard algorithm: dealing with the problem ofGaussian noise. This approach uses statistical methods forsolving the triangulation puzzle, namely computing F byminimizing the Likelihood function. (proposed in the book:”Multiple View Geometry in Computer Vision” - RichardHartley and Andrew Zisserman)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

3D point reconstructionComputation of the Fundamental matrix F

Algorithms for computing F

Having F computed gives us the possibility to estimate the scenepoints. There are some algorithms available:

I Eight point algorithm: F has 8 degrees of freedom, thereforewe need 8 unique point pairs to compute it. Every pair definesequation, which solution contains the nine coefficients of F

I Algebraic minimization algorithm: based on the eight pointalgorithm, but tries to minimize the algebraic error caused bynoisy measurement.

I Gold standard algorithm: dealing with the problem ofGaussian noise. This approach uses statistical methods forsolving the triangulation puzzle, namely computing F byminimizing the Likelihood function. (proposed in the book:”Multiple View Geometry in Computer Vision” - RichardHartley and Andrew Zisserman)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Active triangulation

Light spot technique

I Simple construction: laser ray, lens, detector (CCD or PSD)I Advantages: fast, accurate, independent from surface colorI Disadvantages: the surface should be no ideal mirror

h

ӨPSD

lase

r

p

measured object

p’

q

q’

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Active triangulation

Stripe projection

I The object’s surface manipulates the scan lineI Resulting displacement in the light stripe ∼ to obj. distance

reference surface

camera

laser

hӨ

d

measured object

(a) (b)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Active triangulation

Projection of encoded patterns

I Disadvantage of the stripe projection: too slowI Correspondence problem by static line pattern projectionI Solutions: Binary coding, Grey coding, Phase shifted pattern

projection, Colored pattern (the picture is taken from thebook ”Digitale Bildverarbeitung” - Bernd Jahne)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Appliance in the medical roboticsClosing words

Polaris R©, NDI

I Standard optical tracking system in medicine, produced byNorthern Digital Inc. (NDI)

I Offers passive, active and hybrid tracking.

I The triangulated points are fixed on the surgical instrument.

I http://www.ndigital.com/medical/polarisfamily.php

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Appliance in the medical roboticsClosing words

A.R.T. R© Systems

I Advanced Realtime Tracking GmbH (A.R.T. GmbH)I Multiple camera systems - 3, 4, 5 cameras for better resultsI Example system: smARTtrack - two ARTtrack2 cameras

mounted on a rigid bar, so that no calibration needed.I different configurations depending on focal length, angle

between both cameras, baselineI http://www.ar-tracking.de/smARTtrack.49.0.html

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Appliance in the medical roboticsClosing words

Da Vinci R© Surgical System

I a) A high-resolution 3D endoscope coupled with two 3-chipcameras take the surgeon ”inside” the patient

I b) The console helps by visualizing the camera records and byrepositioning the surgical camera inside the patient.

I www.intuitivesurgical.com/products/davinci surgicalsystem

(c) (d)

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Appliance in the medical roboticsClosing words

Conclusion

Through the methods of triangulation the robots similar tohumans process the visual information.For triangulation the following prerequisites are needed:

I at least 2 points of view (implemented either with cameras ormixed with light sources)

I object point, placed on a comparably closer distance (not atinfinity)

I statistically stable algorithms for computing the pointcorrespondences, respectively the distance to the world point

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods

OutlineIntroduction

Triangulation methodsPractical examples

Conclusion

Appliance in the medical roboticsClosing words

Questions time

Thank You for Your attention!

Zlatka Mihaylova Supervisor: M.Phys. Matteo Ciucci Triangulation Methods