Aplicaciones de los métodos de reconstrucción estéreo...
Transcript of Aplicaciones de los métodos de reconstrucción estéreo...
![Page 1: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/1.jpg)
Aplicaciones de los métodosde reconstrucción estéreo multivista
Dr. Guillermo Gallego
Grupo de Tratamiento de Imágenes
E.T.S.I. Telecomunicación
Universidad Politécnica de Madrid
Seminario VPULab 30/11/2012
Guillermo Gallego (GTI-UPM) 3D Reconstruction VPULab UAM 2012 1 / 77
![Page 2: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/2.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction2
/ 77
![Page 3: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/3.jpg)
Ideas of this talk
1 Advances in technology arise from connections between geometry,algebra, optimization, statistics, etc.
2 So, what's in your toolbox?Review two problems: monocular and binocular vision.
3 Overview of recent results in multi-view stereo (3D) reconstruction.
Guillermo Gallego (GTI-UPM) 3D Reconstruction3
/ 77
![Page 4: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/4.jpg)
Ideas of this talk
1 Advances in technology arise from connections between geometry,algebra, optimization, statistics, etc.
2 So, what's in your toolbox?Review two problems: monocular and binocular vision.
3 Overview of recent results in multi-view stereo (3D) reconstruction.
Guillermo Gallego (GTI-UPM) 3D Reconstruction3
/ 77
![Page 5: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/5.jpg)
Ideas of this talk
1 Advances in technology arise from connections between geometry,algebra, optimization, statistics, etc.
2 So, what's in your toolbox?Review two problems: monocular and binocular vision.
3 Overview of recent results in multi-view stereo (3D) reconstruction.
Guillermo Gallego (GTI-UPM) 3D Reconstruction3
/ 77
![Page 6: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/6.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction4
/ 77
![Page 7: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/7.jpg)
Motivation
Understanding human vision has always fascinated people......and implementing it on a computer for practical applications, too.
Today's topic: 3D reconstruction.Goal: Build a 3D model of the scene under study and �nd thepositions of the objects in front of the camera.
How to generate such 3D model?
Need to study the image formation process (geometry andphotometry).Geometric descriptors/primitives at hand: points, planes, lines, conics,quadrics, curves, surfaces, etc.Simplest camera model: pin-hole model (e.g. no lens distortion).
Guillermo Gallego (GTI-UPM) 3D Reconstruction5
/ 77
![Page 8: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/8.jpg)
Motivation
Understanding human vision has always fascinated people......and implementing it on a computer for practical applications, too.
Today's topic: 3D reconstruction.Goal: Build a 3D model of the scene under study and �nd thepositions of the objects in front of the camera.
How to generate such 3D model?
Need to study the image formation process (geometry andphotometry).Geometric descriptors/primitives at hand: points, planes, lines, conics,quadrics, curves, surfaces, etc.Simplest camera model: pin-hole model (e.g. no lens distortion).
Guillermo Gallego (GTI-UPM) 3D Reconstruction5
/ 77
![Page 9: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/9.jpg)
Motivation
Understanding human vision has always fascinated people......and implementing it on a computer for practical applications, too.
Today's topic: 3D reconstruction.Goal: Build a 3D model of the scene under study and �nd thepositions of the objects in front of the camera.
How to generate such 3D model?
Need to study the image formation process (geometry andphotometry).Geometric descriptors/primitives at hand: points, planes, lines, conics,quadrics, curves, surfaces, etc.Simplest camera model: pin-hole model (e.g. no lens distortion).
Guillermo Gallego (GTI-UPM) 3D Reconstruction5
/ 77
![Page 10: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/10.jpg)
3D reconstruction examples
Guillermo Gallego (GTI-UPM) 3D Reconstruction6
/ 77
![Page 11: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/11.jpg)
3D reconstruction examples: Video Sur�nghttp://vimeo.com/15990190
Guillermo Gallego (GTI-UPM) 3D Reconstruction7
/ 77
![Page 12: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/12.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction8
/ 77
![Page 13: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/13.jpg)
Classi�cation of 3D reconstruction methods
Pre-calibrated (camera parameters are known)
Image based Bottom-up approach: from local image features to buildingcomplex 3D models.
Voxel based Space discretization. No full 3D model of scene objects.Object based Top-down approach: relating 3D models to image features.
Online Calibrated: camera parameters are estimated using...
Scene constraints Calibration methods (right angles of the objects,calibration pattern, etc.)
Geometric constraints Autocalibration methods.
Guillermo Gallego (GTI-UPM) 3D Reconstruction9
/ 77
![Page 14: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/14.jpg)
Classi�cation of 3D reconstruction methods
Pre-calibrated (camera parameters are known)
Image based Bottom-up approach: from local image features to buildingcomplex 3D models.
Voxel based Space discretization. No full 3D model of scene objects.Object based Top-down approach: relating 3D models to image features.
Online Calibrated: camera parameters are estimated using...
Scene constraints Calibration methods (right angles of the objects,calibration pattern, etc.)
Geometric constraints Autocalibration methods.
Guillermo Gallego (GTI-UPM) 3D Reconstruction9
/ 77
![Page 15: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/15.jpg)
Classi�cation of 3D reconstruction methods
Pre-calibrated (camera parameters are known)
Image based Bottom-up approach: from local image features to buildingcomplex 3D models.
Voxel based Space discretization. No full 3D model of scene objects.Object based Top-down approach: relating 3D models to image features.
Online Calibrated: camera parameters are estimated using...
Scene constraints Calibration methods (right angles of the objects,calibration pattern, etc.)
Geometric constraints Autocalibration methods.
Guillermo Gallego (GTI-UPM) 3D Reconstruction10
/ 77
![Page 16: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/16.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction11
/ 77
![Page 17: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/17.jpg)
Processing Steps
1 Digital image acquisition
2 Feature detection and extraction
3 Structure from Motion
1 Projective calibration2 Auto-calibration3 Euclidean reconstruction
4 3D triangulation and texture selection
5 VRML presentation
Guillermo Gallego (GTI-UPM) 3D Reconstruction12
/ 77
![Page 18: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/18.jpg)
Processing Steps
1 Digital image acquisition
2 Feature detection and extraction
3 Structure from Motion
1 Projective calibration2 Auto-calibration3 Euclidean reconstruction
4 3D triangulation and texture selection
5 VRML presentation
Guillermo Gallego (GTI-UPM) 3D Reconstruction13
/ 77
![Page 19: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/19.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction14
/ 77
![Page 20: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/20.jpg)
Background materialRecall concepts from the slides of Chapter 2
Projective geometry in the plane P2 and in space P3
(homogeneous coordinates).
Pin-hole camera model: x∼ PX.Projective camera model P= (π1,π2,π3)
>. Axial planes {π1,π2},principal plane π3.
Finite (or Euclidean) camera matrix: P∼ K[R | t] (in Ch. 2, my K≡ A).
Epipolar geometry: Fundamental matrix F. Epipolar constraintx′>Fx = 0.
Some linear algebra (QR decomposition, SVD, linear solvers, etc.)
Guillermo Gallego (GTI-UPM) 3D Reconstruction15
/ 77
![Page 21: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/21.jpg)
Structure from motion. Problem Statement.
Given the projections of n points in m images
{xij}i=1,...,m; j=1,...,n,
compute the 3D points {X j} j=1,...,n
and the camera parameters {Ki,Ri, C̃i}i=1,...,msuch that the projections agree with the measurements:
xij ∼ Ki[Ri | −RiC̃i]X j
Typically, # equations = 2mn� # unknowns = 11m+3n−7.Guillermo Gallego (GTI-UPM) 3D Reconstruction
16/ 77
![Page 22: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/22.jpg)
Structure from motion. Problem Statement.
Given the projections of n points in m images
{xij}i=1,...,m; j=1,...,n,
compute the 3D points {X j} j=1,...,n
and the camera parameters {Ki,Ri, C̃i}i=1,...,msuch that the projections agree with the measurements:
xij ∼ Ki[Ri | −RiC̃i]X j
Typically, # equations = 2mn� # unknowns = 11m+3n−7.Guillermo Gallego (GTI-UPM) 3D Reconstruction
16/ 77
![Page 23: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/23.jpg)
Structure from motion. Problem Statement.
Given the projections of n points in m images
{xij}i=1,...,m; j=1,...,n,
compute the 3D points {X j} j=1,...,n
and the camera parameters {Ki,Ri, C̃i}i=1,...,msuch that the projections agree with the measurements:
xij ∼ Ki[Ri | −RiC̃i]X j
Typically, # equations = 2mn� # unknowns = 11m+3n−7.Guillermo Gallego (GTI-UPM) 3D Reconstruction
16/ 77
![Page 24: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/24.jpg)
Structure from motion. Problem Statement.
Optimization approach
In the presence of Gaussian noise on the coordinates of the observed pointsin the images, seek the Maximum Likelihood solution, which is the one thatminimizes the reprojection error
∑i, j
d2Euc
(xi
j,Ki[Ri|−RiC̃i]X j
).
Solution: Bundle Adjustment
Solve the above optimization problem in a large number of variables (andequations).
Guillermo Gallego (GTI-UPM) 3D Reconstruction17
/ 77
![Page 25: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/25.jpg)
Structure from motion. Problem Statement.
Optimization approach
In the presence of Gaussian noise on the coordinates of the observed pointsin the images, seek the Maximum Likelihood solution, which is the one thatminimizes the reprojection error
∑i, j
d2Euc
(xi
j,Ki[Ri|−RiC̃i]X j
).
Solution: Bundle Adjustment
Solve the above optimization problem in a large number of variables (andequations).
Guillermo Gallego (GTI-UPM) 3D Reconstruction17
/ 77
![Page 26: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/26.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction18
/ 77
![Page 27: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/27.jpg)
SfM. Projective Calibration
Unique reconstruction is obtained up to a projective transformation(homography) of space, xi
j ∼ PiX j = (PiH)(H−1X j) = P̂iX̂ j.
Geometric entities based on the number of cameras:
2 cameras: Fundamental matrix F
3 cameras: Trifocal tensor T4 camera: Quadrifocal tensor QMulti-camera geometry: bundle adjustment
Guillermo Gallego (GTI-UPM) 3D Reconstruction19
/ 77
![Page 28: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/28.jpg)
SfM. Projective Calibration
Unique reconstruction is obtained up to a projective transformation(homography) of space, xi
j ∼ PiX j = (PiH)(H−1X j) = P̂iX̂ j.
Geometric entities based on the number of cameras:
2 cameras: Fundamental matrix F
3 cameras: Trifocal tensor T4 camera: Quadrifocal tensor QMulti-camera geometry: bundle adjustment
Guillermo Gallego (GTI-UPM) 3D Reconstruction19
/ 77
![Page 29: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/29.jpg)
SfM. Projective Calibration
Unique reconstruction is obtained up to a projective transformation(homography) of space, xi
j ∼ PiX j = (PiH)(H−1X j) = P̂iX̂ j.
Geometric entities based on the number of cameras:
2 cameras: Fundamental matrix F
3 cameras: Trifocal tensor T4 camera: Quadrifocal tensor QMulti-camera geometry: bundle adjustment
Guillermo Gallego (GTI-UPM) 3D Reconstruction19
/ 77
![Page 30: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/30.jpg)
SfM. Projective Calibration
Unique reconstruction is obtained up to a projective transformation(homography) of space, xi
j ∼ PiX j = (PiH)(H−1X j) = P̂iX̂ j.
Geometric entities based on the number of cameras:
2 cameras: Fundamental matrix F
3 cameras: Trifocal tensor T4 camera: Quadrifocal tensor QMulti-camera geometry: bundle adjustment
Guillermo Gallego (GTI-UPM) 3D Reconstruction19
/ 77
![Page 31: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/31.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction20
/ 77
![Page 32: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/32.jpg)
SfM. Camera resectioningEstimation of the projection matrix P
Useful for building up incremental reconstructions (3D models) adding onecamera to an existing 3D model.
Problem statement
Given a set of n point correspondences xi↔ Xi, compute the projectionmatrix P of the camera such that xi ∼ PXi for all i = 1, . . . ,n.
Estimation methods:
Linear algorithm (algebraic cost, SVD, eigenvalues)
Non-linear algorithm (geometric cost, Gauss-Newton method)
Robust algorithm (RANSAC)
Guillermo Gallego (GTI-UPM) 3D Reconstruction21
/ 77
![Page 33: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/33.jpg)
SfM. Camera resectioningEstimation of the projection matrix P
Useful for building up incremental reconstructions (3D models) adding onecamera to an existing 3D model.
Problem statement
Given a set of n point correspondences xi↔ Xi, compute the projectionmatrix P of the camera such that xi ∼ PXi for all i = 1, . . . ,n.
Estimation methods:
Linear algorithm (algebraic cost, SVD, eigenvalues)
Non-linear algorithm (geometric cost, Gauss-Newton method)
Robust algorithm (RANSAC)
Guillermo Gallego (GTI-UPM) 3D Reconstruction21
/ 77
![Page 34: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/34.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Projection equation: xi ∼ PXi.Data: {xi↔ Xi}n
i=1. Unknown: P.
How can we measure whether xi ∼ PXi is satis�ed or not?
Cross product: xi×PXi?= 0. This is a set of 3 linear equations in the
entries of P. If xi = (xi,yi,wi)>, 0> −wiX>i yiX>i
wiX>i 0> −xiX>i−yi xiX>i 0>
︸ ︷︷ ︸
Ai (3×12)
π1π2π3
︸ ︷︷ ︸
p
= 0.
But only 2 of them are linearly independent.
Guillermo Gallego (GTI-UPM) 3D Reconstruction22
/ 77
![Page 35: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/35.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Projection equation: xi ∼ PXi.Data: {xi↔ Xi}n
i=1. Unknown: P.
How can we measure whether xi ∼ PXi is satis�ed or not?
Cross product: xi×PXi?= 0. This is a set of 3 linear equations in the
entries of P. If xi = (xi,yi,wi)>, 0> −wiX>i yiX>i
wiX>i 0> −xiX>i−yi xiX>i 0>
︸ ︷︷ ︸
Ai (3×12)
π1π2π3
︸ ︷︷ ︸
p
= 0.
But only 2 of them are linearly independent.
Guillermo Gallego (GTI-UPM) 3D Reconstruction22
/ 77
![Page 36: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/36.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Projection equation: xi ∼ PXi.Data: {xi↔ Xi}n
i=1. Unknown: P.
How can we measure whether xi ∼ PXi is satis�ed or not?
Cross product: xi×PXi?= 0. This is a set of 3 linear equations in the
entries of P. If xi = (xi,yi,wi)>, 0> −wiX>i yiX>i
wiX>i 0> −xiX>i−yi xiX>i 0>
︸ ︷︷ ︸
Ai (3×12)
π1π2π3
︸ ︷︷ ︸
p
= 0.
But only 2 of them are linearly independent.
Guillermo Gallego (GTI-UPM) 3D Reconstruction22
/ 77
![Page 37: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/37.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
How do you know there are only 2 independent equations?
Keep using your linear algebra toolbox: Kronecker and vec operator (stacka matrix column-wise).
vec(ABC) = (C>⊗A)vec(B).
Let ε i.= xi×PXi = [xi]×PXi ∈ R3.
Since it is a vector,ε i = vec(ε>i ) = vec
(([xi]×PXi)
>)= ([xi]>×⊗X>i )vec(P) =−Aip.
Now, since rank(A⊗B) = rank(A) rank(B), we see that
rank([xi]>×⊗X>i ) = rank([xi]
>×) rank(X>i ) = 2 ·1 = 2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction23
/ 77
![Page 38: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/38.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
How do you know there are only 2 independent equations?
Keep using your linear algebra toolbox: Kronecker and vec operator (stacka matrix column-wise).
vec(ABC) = (C>⊗A)vec(B).
Let ε i.= xi×PXi = [xi]×PXi ∈ R3.
Since it is a vector,ε i = vec(ε>i ) = vec
(([xi]×PXi)
>)= ([xi]>×⊗X>i )vec(P) =−Aip.
Now, since rank(A⊗B) = rank(A) rank(B), we see that
rank([xi]>×⊗X>i ) = rank([xi]
>×) rank(X>i ) = 2 ·1 = 2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction23
/ 77
![Page 39: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/39.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
How do you know there are only 2 independent equations?
Keep using your linear algebra toolbox: Kronecker and vec operator (stacka matrix column-wise).
vec(ABC) = (C>⊗A)vec(B).
Let ε i.= xi×PXi = [xi]×PXi ∈ R3.
Since it is a vector,ε i = vec(ε>i ) = vec
(([xi]×PXi)
>)= ([xi]>×⊗X>i )vec(P) =−Aip.
Now, since rank(A⊗B) = rank(A) rank(B), we see that
rank([xi]>×⊗X>i ) = rank([xi]
>×) rank(X>i ) = 2 ·1 = 2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction23
/ 77
![Page 40: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/40.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
How do you know there are only 2 independent equations?
Keep using your linear algebra toolbox: Kronecker and vec operator (stacka matrix column-wise).
vec(ABC) = (C>⊗A)vec(B).
Let ε i.= xi×PXi = [xi]×PXi ∈ R3.
Since it is a vector,ε i = vec(ε>i ) = vec
(([xi]×PXi)
>)= ([xi]>×⊗X>i )vec(P) =−Aip.
Now, since rank(A⊗B) = rank(A) rank(B), we see that
rank([xi]>×⊗X>i ) = rank([xi]
>×) rank(X>i ) = 2 ·1 = 2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction23
/ 77
![Page 41: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/41.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
How do you know there are only 2 independent equations?
Keep using your linear algebra toolbox: Kronecker and vec operator (stacka matrix column-wise).
vec(ABC) = (C>⊗A)vec(B).
Let ε i.= xi×PXi = [xi]×PXi ∈ R3.
Since it is a vector,ε i = vec(ε>i ) = vec
(([xi]×PXi)
>)= ([xi]>×⊗X>i )vec(P) =−Aip.
Now, since rank(A⊗B) = rank(A) rank(B), we see that
rank([xi]>×⊗X>i ) = rank([xi]
>×) rank(X>i ) = 2 ·1 = 2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction23
/ 77
![Page 42: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/42.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Having 2 independent equations Aip = 0 per xi↔ Xi correspondence.How many do we need to estimate p≡ P?
Staking equations, we get a homogeneous linear system of equations
Ap =
A1A2...An
p = 0.
Some more linear algebra...
Trivial solution p = 0 (makes no sense as a projection matrix).
Exact solution p∼ ker(A) is unique (up to a scale) if rank(A) = 11.
Assuming points in general position, we need 2n≥ 11 equations.Therefore, n≥ 6 point correspondences.
Guillermo Gallego (GTI-UPM) 3D Reconstruction24
/ 77
![Page 43: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/43.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Having 2 independent equations Aip = 0 per xi↔ Xi correspondence.How many do we need to estimate p≡ P?
Staking equations, we get a homogeneous linear system of equations
Ap =
A1A2...An
p = 0.
Some more linear algebra...
Trivial solution p = 0 (makes no sense as a projection matrix).
Exact solution p∼ ker(A) is unique (up to a scale) if rank(A) = 11.
Assuming points in general position, we need 2n≥ 11 equations.Therefore, n≥ 6 point correspondences.
Guillermo Gallego (GTI-UPM) 3D Reconstruction24
/ 77
![Page 44: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/44.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Having 2 independent equations Aip = 0 per xi↔ Xi correspondence.How many do we need to estimate p≡ P?
Staking equations, we get a homogeneous linear system of equations
Ap =
A1A2...An
p = 0.
Some more linear algebra...
Trivial solution p = 0 (makes no sense as a projection matrix).
Exact solution p∼ ker(A) is unique (up to a scale) if rank(A) = 11.
Assuming points in general position, we need 2n≥ 11 equations.Therefore, n≥ 6 point correspondences.
Guillermo Gallego (GTI-UPM) 3D Reconstruction24
/ 77
![Page 45: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/45.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Having 2 independent equations Aip = 0 per xi↔ Xi correspondence.How many do we need to estimate p≡ P?
Staking equations, we get a homogeneous linear system of equations
Ap =
A1A2...An
p = 0.
Some more linear algebra...
Trivial solution p = 0 (makes no sense as a projection matrix).
Exact solution p∼ ker(A) is unique (up to a scale) if rank(A) = 11.
Assuming points in general position, we need 2n≥ 11 equations.Therefore, n≥ 6 point correspondences.
Guillermo Gallego (GTI-UPM) 3D Reconstruction24
/ 77
![Page 46: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/46.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
Having 2 independent equations Aip = 0 per xi↔ Xi correspondence.How many do we need to estimate p≡ P?
Staking equations, we get a homogeneous linear system of equations
Ap =
A1A2...An
p = 0.
Some more linear algebra...
Trivial solution p = 0 (makes no sense as a projection matrix).
Exact solution p∼ ker(A) is unique (up to a scale) if rank(A) = 11.
Assuming points in general position, we need 2n≥ 11 equations.Therefore, n≥ 6 point correspondences.
Guillermo Gallego (GTI-UPM) 3D Reconstruction24
/ 77
![Page 47: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/47.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
What if correspondences are not exact, i.e., there is noise?
An exact solution may not exist (case rank(A) = 12).What do we do?
Optimization approach.Find the best p≡ P in the 2-norm sense (least-squares) by minimizingthe residual,
minp‖Ap‖. (1)
Issue: p = 0 still solves (1).
Fix: to have a well-posed problem (and to get rid of the scaleambiguity), supplement (1) with an additional constraint on p.Typically, choose ‖p‖= 1. Why?
Guillermo Gallego (GTI-UPM) 3D Reconstruction25
/ 77
![Page 48: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/48.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
What if correspondences are not exact, i.e., there is noise?
An exact solution may not exist (case rank(A) = 12).What do we do?
Optimization approach.Find the best p≡ P in the 2-norm sense (least-squares) by minimizingthe residual,
minp‖Ap‖. (1)
Issue: p = 0 still solves (1).
Fix: to have a well-posed problem (and to get rid of the scaleambiguity), supplement (1) with an additional constraint on p.Typically, choose ‖p‖= 1. Why?
Guillermo Gallego (GTI-UPM) 3D Reconstruction25
/ 77
![Page 49: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/49.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
What if correspondences are not exact, i.e., there is noise?
An exact solution may not exist (case rank(A) = 12).What do we do?
Optimization approach.Find the best p≡ P in the 2-norm sense (least-squares) by minimizingthe residual,
minp‖Ap‖. (1)
Issue: p = 0 still solves (1).
Fix: to have a well-posed problem (and to get rid of the scaleambiguity), supplement (1) with an additional constraint on p.Typically, choose ‖p‖= 1. Why?
Guillermo Gallego (GTI-UPM) 3D Reconstruction25
/ 77
![Page 50: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/50.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
What if correspondences are not exact, i.e., there is noise?
An exact solution may not exist (case rank(A) = 12).What do we do?
Optimization approach.Find the best p≡ P in the 2-norm sense (least-squares) by minimizingthe residual,
minp‖Ap‖. (1)
Issue: p = 0 still solves (1).
Fix: to have a well-posed problem (and to get rid of the scaleambiguity), supplement (1) with an additional constraint on p.Typically, choose ‖p‖= 1. Why?
Guillermo Gallego (GTI-UPM) 3D Reconstruction25
/ 77
![Page 51: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/51.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
The resulting problem is a classic: minimization of a Rayleigh quotient,
p̂ = arg min‖p‖=1
‖Ap‖= argminp
‖Ap‖2
‖p‖2 = argminp
p>A>App>p
. (2)
Solution: Using Lagrange multipliers, compute the extremals ofF(p,λ ) = ‖Ap‖2 +λ (1−‖p‖2).
p̂ is the eigenvector of A>A corresponding to the smallest eigenvalue:(A>A)p̂ = λminp̂.Using the SVD: p̂ is the right singular vector of A corresponding to thesmallest singular value.
But, what is being minimized?An algebraic cost of the veri�cation of the projection equations.
Can we choose a (better) geometrically meaningful cost to minimize?
Guillermo Gallego (GTI-UPM) 3D Reconstruction26
/ 77
![Page 52: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/52.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
The resulting problem is a classic: minimization of a Rayleigh quotient,
p̂ = arg min‖p‖=1
‖Ap‖= argminp
‖Ap‖2
‖p‖2 = argminp
p>A>App>p
. (2)
Solution: Using Lagrange multipliers, compute the extremals ofF(p,λ ) = ‖Ap‖2 +λ (1−‖p‖2).
p̂ is the eigenvector of A>A corresponding to the smallest eigenvalue:(A>A)p̂ = λminp̂.Using the SVD: p̂ is the right singular vector of A corresponding to thesmallest singular value.
But, what is being minimized?An algebraic cost of the veri�cation of the projection equations.
Can we choose a (better) geometrically meaningful cost to minimize?
Guillermo Gallego (GTI-UPM) 3D Reconstruction26
/ 77
![Page 53: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/53.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
The resulting problem is a classic: minimization of a Rayleigh quotient,
p̂ = arg min‖p‖=1
‖Ap‖= argminp
‖Ap‖2
‖p‖2 = argminp
p>A>App>p
. (2)
Solution: Using Lagrange multipliers, compute the extremals ofF(p,λ ) = ‖Ap‖2 +λ (1−‖p‖2).
p̂ is the eigenvector of A>A corresponding to the smallest eigenvalue:(A>A)p̂ = λminp̂.Using the SVD: p̂ is the right singular vector of A corresponding to thesmallest singular value.
But, what is being minimized?An algebraic cost of the veri�cation of the projection equations.
Can we choose a (better) geometrically meaningful cost to minimize?
Guillermo Gallego (GTI-UPM) 3D Reconstruction26
/ 77
![Page 54: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/54.jpg)
SfM. Camera resectioningLinear algorithm for the estimation of P
The resulting problem is a classic: minimization of a Rayleigh quotient,
p̂ = arg min‖p‖=1
‖Ap‖= argminp
‖Ap‖2
‖p‖2 = argminp
p>A>App>p
. (2)
Solution: Using Lagrange multipliers, compute the extremals ofF(p,λ ) = ‖Ap‖2 +λ (1−‖p‖2).
p̂ is the eigenvector of A>A corresponding to the smallest eigenvalue:(A>A)p̂ = λminp̂.Using the SVD: p̂ is the right singular vector of A corresponding to thesmallest singular value.
But, what is being minimized?An algebraic cost of the veri�cation of the projection equations.
Can we choose a (better) geometrically meaningful cost to minimize?
Guillermo Gallego (GTI-UPM) 3D Reconstruction26
/ 77
![Page 55: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/55.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Can we choose a (better) geometrically meaningful cost to minimize?
Yes! Since there is a metric in the image plane, minimize theEuclidean distance
g(p) =n
∑i=1
d2Euc(xi,PXi).
Again, a geometry + optimization framework!
Solution: use standard techniques from �nite-dimensional optimization(your toolbox).
Steepest descent methodConjugate gradient methodNewton's method
Why didn't we start here? Solution method is slower and iterative.
Guillermo Gallego (GTI-UPM) 3D Reconstruction27
/ 77
![Page 56: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/56.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Can we choose a (better) geometrically meaningful cost to minimize?
Yes! Since there is a metric in the image plane, minimize theEuclidean distance
g(p) =n
∑i=1
d2Euc(xi,PXi).
Again, a geometry + optimization framework!
Solution: use standard techniques from �nite-dimensional optimization(your toolbox).
Steepest descent methodConjugate gradient methodNewton's method
Why didn't we start here? Solution method is slower and iterative.
Guillermo Gallego (GTI-UPM) 3D Reconstruction27
/ 77
![Page 57: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/57.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Can we choose a (better) geometrically meaningful cost to minimize?
Yes! Since there is a metric in the image plane, minimize theEuclidean distance
g(p) =n
∑i=1
d2Euc(xi,PXi).
Again, a geometry + optimization framework!
Solution: use standard techniques from �nite-dimensional optimization(your toolbox).
Steepest descent methodConjugate gradient methodNewton's method
Why didn't we start here? Solution method is slower and iterative.
Guillermo Gallego (GTI-UPM) 3D Reconstruction27
/ 77
![Page 58: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/58.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Can we choose a (better) geometrically meaningful cost to minimize?
Yes! Since there is a metric in the image plane, minimize theEuclidean distance
g(p) =n
∑i=1
d2Euc(xi,PXi).
Again, a geometry + optimization framework!
Solution: use standard techniques from �nite-dimensional optimization(your toolbox).
Steepest descent methodConjugate gradient methodNewton's method
Why didn't we start here? Solution method is slower and iterative.
Guillermo Gallego (GTI-UPM) 3D Reconstruction27
/ 77
![Page 59: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/59.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Newton's method.Assume a quadratic approximation of the cost function
g(p+δp)≈ g(p)+∇g ·δp+ 12 δp>H δp, (3)
where ∇g and H are the gradient and the Hessian of g evaluated at p.
Iteration: start from p0 and update pk+1 = pk +(δp)k using step
H(pk)(δp)k =−∇g(pk).
What initialization p0? For example, the linear method.
Guillermo Gallego (GTI-UPM) 3D Reconstruction28
/ 77
![Page 60: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/60.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Newton's method.Assume a quadratic approximation of the cost function
g(p+δp)≈ g(p)+∇g ·δp+ 12 δp>H δp, (3)
where ∇g and H are the gradient and the Hessian of g evaluated at p.
Iteration: start from p0 and update pk+1 = pk +(δp)k using step
H(pk)(δp)k =−∇g(pk).
What initialization p0? For example, the linear method.
Guillermo Gallego (GTI-UPM) 3D Reconstruction28
/ 77
![Page 61: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/61.jpg)
SfM. Camera resectioningNon-linear algorithm for the estimation of P
Newton's method.Assume a quadratic approximation of the cost function
g(p+δp)≈ g(p)+∇g ·δp+ 12 δp>H δp, (3)
where ∇g and H are the gradient and the Hessian of g evaluated at p.
Iteration: start from p0 and update pk+1 = pk +(δp)k using step
H(pk)(δp)k =−∇g(pk).
What initialization p0? For example, the linear method.
Guillermo Gallego (GTI-UPM) 3D Reconstruction28
/ 77
![Page 62: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/62.jpg)
SfM. Camera resectioningRobust algorithm for the estimation of P
Use RANSAC [Fischler & Bolles 1987], etc. to remove data outliers.
Compute a candidate P
Select a random minimal sample (6 correspondences) and compute P(linear algorithm).Calculate the �distance� from each correspondence to the model.Compute the number of inliers consistent with the model P.
Choose the P with the largest number of inliers.
Re�ne P by minimizing the geometric cost (non-linear algorithm) usingonly the inliers.
Guillermo Gallego (GTI-UPM) 3D Reconstruction29
/ 77
![Page 63: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/63.jpg)
SfM. Camera resectioningRobust algorithm for the estimation of P
Use RANSAC [Fischler & Bolles 1987], etc. to remove data outliers.
Compute a candidate P
Select a random minimal sample (6 correspondences) and compute P(linear algorithm).Calculate the �distance� from each correspondence to the model.Compute the number of inliers consistent with the model P.
Choose the P with the largest number of inliers.
Re�ne P by minimizing the geometric cost (non-linear algorithm) usingonly the inliers.
Guillermo Gallego (GTI-UPM) 3D Reconstruction29
/ 77
![Page 64: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/64.jpg)
SfM. Camera resectioningRobust algorithm for the estimation of P
Use RANSAC [Fischler & Bolles 1987], etc. to remove data outliers.
Compute a candidate P
Select a random minimal sample (6 correspondences) and compute P(linear algorithm).Calculate the �distance� from each correspondence to the model.Compute the number of inliers consistent with the model P.
Choose the P with the largest number of inliers.
Re�ne P by minimizing the geometric cost (non-linear algorithm) usingonly the inliers.
Guillermo Gallego (GTI-UPM) 3D Reconstruction29
/ 77
![Page 65: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/65.jpg)
SfM. Camera resectioningRobust algorithm for the estimation of P
Use RANSAC [Fischler & Bolles 1987], etc. to remove data outliers.
Compute a candidate P
Select a random minimal sample (6 correspondences) and compute P(linear algorithm).Calculate the �distance� from each correspondence to the model.Compute the number of inliers consistent with the model P.
Choose the P with the largest number of inliers.
Re�ne P by minimizing the geometric cost (non-linear algorithm) usingonly the inliers.
Guillermo Gallego (GTI-UPM) 3D Reconstruction29
/ 77
![Page 66: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/66.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction30
/ 77
![Page 67: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/67.jpg)
SfM. Projective Calibration
Recall where we are:
We still need to provide an initialization.
Use 2 views with many common features but su�cient baseline(parallax).
Guillermo Gallego (GTI-UPM) 3D Reconstruction31
/ 77
![Page 68: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/68.jpg)
SfM. Projective Calibration
Recall where we are:
We still need to provide an initialization.
Use 2 views with many common features but su�cient baseline(parallax).
Guillermo Gallego (GTI-UPM) 3D Reconstruction31
/ 77
![Page 69: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/69.jpg)
Fundamental matrix F
Useful for initializing an incremental reconstruction (3D model).
F describes the relative geometry of two views of the same scene.
Plano epipolar
(R t)
e e C C
x x
l l
centro óptico centro
óptico
Movimiento Rígido
X
1
1 2
2
2
2 1
1
Guillermo Gallego (GTI-UPM) 3D Reconstruction32
/ 77
![Page 70: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/70.jpg)
Fundamental matrix F
Problem statement
Given a set of n point correspondences xi↔ x′i between two images,compute the fundamental matrix F satisfying the epipolar constraint
x′>i Fxi = 0 for all i = 1, . . . ,n.
Estimation methods:
Exact method (n = 7). Up to 3 solutions.
Linear algorithm (algebraic cost, SVD, eigenvalues). n≥ 8
Non-linear algorithm (geometric cost, Gauss-Newton method).
Robust algorithm (RANSAC)
Guillermo Gallego (GTI-UPM) 3D Reconstruction33
/ 77
![Page 71: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/71.jpg)
Fundamental matrix F
Problem statement
Given a set of n point correspondences xi↔ x′i between two images,compute the fundamental matrix F satisfying the epipolar constraint
x′>i Fxi = 0 for all i = 1, . . . ,n.
Estimation methods:
Exact method (n = 7). Up to 3 solutions.
Linear algorithm (algebraic cost, SVD, eigenvalues). n≥ 8
Non-linear algorithm (geometric cost, Gauss-Newton method).
Robust algorithm (RANSAC)
Guillermo Gallego (GTI-UPM) 3D Reconstruction33
/ 77
![Page 72: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/72.jpg)
Fundamental matrix FAlgorithms for the estimation of F
Each epipolar constraint is a linear equation in f≡ F:
x′>i Fxi = a>i f = 0.
Check: ai = (x′i⊗xi)> ∈ R9 if f = vec(F>).
Linear method: algebraic cost.
Singularity det(F) = 0 is enforced a posteriori.Stack many equations: Af = 0, with A= (a1,a2, . . . ,an)
> ∈ Rn×9.Non-trivial, unique linear solution f = ker(A) if rank(A) = 8.Least-squares: minimize the Rayleigh quotient ‖Af‖/‖f‖, with
‖Af‖2 =n
∑i=1
(x′>i Fxi)2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction34
/ 77
![Page 73: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/73.jpg)
Fundamental matrix FAlgorithms for the estimation of F
Each epipolar constraint is a linear equation in f≡ F:
x′>i Fxi = a>i f = 0.
Check: ai = (x′i⊗xi)> ∈ R9 if f = vec(F>).
Linear method: algebraic cost.
Singularity det(F) = 0 is enforced a posteriori.Stack many equations: Af = 0, with A= (a1,a2, . . . ,an)
> ∈ Rn×9.Non-trivial, unique linear solution f = ker(A) if rank(A) = 8.Least-squares: minimize the Rayleigh quotient ‖Af‖/‖f‖, with
‖Af‖2 =n
∑i=1
(x′>i Fxi)2.
Guillermo Gallego (GTI-UPM) 3D Reconstruction34
/ 77
![Page 74: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/74.jpg)
Fundamental matrix FAlgorithms for the estimation of F
Non-linear method: geometric cost function
g(f,{Xi}ni=1) =
n
∑i=1
(d2Euc(x̂i,xi)+d2
Euc(x̂′i,x′i)).
where x̂i↔ x̂′i truly satisfy x̂′>i Fx̂i = 0.
Singularity det(F) = 0 is enforced a priori (by parameterization).New variables to be estimated: location of 3D points (that is why weuse it to initialize the reconstruction).Optimization in 12+3n parameters, but structure can be exploited tomake it feasible and fast.Solver: Levenberg-Marquardt algorithm (variant of Newton's method).
RANSAC method for computing F is a standard (OpenCV, etc.)
Guillermo Gallego (GTI-UPM) 3D Reconstruction35
/ 77
![Page 75: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/75.jpg)
Fundamental matrix FAlgorithms for the estimation of F
Non-linear method: geometric cost function
g(f,{Xi}ni=1) =
n
∑i=1
(d2Euc(x̂i,xi)+d2
Euc(x̂′i,x′i)).
where x̂i↔ x̂′i truly satisfy x̂′>i Fx̂i = 0.
Singularity det(F) = 0 is enforced a priori (by parameterization).New variables to be estimated: location of 3D points (that is why weuse it to initialize the reconstruction).Optimization in 12+3n parameters, but structure can be exploited tomake it feasible and fast.Solver: Levenberg-Marquardt algorithm (variant of Newton's method).
RANSAC method for computing F is a standard (OpenCV, etc.)
Guillermo Gallego (GTI-UPM) 3D Reconstruction35
/ 77
![Page 76: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/76.jpg)
Fundamental matrix FAlgorithms for the estimation of F
Non-linear method: geometric cost function
g(f,{Xi}ni=1) =
n
∑i=1
(d2Euc(x̂i,xi)+d2
Euc(x̂′i,x′i)).
where x̂i↔ x̂′i truly satisfy x̂′>i Fx̂i = 0.
Singularity det(F) = 0 is enforced a priori (by parameterization).New variables to be estimated: location of 3D points (that is why weuse it to initialize the reconstruction).Optimization in 12+3n parameters, but structure can be exploited tomake it feasible and fast.Solver: Levenberg-Marquardt algorithm (variant of Newton's method).
RANSAC method for computing F is a standard (OpenCV, etc.)
Guillermo Gallego (GTI-UPM) 3D Reconstruction35
/ 77
![Page 77: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/77.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction36
/ 77
![Page 78: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/78.jpg)
VISIREVirtual Image Processing System for Intelligent Reconstruction of 3D Environments.
European project IST-1999-10756. 5th Framework Programme.
Goal: semi-automatic photorealistic 3D reconstructions from videosequences.
Guillermo Gallego (GTI-UPM) 3D Reconstruction37
/ 77
![Page 79: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/79.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction38
/ 77
![Page 80: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/80.jpg)
Recall the Processing Steps
1 Digital image acquisition
2 Feature detection and extraction
3 Structure from Motion
1 Projective calibration2 Auto-calibration3 Euclidean reconstruction
4 3D triangulation and texture selection
5 VRML presentation
Guillermo Gallego (GTI-UPM) 3D Reconstruction39
/ 77
![Page 81: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/81.jpg)
Feature (corner) extraction. Image 1
Guillermo Gallego (GTI-UPM) 3D Reconstruction40
/ 77
![Page 82: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/82.jpg)
Feature (corner) extraction. Image 23
Guillermo Gallego (GTI-UPM) 3D Reconstruction41
/ 77
![Page 83: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/83.jpg)
SfM. Projective Calibration
>�> calib_tool3Recognizing 450 points in 23 images.Normalization doneGold Standard Algorithms for Projective CalibrationResidual Reprojection error = 1.301 pixels
Optimal Projective Calibration (Bundle Adjustment)Previous calibration detected Residual Reprojection error = 0.586 pixels
Mean number of points per image = 372.65Estimated noise level (sigma) = 0.61 pixelsResidual Reprojection error (projective reconstruction) = 0.586 pixelsPixel error: mean = [ -0.00355 0.00025]Pixel error: std = [ 0.71089 0.42730]
Guillermo Gallego (GTI-UPM) 3D Reconstruction42
/ 77
![Page 84: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/84.jpg)
SfM. Projective Calibration
>�> calib_tool3Recognizing 450 points in 23 images.Normalization doneGold Standard Algorithms for Projective CalibrationResidual Reprojection error = 1.301 pixels
Optimal Projective Calibration (Bundle Adjustment)Previous calibration detected Residual Reprojection error = 0.586 pixels
Mean number of points per image = 372.65Estimated noise level (sigma) = 0.61 pixelsResidual Reprojection error (projective reconstruction) = 0.586 pixelsPixel error: mean = [ -0.00355 0.00025]Pixel error: std = [ 0.71089 0.42730]
Guillermo Gallego (GTI-UPM) 3D Reconstruction42
/ 77
![Page 85: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/85.jpg)
SfM. Projective Calibration
>�> calib_tool3Recognizing 450 points in 23 images.Normalization doneGold Standard Algorithms for Projective CalibrationResidual Reprojection error = 1.301 pixels
Optimal Projective Calibration (Bundle Adjustment)Previous calibration detected Residual Reprojection error = 0.586 pixels
Mean number of points per image = 372.65Estimated noise level (sigma) = 0.61 pixelsResidual Reprojection error (projective reconstruction) = 0.586 pixelsPixel error: mean = [ -0.00355 0.00025]Pixel error: std = [ 0.71089 0.42730]
Guillermo Gallego (GTI-UPM) 3D Reconstruction42
/ 77
![Page 86: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/86.jpg)
SfM. AutocalibrationIntrinsic parameters
Algorithm: minimization of the error in the pixel shape
Guillermo Gallego (GTI-UPM) 3D Reconstruction43
/ 77
![Page 87: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/87.jpg)
SfM. Euclidean Bundle Adjustment
Euclidean Bundle AdjustmentResidual Reprojection error = 0.592 pixels
It is slightly larger than the reprojection error of the projectivereconstruction because the Euclidean reconstruction has less degreesof freedom (less parameters to decrease the error).
Guillermo Gallego (GTI-UPM) 3D Reconstruction44
/ 77
![Page 88: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/88.jpg)
3D Mesh generation and Texture selection
Guillermo Gallego (GTI-UPM) 3D Reconstruction45
/ 77
![Page 89: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/89.jpg)
VRML model: Escorial
Guillermo Gallego (GTI-UPM) 3D Reconstruction46
/ 77
![Page 90: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/90.jpg)
VRML model: Escorial (side view)
Guillermo Gallego (GTI-UPM) 3D Reconstruction47
/ 77
![Page 91: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/91.jpg)
VRML model: Escorial (top view)
Guillermo Gallego (GTI-UPM) 3D Reconstruction48
/ 77
![Page 92: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/92.jpg)
Image error summary
Guillermo Gallego (GTI-UPM) 3D Reconstruction49
/ 77
![Page 93: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/93.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction50
/ 77
![Page 94: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/94.jpg)
VRML model: Books
Guillermo Gallego (GTI-UPM) 3D Reconstruction51
/ 77
![Page 95: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/95.jpg)
VRML model: Books
Guillermo Gallego (GTI-UPM) 3D Reconstruction52
/ 77
![Page 96: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/96.jpg)
VRML model: Books
Guillermo Gallego (GTI-UPM) 3D Reconstruction53
/ 77
![Page 97: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/97.jpg)
VRML model: Books
Guillermo Gallego (GTI-UPM) 3D Reconstruction54
/ 77
![Page 98: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/98.jpg)
VRML model: Books
Guillermo Gallego (GTI-UPM) 3D Reconstruction55
/ 77
![Page 99: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/99.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction56
/ 77
![Page 100: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/100.jpg)
Structure from Motion. Photo Tourism(Snavely, Seitz, Szeliski) http://phototour.cs.washington.edu/applet/index.html
A system for browsing large collections of photographs in 3D.
1 Input: large collections of images (a)2 Process: automatically compute each photo's viewpoint and a sparse
3D model of the scene.3 Display & interactively move about the 3D space.
VERY popular because the SfM code is available online: Bundler.Guillermo Gallego (GTI-UPM) 3D Reconstruction
57/ 77
![Page 101: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/101.jpg)
Structure from Motion. Photo Tourism(Snavely, Seitz, Szeliski) http://phototour.cs.washington.edu/applet/index.html
A system for browsing large collections of photographs in 3D.
1 Input: large collections of images (a)2 Process: automatically compute each photo's viewpoint and a sparse
3D model of the scene.3 Display & interactively move about the 3D space.
VERY popular because the SfM code is available online: Bundler.Guillermo Gallego (GTI-UPM) 3D Reconstruction
57/ 77
![Page 102: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/102.jpg)
Structure from Motion. Photo Tourism(Snavely, Seitz, Szeliski) http://phototour.cs.washington.edu/applet/index.html
A system for browsing large collections of photographs in 3D.
1 Input: large collections of images (a)2 Process: automatically compute each photo's viewpoint and a sparse
3D model of the scene.3 Display & interactively move about the 3D space.
VERY popular because the SfM code is available online: Bundler.Guillermo Gallego (GTI-UPM) 3D Reconstruction
57/ 77
![Page 103: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/103.jpg)
PhotoSynth (Microsoft & U. Washington).http://photosynth.net/
Photosynth is based on Photo Tourism. It is software for:
3D modeling (location of the cameras in the scene & sparse point cloud)Generation of huge panoramas (100's Mpixels), since 2010.
More user-friendly than Bundler (research code), but less con�gurable.
Guillermo Gallego (GTI-UPM) 3D Reconstruction58
/ 77
![Page 104: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/104.jpg)
PhotoCity Game (U.Washington & Cornell)http://grail.cs.washington.edu/projects/photocity/
PhotoCity is a game for reconstructing the world in 3D out of photos.
Players take pictures of building exteriors from all di�erent angles, which arethen used to automatically generate 3D models and calculate virtualownership of the buildings.
Our ultimate goal is to reconstruct the entire world, one photo at a time.
Guillermo Gallego (GTI-UPM) 3D Reconstruction59
/ 77
![Page 105: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/105.jpg)
City-scale SfM
Building Rome in a day (Agarwal et al, ICCV 2009)
∼500 cores, ∼200.000 images, 1 day of processing(from image matching to large scale optimization).Experiment in 3 cities: Rome, Venice, Dubrovnik.http://www.youtube.com/watch?v=kxtQqYLRaSQ
Guillermo Gallego (GTI-UPM) 3D Reconstruction60
/ 77
![Page 106: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/106.jpg)
City-scale SfM
Building Rome in a day (Agarwal et al, ICCV 2009)
∼500 cores, ∼200.000 images, 1 day of processing(from image matching to large scale optimization).Experiment in 3 cities: Rome, Venice, Dubrovnik.http://www.youtube.com/watch?v=kxtQqYLRaSQ
Guillermo Gallego (GTI-UPM) 3D Reconstruction61
/ 77
![Page 107: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/107.jpg)
City-scale SfM
4D Cities (Georgia Tech, 2007)
Browse large collection of photos in space and time. Show history of the city.Space-time reconstruction (based on Bundler) and navigation.Model of Atlanta: http://4d-cities.cc.gatech.edu/atlanta/
Guillermo Gallego (GTI-UPM) 3D Reconstruction62
/ 77
![Page 108: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/108.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction63
/ 77
![Page 109: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/109.jpg)
Multi-View Stereo: PMVS & CMVS(Furukawa and Ponce).
Patch-based Multi-View Stereo (PMVS). CVPR 2007.
Computes a dense reconstruction of the scene using patches (vs points).
Cameras are calibrated. The only unknown is the structure of the scene.
Again, very popular because the code is available online.
http://grail.cs.washington.edu/software/pmvs/gallery.html
Clustering Views for Multi-view Stereo (CMVS). CVPR 2010.
Management of a very large data set (divide-and-conquer):
Split the dataset into clusters of images (using graph partitioningmethods) to compute partial 3D reconstructions.Merge the 3D reconstructions.
http://www.youtube.com/watch?v=ofHFOr2nRxU
Guillermo Gallego (GTI-UPM) 3D Reconstruction64
/ 77
![Page 110: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/110.jpg)
Multi-View Stereo: PMVS & CMVS(Furukawa and Ponce).
Patch-based Multi-View Stereo (PMVS). CVPR 2007.
Computes a dense reconstruction of the scene using patches (vs points).
Cameras are calibrated. The only unknown is the structure of the scene.
Again, very popular because the code is available online.
http://grail.cs.washington.edu/software/pmvs/gallery.html
Clustering Views for Multi-view Stereo (CMVS). CVPR 2010.
Management of a very large data set (divide-and-conquer):
Split the dataset into clusters of images (using graph partitioningmethods) to compute partial 3D reconstructions.Merge the 3D reconstructions.
http://www.youtube.com/watch?v=ofHFOr2nRxU
Guillermo Gallego (GTI-UPM) 3D Reconstruction64
/ 77
![Page 111: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/111.jpg)
Sparse vs. Dense reconstructionSparse: output of Structure from Motion (SfM: Bundler)
Guillermo Gallego (GTI-UPM) 3D Reconstruction65
/ 77
![Page 112: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/112.jpg)
Sparse vs. Dense reconstructionDense: SfM −→PMVS (output)
Guillermo Gallego (GTI-UPM) 3D Reconstruction66
/ 77
![Page 113: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/113.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction67
/ 77
![Page 114: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/114.jpg)
Books (SLCV)Using PMVS for visualizing the result (dense reconstruction) of autocalibration methods.
Guillermo Gallego (GTI-UPM) 3D Reconstruction68
/ 77
![Page 115: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/115.jpg)
Books (SLCV)Using PMVS for visualizing the result (dense reconstruction) of autocalibration methods.
Guillermo Gallego (GTI-UPM) 3D Reconstruction69
/ 77
![Page 116: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/116.jpg)
Plaza De La Villa (Madrid)
Guillermo Gallego (GTI-UPM) 3D Reconstruction70
/ 77
![Page 117: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/117.jpg)
Plaza De La Villa (Madrid)
Guillermo Gallego (GTI-UPM) 3D Reconstruction71
/ 77
![Page 118: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/118.jpg)
Outline1 Introduction
MotivationClassi�cation of 3D reconstruction methods
2 Image based 3D reconstructionProcessing stepsSfM. Problem StatementProjective CalibrationSfM. Camera resectioningSfM. Initial camera pair estimation
3 ExamplesVISIRE project (2000)ADREP 3D project (2003). San Lorenzo de El Escorial MonasteryADREP 3D project (2003). Books scenePhoto Tourism (2006), PhotoSynth (2008), etc.Multi-View Stereo: PMVS (2007) & CMVS (2010)Autocalibration. Six-Line Conic Variety (SLCV)Video Sur�ng project. Telefonica R&D (2009)
Guillermo Gallego (GTI-UPM) 3D Reconstruction72
/ 77
![Page 119: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/119.jpg)
Video Sur�ng. http://vimeo.com/159901903 modes of interaction: video player.
Guillermo Gallego (GTI-UPM) 3D Reconstruction73
/ 77
![Page 120: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/120.jpg)
Video Sur�ng. http://vimeo.com/159901903 modes of interaction: extended video player (behind the camera).
Guillermo Gallego (GTI-UPM) 3D Reconstruction74
/ 77
![Page 121: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/121.jpg)
Video Sur�ng. http://vimeo.com/159901903 modes of interaction: free-�ight navigation in the 3D scene.
Guillermo Gallego (GTI-UPM) 3D Reconstruction75
/ 77
![Page 122: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/122.jpg)
Summary
1 Advances in technology arise from connections between geometry,algebra, optimization, statistics, etc.
2 Improve your habilities / toolbox in the above areas.
3 Overview of recent results in (image based) multi-view stereo (3D)reconstruction.
Guillermo Gallego (GTI-UPM) 3D Reconstruction76
/ 77
![Page 123: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/123.jpg)
Summary
1 Advances in technology arise from connections between geometry,algebra, optimization, statistics, etc.
2 Improve your habilities / toolbox in the above areas.
3 Overview of recent results in (image based) multi-view stereo (3D)reconstruction.
Guillermo Gallego (GTI-UPM) 3D Reconstruction76
/ 77
![Page 124: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/124.jpg)
Summary
1 Advances in technology arise from connections between geometry,algebra, optimization, statistics, etc.
2 Improve your habilities / toolbox in the above areas.
3 Overview of recent results in (image based) multi-view stereo (3D)reconstruction.
Guillermo Gallego (GTI-UPM) 3D Reconstruction76
/ 77
![Page 125: Aplicaciones de los métodos de reconstrucción estéreo multivistaoa.upm.es/15340/1/Seminario_UAM_ggb.pdf · 2014-09-22 · VISIRE project (2000) ADREP 3D project (2003). San Lorenzo](https://reader033.fdocuments.in/reader033/viewer/2022042216/5ebdb38d5bef84786b6aafab/html5/thumbnails/125.jpg)
References I
Richard Hartley and Andrew Zissermann.Multiple View Geometry in Computer Vision (2nd Ed.)
Cambridge University Press, March 2004.
Guillermo Gallego.Sistema de Autocalibración de Cámaras y Reconstrucción 3D.Proyecto Fin de Carrera, ETSI Telecomunicación, February 2004.Tutor: José Ignacio Ronda Prieto.
Guillermo Gallego (GTI-UPM) 3D Reconstruction77
/ 77