3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and...
-
Upload
sophia-shakespeare -
Category
Documents
-
view
220 -
download
0
Transcript of 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and...
![Page 1: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/1.jpg)
3D Priors for Scene Learning3D Priors for Scene Learningfrom a Single Viewfrom a Single View
Diego Rother, Kedar Patwardhan, Iman Aganj Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiroand Guillermo Sapiro
University of MinnesotaUniversity of Minnesota
1
Search in 3D Workshop (CVPR 2008)Search in 3D Workshop (CVPR 2008)
![Page 2: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/2.jpg)
AutoCalibration AlgorithmsCamera
Calibration
Moving Camera?
Tracking Local Features
Boujou, 3D-Equalizer, Matchmover, Voodoo,
…
yes no
Known Structure?
[1] D. Liebowitz and A. Zisserman, “Metric Rectification for Perspective Images of Planes.” CVPR, 1998.
[2] A. Criminisi, I. Reid and A. Zisserman, “Single View Metrology.” IJCV, 1999.
[1][1] [2][2]
Exploit Known Structure
yesExploit Known
Objects
no
Common in Surveillance
![Page 3: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/3.jpg)
Main Idea 1
• Correct Camera Matrix → Pedestrian observations are consistent (no height change).
3
![Page 4: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/4.jpg)
Main Idea 1
• Incorrect Camera Matrix → Pedestrian grows or shrinks.
• Pedestrians can be used as a measuring stick to calibrate the camera.
4
![Page 5: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/5.jpg)
Main Idea 2
Image
Plane
Camera Center
3D WorldP1
Camera Matrix (PF):
PF= P1
Light Source
P1
P2
Shadow Camera Matrix (PS):
PS= P1 o P2
5
![Page 6: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/6.jpg)
Main Idea 2
• Correct light source position → Pedestrian shadow observations are consistent .
6
• Analogously, a reflection camera can be defined.
![Page 7: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/7.jpg)
In summary
Simultaneously Estimate: 1- Ground Positions (in 3D) 2- Horizon height (in 2D) 3- Light source position (in 3D) 4- Pedestrian height (in world units) 5- Axes scaling (to define the unit of length)
X
Z
Y
7That are Mutually Consistent and Explain the observations.
![Page 8: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/8.jpg)
Object 3DBounding Box
Single Frame Consistency?
Camera
ConsistencyTest
Height Ground PositionObservation
Consistency(Likelihood)
Camera matrix(or Shadow Camera)
Model(3D prior)
8
![Page 9: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/9.jpg)
3D Priors
Voxel
V4 V6
V1 V2 V3
V7 V8 V9
V5Pixel
Camera Q2Q3
Q1
Q4
Voxel Vi:- Occupied (vi = 1) with probability pi.- Blocks light if it is occupied.- Independent of other voxels.
Problems:- Discretization matters.- Equal contributions voxels ray.
Solution: Beer-Lambert law correction (predicts light attenuation in solutions),
R1,1R2,1
R6,1R3,1
- measured in [blocking probability / meter].- Same to traverse 1 big voxel or 2 of half the size.
9
2D Prior 3D Prior
![Page 10: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/10.jpg)
3D Priors
10
Whole walking cycle Part of the walking cycle
![Page 11: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/11.jpg)
Graphical Model
Observed PixelColors in Frame tC1 CM
Voxels(3D Prior)PV1 PV2 PVN
Pixel Class(2D Prior)
ForegroundQF1 QFM QS1 QSMShadow
Geometry(Projection)
CameraMatrixLight PositionGround Position
BackgroundShadowColor
Models
11
F1
Likelih
ood
![Page 12: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/12.jpg)
Trajectory unregularized
Scene ParametersF1
Likelih
oodG1
12
F1
Likelih
oodG2
F1
Likelih
oodG3
![Page 13: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/13.jpg)
Trajectory Regularized
F1
Likelihood(F2)
G1
F1
G2
F1
G3Prior
Acceleration
Optimum trajectory and F2 computed in O(NF . NG3) using Dynamic Programming.
13
F2
![Page 14: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/14.jpg)
Search Solution Space
• Search the solution in the whole 4D parameter space:1. Horizon Height2. Y-Axis Scale.3. Light Theta4. Light Phi
14
Likelihood
CameraMatrixLight Position
F2
Optimum trajectory
Camera Matrix
Light Direction
![Page 15: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/15.jpg)
Results
• To speed up computation, search first in the lowest resolution.
15
Half Resolution
Original Resolution
• Then, refine in the next higher, and so on.• Fast, so the whole space can be searched.
![Page 16: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/16.jpg)
Results
16
Half Resolution
Original Resolution
• Solution superimposed.• Shape of the peak defines the types of errors.
Estimated Horizon
![Page 17: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/17.jpg)
ResultsMetrology comparison
MeasureGround Truth
(m)
Estimated (m)
P1 4.18 4.27
P2 4.26 4.25
P3 4.38 4.36
P4 4.13 4.29
Localization errorNo Shadows
(cm)Shadows
(cm)32.3 21.5
• Mean error lower than 2% (relative to the people average height).
17
• Shadows are not disturbances, their use improve localization.
Estimated Horizon
![Page 18: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/18.jpg)
ConclusionsPresented:
• Novel object model (not limited to people) and probabilistic framework • For camera calibration and simple lighting estimation.• Using the Foreground and the Shadows.• That works in situations where other methods fail.
18
![Page 19: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/19.jpg)
Learning 3D Priors
V4 V6
V1 V2 V3
V7 V8 V9
V5
C1 C2
Method of Moments, yields one Equation per ray:
This is the Fan Beam Radon transform.Just solve linear system.
Silhouette in frame t Average
19
![Page 20: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/20.jpg)
3D Priors
20
![Page 21: 3D Priors for Scene Learning from a Single View Diego Rother, Kedar Patwardhan, Iman Aganj and Guillermo Sapiro University of Minnesota 1 Search in 3D.](https://reader035.fdocuments.in/reader035/viewer/2022062304/56649c425503460f948f059c/html5/thumbnails/21.jpg)
Search Solution Space
x
y
z
21