New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation
description
Transcript of New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation
![Page 1: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/1.jpg)
New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation
Jin ZhouJune 16th, 2009
Dissertation Defense
![Page 2: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/2.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 3: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/3.jpg)
The Geometry of 3D to 2D
Images are 2D projections of the 3D world
![Page 4: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/4.jpg)
3D Vision – The Problem
?
How do we extract 3D information from 2D images?
? ? ??
3D of the objects
3D of the cameras
![Page 5: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/5.jpg)
3D Vision – Applications
Augmented Reality Iron Man, Lord of Rings, Kingkong
Scene ModelingGoogle SketchUp, Autodesk ImageModeler
Virtual Touring Microsoft PhotoSynth
3D Visualization 3D Gaming, 3D Movies, 3D Medical Images
Robots
![Page 6: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/6.jpg)
3D Vision – A Human Perspective
Size Linear Perspective Object Connections Stereo Motion Occlusion Shading
![Page 7: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/7.jpg)
3D Vision – Computational Approaches
Different approaches use different cues Different problems require different approaches. Structure from Motion (SfM)
Rely on point correspondences Single View Based Modeling (SVBM)
Rely on knowledge of the scene Camera calibration
All approaches require the images are calibarated first (either manual or automatic)
![Page 8: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/8.jpg)
3D Vision – Practical Challenges
Camera information is often unavailable Point correspondences are not reliable
and computationally costly to compute Image resolution is limited in many
applications Degeneracy
![Page 9: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/9.jpg)
3D Vision – Limitations of Current Approaches
Distortion Degeneracy
Due to high degree of freedom of geometric models
Lack of geometric meaningMost approaches are purely based on
algebraic derivations or imaginary objects. Not accurate or not convenient (SVBM).
![Page 10: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/10.jpg)
3D Vision – Our Contributions
Novel image rectification schemes are proposed based on sequential virtual rotation
Novel approaches are proposed for the following problems Image rectification for stereoscopic visualization Camera calibration Stereoscopic view synthesis from monocular
endoscopic sequences Rapid cones and cylinders modelingMonocular-vision-guided mobile robot navigation
![Page 11: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/11.jpg)
3D Vision – Results of Our Approaches
No affine/projective distortion Can handle degeneracy Intuitive geometric meanings
Lead to useful insights for some practical problems
Accurate and fast
![Page 12: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/12.jpg)
Publications & Manuscripts Journals
Jin Zhou and Baoxin Li, "Image Rectification for Stereoscopic Visualization", Journal of the Optical Society of America A (JOSA), Vol. 25, No. 11, pp. 2721-2733, 2008.
Wenfeng Li, Jin Zhou, Baoxin Li, and M. Ibrahim Sezan, " Virtual View Specification and Synthesis for Free Viewpoint Television", IEEE Transactions on Circuit and Systems for Video Technologies , Vol, 19, No. 4, pp.533-546, 2009.
Jin Zhou and Baoxin Li, "Rapid Cones and Cylinders Modeling from a Single Calibrated Image Using Minimal 2D Control Points", submitted to Machine Vision and Applications (under revision)
Jin Zhou and Baoxin Li, “Stereoscopic View Synthesis from Monocular Endoscopic Image Sequences“,(manuscript under internal revision; to be submitted to IEEE Transactions on Medical Imaging by the end of June 2009)
Jin Zhou and Baoxin Li, “Image Rectification Based on Infinity Homography”, (manuscript under internal revision; to be submitted to IEEE Transactions on Image Processing by the end of June 2009)
Refereed Conference papers Jin Zhou, Ananya Das, Feng Li, Baoxin Li, "Circular Generalized Cylinder Fitting for 3D
Reconstruction in Endoscopic Imaging Based on MRF", In 9th IEEE Computer Society Workshop on Mathematical Methods in Biomedical Image Analysis (Joint with CVPR 2008).
Jin Zhou and Baoxin Li, "A Four Point Algorithm for Fast Metric Cone Reconstruction from a Calibrated Image", In 4th International Symposium on Visual Computing (ISVC), 2008.
Xiaolong Zhang, Jin Zhou and Baoxin Li, "Robust Two-view External Calibration by Combining Lines and Scale Invariant Point features", In 4th International Symposium on Visual Computing (ISVC) 2008.
![Page 13: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/13.jpg)
Publications (Cont’d) Jin Zhou and Baoxin Li, “Exploiting Vertical Lines in Vision-Based Navigation for
Mobile Robot Platforms”, International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2007.
Xiaokun Li, Roger Xu, Jin Zhou, Baoxin Li, "Creating Stereoscopic (3D) Video from a 2D Monocular Video Stream", In 3rd International Symposium on Visual Computing (ISVC), 2007.
Wenfeng Li, Jin Zhou, Baoxin Li, M. Ibrahim Sezan, "Virtual View Specification and Synthesis in Free Viewpoint Television Application", 3D Data Processing, Visualization and Transmission (3DPVT), 2006.
Jin Zhou and Baoxin Li, "Image Rectification for Stereoscopic Visualization without 3D Glasses", ACM International Conference on Image and Video Retrieval (CIVR), 2006.
Jin Zhou and Baoxin Li, “Homography-based Ground Detection for a Mobile Robot Platform using a Single Camera”, International Conference on Robotics and Automation (ICRA), 2006.
Jin Zhou and Baoxin Li, “Robust Ground Plane Detection with Normalized Homography in Monocular Sequences from a Robot Platform”, International Conference on Image Processing (ICIP), 2006.
Jin Zhou and Baoxin Li, “Rectification with Intersecting Optical Axes for Stereoscopic Visualization”, International Conference on Pattern Recognition (ICPR), 2006.
![Page 14: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/14.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 15: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/15.jpg)
What is Image Rectification?
Image rectification is a process to transform the original images to new images which have desired properties.
![Page 16: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/16.jpg)
General Image Transformations
H?
Image transformation can be defined by a 3x3 matrix H, which is called Homography.
xx' H
![Page 17: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/17.jpg)
Image Rectification based on Virtual Rotation
Homography of camera rotation/zooming
If we normalize the coordinates
Assume R = I
[ | ], ' ' '[ | ]P KR I C P K R I C
1 1x ' ' ' '[ | ] ' '( ) [ | ] ( ' ')( ) xP X K R I C X K R KR KR I C X K R KR
1( ' ')( )H K R KR
1x xK 1x' ' x'K
ˆ 'H R Camera orientation is determined at the same time
![Page 18: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/18.jpg)
Advantages of the New Rectification Schemes Intuitive geometric meaning Robust
Rotation parameters can be computed by various basic image features, such as points, lines and circles.
Can be used for camera calibration. Can be used for 3D information extraction. Lead to non-distorted results
Reason: Rotation do not introduce affine/projective distortion
![Page 19: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/19.jpg)
Rotations are Decomposed as Euler Angles
1 0 0
( ) 0 cos( ) sin( )
0 sin( ) cos( )xR
cos( ) 0 sin( )
( ) 0 1 0
sin( ) 0 cos( )yR
cos( ) sin( ) 0
( ) sin( ) cos( ) 0
0 0 1zR
pitch
yaw
roll
![Page 20: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/20.jpg)
Rotation Parameters Can Be Estimated by Basic Image Features
Each rotation has only one degree of freedom and thus only needs one constraint.
Example: transforming a point on to y axis
ˆ (0, , )TzR p d c
cos( ) sin( ) 0a b
tan( , )arc a b
/ 2
[ ] / 2
otherwize
[ tan( , )]arc a b Ambiguity!
Normalize
![Page 21: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/21.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 22: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/22.jpg)
Image Rectification for Stereoscopic (3D) Visualization
The Principle of Stereoscopic (3D) Visualization
Nielsen Consumer Insight 2009 Industry Outlook: Cinema admissions will increase fueled by growing availability of 3D movies and theatres.
Deloitte: 3D becomes an obligation, not an option, at the movies
![Page 23: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/23.jpg)
The Problem
Given two arbitrary images, rectify them so that the results look like a stereo pair.
stereo camera setupgeneral camera setup
![Page 24: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/24.jpg)
An Example
![Page 25: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/25.jpg)
Motivation
Stereo content is scarce Stereo cameras/camcorders are expensive Common users seldom use stereo
cameras/camcorders Price of a current 3D Display is affordable We want to generate stereo content from
images/videos taken by common cameras
![Page 26: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/26.jpg)
Our Approach – Rectification based on Virtual Rotation
We can “rotate” cameras to standard stereo setup.
1( )i i iH KR K R
1H
2H
![Page 27: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/27.jpg)
Calibrated Case
( , , )TR r s t
1 2 1 2( ) / || ||r C C C C
( ) / || ||t r p r p
s r t
1( )i i iH KR K R
Known
UnKnown
Constraints of the stereo camera pair:1. The two cameras have the same intrinsic parameters (K)
and orientation (R)2. The camera’s optical axis is perpendicular to the baseline
(C1 – C2) i.e. the camera’s x axis has the same direction with the
baseline
K1 Any vector
![Page 28: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/28.jpg)
Uncalibrated Case
For the uncalibrated case, all K, R and C are unknown. We can only start from the fundamental matrix and point correspondences.
Estimate H2 (homography for the second image)
2
0 / 2
0 with / 2
0 0 1 ( ) / 2
x x
y y
f p p w
K K f p p h
f w h
2R I
1( )i i iH KR K R ?
![Page 29: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/29.jpg)
Determine R based on Sequential Virtual Rotation
2ˆ (1,0,0)TRe 12 2 2e K eConstraints:
First rotate around z axis so that the point is transformed to x axis (i.e. y = 0)
y zR R R
( , , ) ( ,0, )T TzR a b c d c
( ,0, ) ( ,0,0) (1,0,0)T T TyR d c e
[arctan( , )]z b a
[arctan( , )]y c d
Rotate around y axis so that the point is transformed to infinity.
In stereo pair, the epipole is at horizontal infinity
![Page 30: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/30.jpg)
Estimate H1
1 11 1 1 2 2 2 1 1 2( ) ( )H KR K R H K R K R H M
2 0( )TM I e v M 2[ ]F e M
1 2
2 2 0
2 0 2 0
12 0 2 2 2 2 0
12 2 1 2 0 1 2
( )
( )
( ) ( = )
T
T
T
T T T
H H M
H I e v M
H M e v M
H M H e v H H M
I H e v H M v v H
12 2 2 2 [ , , ]
(1,0,0) ( ,0,0)
T
T T
H e KRK e K r s t r
K k
1 2 2 2 0 2 0( )
0 1 0
0 0 1
TA
A
H I H e v H M H H M
a b c
H
0 2 2[ ] TM e F e v
![Page 31: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/31.jpg)
Estimate H1
Determine a, b and c Property of standard stereo setup:
For two points with the same depth, their projection on different images should have the same distance (Points with the same depth should have the same disparity).
ApproachGroup points by similar disparities Then compute a, b by minimizing
2
1 1 2 2,
(|| x x || | x ' x ' ||)p
i j i jp i j A
H H H H
2
,
ˆ ˆ ˆ ˆ ˆ ˆ( ( ) ( ) ( ' '))p
i i i j i jp i j A
a x x b y y x x
0 0 0ˆ ˆ ˆ 'ax by c x
![Page 32: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/32.jpg)
Results
Original Pair Hartley’s Method Our Method
Shear distortion
![Page 33: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/33.jpg)
Results
Original Pair Hartley’s Method Our Method
Shrink horizontally
![Page 34: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/34.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 35: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/35.jpg)
Camera Auto-Calibration from the Fundamental Matrix
Ti i iw K K
2 2 2 1[ ] [ ] Te w e Fw F Kruppa Equations
dual image of the absolute conic
Huang-Faugeras constraints
2 2
2 1
1( ) ( ) 0
2T T
T
trace EE trace EE
E K FK
Cons: Complex and hard to understand!Derivation for degenerate cases are purely algebraic.
Traditional Approaches:
![Page 36: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/36.jpg)
Our Approach
12 2 2 1 1 1
1 1 1 2 2 2( , , , , , , )
T T T Sz y x y z
z y x z y
F K R R F R R R K
f f
We transform the original pair to a standard stereo pair through sequential virtual rotation and zooming
0 0 0
0 0 1
0 1 0
SF
F
7 DOFs
7 Parameters
![Page 37: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/37.jpg)
Decomposition Illustration
![Page 38: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/38.jpg)
Solving The Equation1
2 2 2 1 1 1
1 1 1 2 2 2( , , , , , , )
T T T Sz y x y z
z y x z y
F K R R F R R R K
f f
1 12 1 2 2 1 1
T T Sz z z y x yF R FR K R F R R K
2 1
2 1
0 0 0 0
0 1 0 0 1 0
0 0 0 0z
c a b a c
F c d c
d a b a d
1 21 2
2
2 1 1 2
1 1 2
sin( ), cos( ),cos( )
cos( ), sin( )cos( ) cos( ) cos( )
x xy
x xy y y
f da f f c
f d d db d
1 1 1 1( ) ( ,0, )Tz zR e d c
2
2 22 21 2
1 22 21 2
( )
,
x
adtg
bc
acd abdf f
bd acc cd abc
2 2 22 1 1 1
1 2 21 1
2 2 22 2 2 2
2 2 22 2
( )
( )
y
y
f c acctg
d bd acc
f c abctg
d cd abc
1 1 1 1( , , )e a b c1 1 1arctan( / )z b a
1 0Fe
1 1 1 1( ) /ytg f c d
![Page 39: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/39.jpg)
Degeneracy Analysis
![Page 40: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/40.jpg)
Degeneracy Analysis
![Page 41: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/41.jpg)
Results of Monte Carlo Simulation
![Page 42: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/42.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 43: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/43.jpg)
Stereoscopic View Synthesis From Monocular Endoscopic Videos
3D imaging helps to enable faster and safer surgical operations
Two-view image rectification cannot be applied to the new problem
![Page 44: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/44.jpg)
The Framework
We proved:1. Affine 3D reconstruction is
sufficient.2. Linear interpolation in
normalized disparity field is equal to linear interpolation in 3D space.
Challenges: 1. Image quality is poor 2. Degeneracy
![Page 45: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/45.jpg)
Strategy for Solving Degeneracy
We assume the initial two frames have same orientation (i.e. they are rectified)
The assumption makes the DOF of the fundamental matrix from 8 to 2!
No Assumption
Assume the two frame are rectified
Degeneracy!
![Page 46: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/46.jpg)
Interpolation
a) Shows the disparities based on the SfM results
b) We do Delaunay triangulation and interpolate each triangle
c) We pick a set of grid points from b) and do bilinear interpolation
d) We fill holes using Laplacian interpolation and do smoothing.
![Page 47: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/47.jpg)
Results of Synthetic Data
Ground truth
Stereo images
Final disparity image
Disparity image after triangulation
![Page 48: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/48.jpg)
Results of Real Data
gray
fryover
creel
![Page 49: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/49.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 50: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/50.jpg)
Rapid Cylinders and Cones Modeling from A Single Image
![Page 51: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/51.jpg)
Overview of Our Approach
Goal: Rapid + Accurate Camera Calibration (Orientation Estimation)
Vertical lines Vanishing line of horizontal plane A Cone
Modeling from Image Cones (two points / four points) Cylinders (two points / four points)
![Page 52: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/52.jpg)
The Coordinate Systems
(X,Y,Z,O) -- World Coordinate SystemY is perpendicular to the ground
(x,y,z,O) -- Camera Coordinate SystemCamera center is at the origin
Observation: Most objects stand on the ground
![Page 53: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/53.jpg)
Orientation Estimation from Vertical Lines
(0,1,0)T( , , )Ta b c R
First rotate around z axis so that the vanishing point is transformed to y axis
Rotate around x axis so that the vanishing point is transformed to infinity.
![Page 54: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/54.jpg)
Orientation Estimation from a Cone
Rx
Rx (π/2)
RzRxRz
Edges are symmetric to y axisR Cross section is a circle
![Page 55: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/55.jpg)
Illustration
![Page 56: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/56.jpg)
Metric Rectification of the Ground Plane1 0 0
( / 2) 0 0 1
0 1 0g xH R
Original
Original Rectified
Rectified
![Page 57: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/57.jpg)
Modeling Cones
0Μ ( (0, ,1) , , )Tc vc y r y
Standard view Cones on Ground General Cones
0Μ ( , ,M )R d
Cone Parameters:
2D Control Points for Cones Modeling:
![Page 58: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/58.jpg)
Cones Modeling from the Standard View
Rectify the standard view to the ground plane view by Rx(π/2) The cross section is rectified to a circle The edge line is tangent to the circle The center and radius can be determined for any
point on the edge line
![Page 59: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/59.jpg)
Modeling Cones Standing on the Ground
Ry
Four points Standard view Ground view 3D Mesh
![Page 60: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/60.jpg)
Modeling General Cones
R
Five points Standard View 3D Mesh
![Page 61: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/61.jpg)
Modeling Cylinders
0L ( (0, ,1) , , )Tb tc y r y
0L ( , ,L )R d
Standard View Cylinders on Ground General Cylinders
Cylinder Parameters:
2D Control Points for Cones Modeling:
![Page 62: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/62.jpg)
Modeling Cylinders Standing on the Ground
Ry
Two points Standard view Ground view 3D Mesh
![Page 63: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/63.jpg)
Modeling General Cylinders
R
Four points Standard View 3D Mesh
![Page 64: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/64.jpg)
Screenshots of Real Data Experiments
demo
![Page 65: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/65.jpg)
Outline
Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular
Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single
Images Robot Vision Conclusions and Future Work
![Page 66: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/66.jpg)
Exploiting Vertical Lines for Monocular Vision based Mobile Robot Navigation
In man-made environments, vertical lines are omni-present: buildings, boxes, bookshelves, cubicle walls, door frames
Many vision based systems assume the image plane is perpendicular to the ground plane
We proposed methods to rectify an image plane with general pose to be vertical based on vertical lines.
![Page 67: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/67.jpg)
Rectified Images for Ground Plane Detection
ˆ ( / )TH R I Cn d
0 (0,1,0)Tn 0 0( ,0, )TC x z
0
0
cos( ) / sin( )ˆ 0 1 0
sin( ) / cos( )
x d
H
z d
The normalized homography of the ground plane has a special form in the rectified images:
3 DOFs
![Page 68: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/68.jpg)
Results
Special Form
General Form
3 DOFs
8 DOFs
![Page 69: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/69.jpg)
Ground Rectification for Mosaic based SLAM
After rectification, the relationship between any two views directly indicates their relative locations and orientations.
From vertical image plane, it’s easy to get the ground plane image
![Page 70: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/70.jpg)
Results
![Page 71: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/71.jpg)
Vision Based Control
Vanishing line
Occupied
OccupiedTurn angle
876
5…
910
11…
Nearest left Nearest right
Given an rectified imageFind the object to trackIdentify the obstacle or ground planeOutput a turn angle.Adjust the camera to make object always stays at the center of the image.
Movie 1Movie 2ImagesObstacles
Rectified Image
![Page 72: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/72.jpg)
Other Potential Applications
Surveillance/Activity Recognition/ Path Planning
Different object’s size
Same object’s sizeGround is rectifiedRight angles are recoveredSpeed is reflected on the image
![Page 73: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/73.jpg)
Conclusions and Future Works
Novel image rectification schemes are proposed in the context of exploring several practical 3D vision problems
For each of the problem, we designed novel algorithms and nice results are achieved. Moreover, we gain new insights to the problems.
In the future, we should combine different approaches and exploiting more visual cues for 3D vision problems.
![Page 74: New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation](https://reader035.fdocuments.in/reader035/viewer/2022062315/56815290550346895dc0b2d7/html5/thumbnails/74.jpg)
Questions?
Thanks!