senior poster
1
Image Sequences Interpolation based on Dualhomography Transformation Student:EricHuang Supervisor:Stergios Roumeliotis Background Results Conclusion Introduction: In many robotics applications, such as search, rescue or exploration, images are collected by a camera at high frame rate, e.g.,30Hz. Due to processing constraints, however, they are stored at a lower rate, e.g.,6Hz. Thus, such discontinuous image sequences are not informative enough for human to understand the explored scene. To address this problem, this project develops a program using the geometry information to stitch adjacent images and interpolate them into original image sequences. Inthis way, the generated new images sequences are able to run at a higher frequency and provide natural, smooth visual perceptions to users. Conclusions: 1. Based on thetest dataset, the new generated image sequences are able torun at original and 2x frequency and at the same time present a smooth perspective to the viewer. 2. In this project, a robust method is developed to compute the binary mask for Laplacian blending procedure, which has able to handle any overlap cases between two stitched images. 3. Estimate the subtransformation between interpolated frames using square root of homography matrix has been proved to be very accurate and has able to generate smooth transformations. 4. The program has able to check the interpolation result of each frame by computing the similarity between current frame and previous frame. Also, restart interpolation on current frame when illestimation oc c urs. 5. The program is relatively robust and successfully interpolate over 93% images in test dataset. Some sequences are dropped due to insufficient feature points, which can happen when image is too dark or too blurry. 6. The program is currently implemented based on CPU and the runtime is tolerable. About 30 sec per frame on anIntel i7, 8 GB Desktop computer. Future direction: • Implement the algorithm based on GPU, which can significantly accelerate the running speed. • Given sparse points cloud of the scene and camera rotation, translation matrices, explore the way to interpolate views from any viewpoint in the scene. Namely, treat the real 3D world as a black box and use only local homography transformations to represent the 2D relationship between pixels in different frames. • Solve parallel effects in the interpolation procedure. Due to the translation of camerasand the naive assumption that homograph transformation ignores such depth difference, parallel effect significantly affectsinterpolation results when depth varies dramatically. Reference [1] Gao, J., Kim, S. J.,& Brown, M. S. (2011, June). Constructing image panoramas using dualhomography warping. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 4956). IEEE. [2] Xiong, Y.,& Turkowski, K. (1998, October). Registration, calibration and blending in creating high quality panoramas. In Applications of Computer Vision, 1998. WACV'98. Proceedings., Fourth IEEE Workshop on (pp. 6974). IEEE. [3] Forsyth, D. A., & Ponce, J. (2003). A Modern Approach. Computer Vision: A Modern Approach. [4] Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge university press. [5] Chen, S. E., & Williams, L. (1993, September). View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques (pp. 279288). ACM. [6] Higham, N. J. (1987). Computing real square roots of a realmatrix. Linear Algebra and its applications, 88, 405430. Electric & Computer Engineering College of Science & Engineering Approach Dualhomography Transformation Interpolation New image sequences that can run at original or even higher frequency Original discontinuous image sequences Stitching Homography Matrix related two images For eachpair of consecutive images, the views are not strictly related by a homography transformation due to the translation between their viewpoints. The 3D structure of scene affectsthis transformation. However, we can approximate this by using several different homography transformations in spatial layout of images. Real Sc hur dec omposition In order to Interpolate the the transformation, we decompose homography matrix by computing its real square root. f(A) function computes its square root, where Q is real orthogonal matrix.And T inherits R’s upper quasitriangular structure. Kmeans In order to divide the image in special layout, we apply 3demissional kmeanson the scaled 3d coordinates [x’, y’ , z’] for eachfeature point extracted from image pairs. The relative depth Z of each feature point is computed using optical flow. Then, for each cluster of feature points, compute local homography matrix using RANSAC and leastsquare method. Algorithm: Kmeans cluster result Cluster feature points based on kmeans with 2 bins Dualhomgraphy transformation result Stitched images without blending L aplac ian Blending result Blend images and eliminate seamline
-
Upload
yujia-huang -
Category
Documents
-
view
74 -
download
1
Transcript of senior poster
Image Sequences Interpolation based on Dual-‐homographyTransformation
Student:EricHuangSupervisor:Stergios Roumeliotis
Background Results ConclusionIntroduction:In many robotics applications, such as search, rescue or exploration, images are collected by a camera at high frame rate, e.g., 30Hz. Due to processing constraints, however, they are stored at a lower rate, e.g., 6Hz. Thus, such discontinuous image sequences are not informative enough for human to understand the explored scene.
To address this problem, this project develops a program using the geometry information to stitch adjacent images and interpolate them into original image sequences. In this way, the generated new images sequences are able to run at a higher frequency and provide natural, smooth visual perceptions to users.
Conclusions:1. Based on the test dataset, the new generated image sequences are able to run at
original and 2x frequency and at the same time present a smooth perspective tothe viewer.
2. In this project, a robust method is developed to compute the binary mask forLaplacian blending procedure, which has able to handle any overlap casesbetween two stitched images.
3. Estimate the sub-‐transformation between interpolated frames using square rootof homography matrix has been proved to be very accurate and has able togenerate smooth transformations.
4. The program has able to check the interpolation result of each frame bycomputing the similarity between current frame and previous frame. Also, restartinterpolation on current frame when ill-‐estimation occurs.
5. The program is relatively robust and successfully interpolate over 93% images intest dataset. Some sequences are dropped due to insufficient feature points,which can happen when image is too dark or too blurry.
6. The program is currently implemented based on CPU and the runtime is tolerable.About 30 sec per frame on an Intel i7, 8 GB Desktop computer.
Future direction:• Implement the algorithm based on GPU, which can significantly accelerate therunning speed.•Given sparse points cloud of the scene and camera rotation, translation matrices,explore the way to interpolate views from any viewpoint in the scene. Namely, treatthe real 3D world as a black box anduse only local homography transformations torepresent the 2D relationship between pixels in different frames.• Solve parallel effects in the interpolation procedure. Due to the translation ofcamerasand the naive assumption that homograph transformation ignores suchdepth difference, parallel effect significantly affects interpolation results when depthvaries dramatically.
Reference[1] Gao, J. , Kim, S. J. , & Brown, M. S. (2011, June). Constructing image panoramas using dual-‐homography warping. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 49-‐56). IEEE.
[2] Xiong, Y., & Turkowski, K. (1998, October). Registration, calibration and blending in creating high quality panoramas. In Applications of Computer Vision, 1998. WACV'98. Proceedings., Fourth IEEE Workshop on (pp. 69-‐74). IEEE.
[3] Forsyth, D. A., & Ponce, J. (2003). A Modern Approach. Computer Vision: A Modern Approach.
[4] Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge university press.
[5] Chen, S. E., & Williams, L. (1993, September). View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques (pp. 279-‐288). ACM.
[6] Higham, N. J. (1987). Computing real square roots of a real matrix. Linear Algebra and its applications, 88, 405-‐430.
Electric & Computer EngineeringCollege of Science &
Engineering
ApproachDual-‐homography Transformation
Interpolation
New image sequences that canrun at original or even higher
frequency
Original discontinuous imagesequences
Stitching
Homography Matrix relatedtwo images
For eachpair of consecutive images, the viewsare not strictly related by a homographytransformation due to the translation betweentheir viewpoints. The 3D structure of sceneaffects this transformation. However, we canapproximate this by using several differenthomography transformations in spatial layout ofimages.
Real Schur decompositionIn order to Interpolate the the transformation,we decompose homography matrix bycomputing its real square root.f(A) function computes its square root, where Qis real orthogonal matrix.And T inherits R’s upper quasitriangular structure.
K-‐meansIn order todivide the image in special layout, weapply 3-‐demissional k-‐meanson the scaled 3dcoordinates [x’, y’, z’] for each feature pointextracted from image pairs.The relative depth Z of each feature point iscomputed using optical flow.Then, for each cluster of feature points, computelocal homography matrix using RANSAC andleast-‐square method.
Algorithm:
K-‐means cluster result
Cluster feature points based on k-‐means with 2 bins
Dual-‐homgraphytransformation result
Stitched imageswithout blending
Laplacian Blending result
Blend images andeliminate seamline
