(12) Ulllted States Patent (10) Patent N0.: US 8,593,506 B2peleg/patents/US8593506.pdf · US....

US008593506B2

(12) Ulllted States Patent (10) Patent N0.: US 8,593,506 B2 Peleg et al. (45) Date of Patent: Nov. 26, 2013

(54) METHOD AND SYSTEM FOR FORMING A (58) Field of Classi?cation Search PANORAMIC IMAGE OFA SCENE HAVING USPC ........................... .. 348/36; 382/284, 294, 154 MINIMAL ASPECT DISTORTION See application ?le for complete search history.

(75) Inventors: Shmuel Peleg, Mevasseret Zion (IL); (56) References Cited Alex Rav-Acha, Ramat Denia (IL); Giora Engel, Mevasseret Zion (IL) US‘ PATENT DOCUMENTS

_ _ 5,850,352 A * 12/1998 MoeZZi et al. .............. .. 345/419 (73) Ass1gnee: Ylssum Research Development 6,078,701 A >l< @2000 Hsu et a1‘ ,,,,,,,,,,,,,,,,,, n 382/294

Company of the Hebrew University of (Continued) Jerusalem, Jerusalem (IL)

( * ) Notice: Subject to any disclaimer, the term of this FOREIGN PATENT DOCUMENTS

patent is extended or adjusted under 35 W0 WO 99/06943 2/l999 U.S.C. 154(b) by 844 days. wo wo 2008/111080 9/2008

(21) Appl. N0.: 12/530,064 OTHER PUBLICATIONS

_ _ International Search Report issued Jul. 24, 2008 in International (22) PCT Flled- Mar- 13’ 2008 Application No. PCT/IL2008/000356.

(86) PCT N0.: PCT/IL2008/000356 (Continued)

’ rzmar xammeri ma . outa §371(C)(1)~ P ' yE ' Al' N B h ’ ' ‘ ’ [Zorn , enl, 0r zrm * toe 1ves (2) (4) Date Sep 4 2009 74 A ey Ag F- 8 1R- LLP

PCT Pub- Date? seP- 18: 2008 A panoramic image is generated from a sequence of input _ _ _ frames captured by a camera that translates relative to a scene

(65) P1101‘ Publleatlen Data having at least tWo points at different distances from the Us 2011/0043604 A1 Feb‘ 24’ 2011 camera. A processor (13) is responsive to optical ?oW

between corresponding points in temporally different input frames for computing ?oW statistics for at least portions of

Related US- Application Data some of the input frames and for computing respective stitch

(60) Provisional application No. 60/894 946 ?led on Mar. ingposts between some ofthe pomqns and.respective neigh 15 2007 provisional application’ NO’ 60/945 338 borlng porttlons' thereodf. A selection £111' (18) selectsh a

’ ’ ' ’ ’ sequence 0 portions an respective ne1g ormg portions t at ?led on Jun. 20, 2007. minimizes a cost function that is a function of the How statis

tics and stitching costs. A stitching unit (21) stitches the (51) Int‘ Cl‘ selected portions and respective neighboring portions so as to

G06K 9/00 (200601) form a panoramic image of the scene, Which may then be H04N 7/ 00 (201101) displayed or post-processed.

(52) US. Cl. USPC ............ .. 348/36; 382/154; 382/294; 382/284 20 Claims, 17 Drawing Sheets

@ Obtain an optical ?ow between

corresponding points in temporally different input frames

Use the optical flow to compute ?ow statistics for at least portions

of some of the input flames

Use the optical flow to compute rapective stitching costs between some of the portions and resper» tive neighboring portions thereof

Identify a sequence ofselecied pDl’liDrlS and respective neigh boring portions that minimizes a cost function that is a function of the ?ow statistics and stitching

oosts

Stitch the selected portions and respective neighboring portions so as to form a panoramic image of

the scene

US 8,593,506 B2

(56) References Cited

U.S. PATENT DOCUMENTS

6,157,747 A * 12/2000 SZeliski et al. .............. .. 382/284 6,532,036 B1 * 3/2003 Peleg et al. ................... .. 348/36

6,665,003 B1 12/2003 Peleg et al. 6,813,391 B1* 11/2004 Uyttendaele et al. ....... .. 382/284 6,956,603 B2 * 10/2005 Fujii ........................... .. 348/169

7,006,124 B2 2/2006 Peleg et al. 7,110,617 B2 * 9/2006 Zhang et a1. ................ .. 382/284

8,102,406 B2 * 2002/0061131 A1* 2002/0135672 A1*

1/2012 Peleg et al. . 345/629 5/2002 SaWhney et a1. . 382/154 9/2002 SeZan et al. ................... .. 348/36

2003/0184778 A1 10/2003 Chiba 2004/0022419 A1* 2/2004 Kesaniemi .................. .. 382/107

2005/0275727 A1* 12/2005 Laiet al. .................. .. 348/208.1 2006/0177150 A1* 8/2006 Uyttendaele et al. . 2006/0187305 A1* 8/2006 Trivedi et al. .... ..

2006/0228002 A1 * 10/2006 Zitnick et al. 2008/0187172 A1* 8/2008 Otsu et a1. ........ ..

2009/0303312 A1* 12/2009 Di Bernardo et al. 348/36 2010/0238266 A1* 9/2010 Jojic et a1. ........ .. 348/36 2010/0272311 A1* 10/2010 Nir et al. ..................... .. 382/100

382/284 . 348/169

. 382/107

. 382/103

OTHER PUBLICATIONS

AgarWala, et al, Photographing Long Scenes with Multi-I/leWpoint Panoramas, ACM Trans. Graph, 25(3):853-861, 2006.

Buehler, et al, Unstructured Lumigraph Rendering, SIGGRAPH 2001. Lucas-Kanada Method, Wikipedia, XP-002488219, Jul. 15, 2008. Ono, et al, Ego-Motion Estimation for Ej?cient City Modeling by Using Epipolar Plane Range ImageAnalysis, in ITSWC2003, Nov. 2003. Rav-Acha, et al, A Uni?ed Approach for Motion Analysis and Wew Synthesis, in Second IEEE International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT), Thessaloniki, Greece, Sep. 2004. Rav-Acha, et al, Mosaicing with Parallax Using Time Warping, in Second IEEE Workshop on Image and Video Registration, Washing ton, DC, Jul. 2004. Roman, et al, Automatic Multiperspective Images, in Proceedings of Eurographics Symposium on Rendering, pp. 161-171, 2006. Roman, et al, Interactive Design of Multi-Perspective Images for I/lsualizing Urban Landscapes, in IEEE Visualization 2004, pp. 537 544, Oct. 2004. WeXler, et al, Space-Time Scene Manifolds, in ICCV’05, vol. 1, pp. 858-863, 2005. Zheng, Digital RoutePanoramas, IEEE Multimedia, 7(2):7-10, Apr. Jun. 2000. Zhu, et al, Generalized Parallel-Perspective Stereo Mosaics from Airborne I/ldeo, PAMI, 26(2):226-237, Feb. 2004. Zomet, et al, Mosaicing New Wews.‘ The Crossed-Slits Projection, PAMI, pp. 741-754, Jun. 2003.

* cited by examiner

US. Patent Nov. 26, 2013 Sheet 1 0f 17 US 8,593,506 B2

FIG. 1a FIG. 1b FIG. 10

FIG. 1d FIG. 1e

US. Patent N v. 26, 2013 Sheet 2 0f 17 US 8,593,506 B2

FIG. 2b FIG. 2a

FIG. 2c

Sheet 3 0f 17 US 8,593,506 B2 US. Patent Nov. 26, 2013

lk

Rotation Warp

Dk-1

Transaction Warp

FIG. 3a

Graph cuts stereo

Mpedsgq

Dk

FIG. 3b

U S. Patent Nov. 26, 2013 Sheet 4 0f 17 US 8,593,506 B2

FIG. 4b FIG. 4a

FIG. 4d FIG. 40

US. Patent Nov. 26, 2013 Sheet 5 0f 17

- Compute motion (u, v) between I, and I2 - Warp lztowards /, according to v - Set the relative translation: TX = u

V

Estimate inverse depth 01 using graph cuts between I; ‘

and the warped /2

V

- Estimate relative motion (7: Fr) between /;and /2, given D1

Until the motion conv

- Warp /2 towards /7 according to R

FIG. 5

=estihta1te matron par‘arzreltere {7 {$8,} mm, between 11,5, and t5, giver?

'Wam I, rewards EH, with Interim} E1,

predict the itweise depth map 5,, using D,“ and the manna pammezesrs- {teak}

US 8,593,506 B2

My» m-e?iimm? {3,- H5315 hhkq a’ Mr FEE ' graph ems between

warned t, and‘

FIG. 6


an .9“.

mm .QE A/f/Pl.

fl]

a to

E0

E0


Emma?“


no .QE

)(A

mm .OE w OOQOOGOOQWQ OQGGGQOO OQG. OGQDSOOOOQQD


0.4

FIG. 10

FIG. 11a

FIG. 11b


FIG. 11c

FIG. 11d

FIG. 13b FIG. 13a

US. Patent Nov. 26, 2013 Sheet 11 0f 17

FIG. 12a FIG. 12b

US 8,593,506 B2

5 5% , f

FlG. 12c

US. Patent N v. 26, 2013 Sheet 12 0f 17 US 8,593,506 B2

FIG. 14b FIG. 14a

FIG. 15a

FIG. 15b


“22:... \mxbx

\ , a ...\.“.\.“.\.“....\ ...............................\.“.... z????? a: .“............ .. ............................................ ~ ~Szzzz?aaaaa w

a .99.". ........r." ................ ..

‘5:: 252a: Fmme m.

‘773 me Ms,

1

‘i 5%

:

w

29

@??m an MK QBNSW mm WE, wmmkmwb an cmmmwmm .mmmg


FIG. 17b FIG. 176 FIG. 17d FIG. 179

FIG. 180 FIG. 18d


Obtain an optical flow between corresponding points in temporally

different input frames

l Use the optical flow to compute

flow statistics for at least portions of some of the input frames

1 Use the optical flow to compute

respective stitching costs between some of the portions and respec tive neighboring portions thereof

1 Identify a sequence of selected portions and respective neigh boring portions that minimizes a cost function that is a function of the flow statistics and stitching

costs

1 Stitch the selected portions and

respective neighboring portions so as to form a panoramic image of

the scene

FIG. 19


11,’

112 Memory \

/'“ _ 14

13"’ Optical ?ow \ i

computation unit :5 Depth 4

% computation unit 8

2 a_

Rectification unit Pie-warping unit \ 17

Scaling unit ,__\ l 139

Selection unit 22 ~i I 20

1’ Post-warping AV’ 1, unit

. .l . 21 Stitching unit -\/

k

\ 10

FIG. 20


on

Pu .OE

US 8,593,506 B2 1

METHOD AND SYSTEM FOR FORMING A PANORAMIC IMAGE OF A SCENE HAVING

MINIMAL ASPECT DISTORTION

RELATED APPLICATIONS

This application claims bene?t of US. provisional appli cation Ser. No. 60/894,946 ?led Mar. 15, 2007 and 60/945, 338 ?led Jun. 20, 2007 Whose contents are included herein by reference.

FIELD OF THE INVENTION

The invention relates generally to the ?eld of image and video mosaicing and in particular to the presentation of mosaic images having perceived depth.

PRIOR ART

Prior art references considered to be relevant as a back ground to the invention are listed beloW and their contents are incorporated herein by reference. Additional references are mentioned in the above-mentioned US. provisional applica tion Nos. 60/894,946 and 60/945,338 and their contents are incorporated herein by reference. Acknowledgement of the references herein is not to be inferred as meaning that these are in any Way relevant to the patentability of the present invention. Each reference is identi?ed by a number enclosed in square brackets and accordingly the prior art Will be referred to throughout the speci?cation by numbers enclosed in square brackets. [1] A. AgarWala, M. AgraWala, M. Cohen, D. Salesin, and R.

SZeliski. Photographing long scenes with multi-viewpoint panoramas. ACM Trans. Graph, 25(3):853-861, 2006.

[2] J. Bergen, P. Anandan, K. Hanna, and R. Hingorani. Hier archical model-based motion estimation. In ECCV, pages 237-252, 1992.

[3] S. Birch?eld and C. Tomasi. A pixel dissimilarity measure that is insensitive to image sampling. PAMI, 20(4):401 406, 1998.

[4] S. Birch?eld and C. Tomasi. Multiway cut for stereo and motion with slanted surfaces. In ICCV, volume 1, pages 489-495, 1999.

[5] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. PAMI, 23(11):1222 1239, 2001.

[6] S. Gortler, R. GrZesZcZuk, R. SZeliski, and M. Cohen. The lumigraph. SIGGRAPH, 30:43-54, 1996.

[7] K. Hanna. Direct multi-resolution estimation ofego-mo tion andstructurefrom motion. In MOTION91, pages 156 162, 1991.

[8] R. Hartley and A. Zisserman. Multiple Wew Geometry. Cambridge University Press, second edition, 2004.

[9] L. Hong and G. Chen. Segment-based stereo matching using graph cuts. In CVPR, volume 1, pages 74-81, Los Alamitos, Calif., USA, 2004.

[10] M. Irani, P. Anandan, and M. Cohen. Direct recovery of planar-parallax from multiple frames. PAMI, 24(11): 1528-1534, November 2002.

[11] M. Irani, B. Rousso, and S. Peleg. Detecting and tracking multiple moving objects using temporal integration. In ECCV’92, pages 282-287, 1992.

[12] V. Kolmogorov and R. Zabih. Computing visual corre spondence with occlusions via graph cuts. In ICCV, vol ume 2, pages 508-515, July 2001.

[13] M. Levoy and P. Hanrahan. Light ?eld rendering. SIG GRAPH, 30:31-42, 1996.

20

25

30

35

40

45

50

55

60

65

2 [14] S. Ono, H. Kawasaki, K. Hirahara, M. KagesaWa, and K.

Ikeuchi. Ego-motion estimationfor e?icient city modeling by using epipolar plane range image. In ITSWC2003, November 2003.

[1 5] A. Rav-Acha and S. Peleg. A uni?ed approach for motion analysis and view synthesis. In Second IEEE International Symposium on 3D Data Processing Visualization, and Transmission (3DPVT), Thessaloniki, Greece, September 2004.

[16] A. Rav-Acha, Y. Shor, and S. Peleg. Mosaicing with parallax using time warping. In Second IEEE Workshop on Image and Video Registration, Washington, DC, July 2004.

[17] A. Roman, G. Garg, and M. Levoy. Interactive design of multi-perspective images for visualizing urban land scapes. In IEEE Visualization 2004, pages 537-544, Octo ber 2004.

[18] A. Roman and H. P. A. Lensch. Automatic multiperspec tive images. In Proceedings of Eurographics Symposium on Rendering, pages 161-171, 2006.

[19] M. Shi and J. Y. Zheng. A slit scanning depth ofroute panorama from stationary blur. In CVPR’05, volume 1, pages 1047-1054, 2005.

[20] Y. Wexler and D. Simakov. Space-time scene manifolds. In ICCV’05, volume 1, pages 858-863, 2005.

[21] J. Y. Zheng. Digital route panorama. IEEE Multimedia, 7(2):7-10, April-June 2000.

[22] Z. Zhu, E. Riseman, and A. Hanson. Generalized paral lel-perspective stereo mosaics from airborne videos. PAMI, 26(2):226-237, February 2004.

[23] A. Zomet, D. Feldman, S. Peleg, and D. Weinshall. Mosaicing new views: The crossed-slits projection. PAMI, pages 741-754, June 2003.

[24] US. Pat. No. 7,006,124. GeneraliZedpanoramic mosaic. [25] US. Pat. No. 6,665,003, System and method for gener

ating and displaying panoramic images and movies. [26] US2007003034 Apparatus and method for capturing a

scene using staggered triggering of dense camera arrays. [27] Chris Buehler, Michael Bosse, Leonard McMillan,

Steven Gortler, Michael Cohen, Unstructured lumigraph rendering, SIGGRAPH 2001.

BACKGROUND OF THE INVENTION

Many mosaicing applications involve long image sequences taken by translating cameras scanning a long scene. Thus applications are knoWn that include a video cam era mounted on a vehicle scanning city streets [14,1,17,21, 19], or a video camera mounted on a loW altitude aircraft

scanning a terrain [22]. Earlier versions of our Work on ego motion computation for sideWays moving cameras Were pro posed in [16,15]. They had initialiZation and robustness prob lems that are addressed in this patent. In addition, they did not address the computation of dense depth maps and the creation of undistorted mosaics.

In [1,17] methods are described for creating a multi-per spective panorama. These methods recover camera motion using structure-from-motion [8], matching features betWeen pairs of input images. Matched points are used to recover the camera parameters as Well as a sparse cloud of 3D scene points, recovery that is much easier When ?sheye lens are used as in [1]. Feature points as used in the above-described approaches Will be preferred in clean, high contrast, and unambiguous imagery. HoWever, direct methods may be pre ferred When feature points are rare, ambiguous, or noisy.

Image mosaicing can be regarded as a special case of creating a model of the observed scene. Having multiple

(12) Ulllted States Patent (10) Patent N0.: US 8,593,506 B2peleg/patents/US8593506.pdf · US....

Documents

Transcript of (12) Ulllted States Patent (10) Patent N0.: US 8,593,506 B2peleg/patents/US8593506.pdf · US....