Automatic Registration of Mobile LiDAR and Spherical Panoramas

Ruisheng WangNOKIA

Chicago, IL, USAruisheng.wang@nokia.com

Frank P. FerrieCentre for Intelligent Machines

McGill University, Canadaferrie@cim.mcgill.ca

Jane MacfarlaneNOKIA

Chicago, IL, USAjane.macfarlane@nokia.com

Abstract

We present an automatic mutual information (MI) regis-tration method for mobile LiDAR and panoramas collectedfrom a driving vehicle. The suitability of MI for registra-tion of aerial LiDAR and aerial oblique images has beendemonstrated in [17], under an assumption that minimiza-tion of joint entropy (JE) is a sufficient approximation ofmaximization of MI. In this paper, we show that this as-sumption is invalid for the ground-level data. The entropy ofa LiDAR image can not be regarded as approximately con-stant for small perturbations. Instead of minimizing the JE,we directly maximize MI to estimate corrections of cameraposes. Our method automatically registers mobile LiDARwith spherical panoramas over an approximate 4 kilome-ter drive, and is the first example we are aware of that testsmutual information registration in large-scale context.

1. IntroductionImage-to-range registration is prerequisite for many ap-

plications. The registration result is critical not only fortexture-mapping 3D models of large-scale scenes, but alsofor applications such as image based upsampling of rangedata [6, 8, 21, 24], image-guided range segmentation [4, 2],and 3D scene modeling [5]. The problem of image-to-rangeregistration involves the alignment of 2D images with 2Dprojections of 3D range data, consisting of estimating therelative camera pose with respect to the range sensors.

There has been a considerable amount of research in reg-istering images with range data. Existing methods rangefrom keypoint-based matching [3, 7, 11], structural featuresbased matching [13, 14, 20, 23], to Mutual Informationbased registration [17]. The range data include terrestrialor aerial LiDAR, and the images include vertical or obliqueaerial images, and ground-level images.

Keypoint based matching [3, 11] is based on the similar-ity between laser intensity images and corresponding cam-era images. First, each pixel of the laser intensity imageis encoded with its corresponding 3D coordinate. Then

feature points are extracted by using either SIFT [16] orForstner operators [10] from both images. A robust match-ing strategy based on RANSAC [9] and/or epipolar geome-try constraint is employed to determine the correspondencepairs for computing the fundamental matrix. Sensor regis-tration is then achieved based on a robust camera spatial re-section. Ding et al. [7] registered oblique aerial images witha 3D model generated from aerial LiDAR data based on 2Dand 3D corner features in the 2D images and 3D LiDARmodel. The correspondence between extracted corners wasbased on a Hough transform and generalized M-estimatorsample consensus. The resultant corner matches are usedin Lowe’s algorithm [15] to refine camera parameters esti-mated from a combination of vanishing point computationand GPS/IMU readings. In general, the feature point extrac-tion and robust matching are the key to a successful regis-tration for this type of approaches.

Instead of matching points, structural feature basedmethods [13, 14, 20, 23] match structural features in both2D and 3D space to estimate the relative camera pose. Di-rect matching single line features is error-prone because ofthe noise in both LiDAR and image data as well as the ro-bustness of the detection algorithms. High-level structuralfeatures are helpful to increase the robustness of both de-tection and matching. Wang and Neumann [23] registeredaerial images with aerial LiDAR based on matching so-called “3 Connected Segments” in which each linear fea-ture contains 3 segments connected into a chain. Theyused a two-level RANSAC algorithm to refine the puta-tive feature matches, and estimated camera pose using themethod described in [12]. Liu et al. [13, 14, 20] extractedso called “rectangular parallelepiped” features, which arecomposed of vertical or horizontal 3D rectangular paral-lelepipeds in the LiDAR and 2D rectangles in the images,to estimate camera translation with a hypothesis-and-testscheme. Camera rotation was estimated based on at leasttwo vanishing points. Since vanishing points are required,their methods work well for ground-level data but are notefficient to handle aerial data with a weak perspective ef-fect.

All the above methods are dependent on either the strongpresence of parallel lines to infer vanishing points, or avail-ability of feature pair correspondence, which limits theirapplicability and robustness. A recent method [17] usingstatistical metrics, such as Mutual Information [22], as asimilarity measure for registering oblique aerial images andaerial LiDAR does not require any feature extraction pro-cess. This method searches for the optimal camera posethrough maximizing the mutual information between cam-era images and different attributes of LiDAR such as theLiDAR intensity images, depth maps, or a combination ofboth. Instead of using features, the mutual informationmethod evaluates statistical measures using all the pixels inboth images, which avoids the problems of feature extrac-tion and correspondence. Thus mutual information registra-tion method holds potential to be a robust solution.

This paper deals with the problem of the registration be-tween mobile LiDAR and spherical panoramas. We adoptedmutual information metric for the similarity measure. Thepaper [17] registers aerial LiDAR with aerial oblique im-ages. The assumption is that the entropy of the LiDARimages remains approximately constant for small perturba-tions, and minimizing the joint entropy is equivalent to max-imizing the mutual information. This may be the case forthe airborne data, but is not suitable for our ground basedapplications. The small perturbations may have larger ef-fect on the LiDAR rendering compared with airborne case.Instead of minimizing the joint entropy, we maximize themutual information with a different implementation to [17].Our method is complementary, since we deal with the datacollected from ground level. Second, our algorithm is fullyautomatic and has been designed to run efficiently on ametropolitan scale LiDAR/spherical image database usingdifferent representations of LiDAR from [17]. To the bestof our knowledge, our work is the first registration examplethat automatically runs through a large-scale database in thecontext of mutual information registration.

2. Data Acquisition

Figure 1: “NAVTEQ TRUE” mobile mapping system

Data is collected from a mobile mapping system shownin Figure 1, which is composed of a 360◦ LiDAR sensor(Velodyne HDL-64E), six high-resolution cameras, a La-dybug 3 camera, GPS, Inertial Measurement Unit (IMU),and Distance Measurement Instrument (DMI). The Velo-dyne LIDAR sensor consists of 64 lasers mounted on upperand lower blocks of 32 lasers each and the entire unit spins,and generates over one million points per second. The La-dybug 3 covers more than 80 percent of a full sphere, withsix high quality 1600x1200 Sony CCD sensors, and pro-vides up to 12 MP images at 15 Frames Per Second (FPS).All of these sensors are geo-referenced through a GPS andIMU.

3. The methodWe start with the work [17] that registers aerial LiDAR

with aerial oblique images based on MI. LiDAR intensityimages normally look very similar to gray-scale camera im-ages with, of course, a much lower resolution. This corre-lation makes MI a suitable measure to evaluate their simi-larity. In [17], Mastin et al. define p(x, y) and l(x, y) asthe intensity camera image and projected LiDAR featuresrespectively on the xy image plane. For a specific cam-era matrix T , the projected LiDAR features is given by lT .MI based registration methods find the optimal camera ma-trix that maximizes the MI between camera images and pro-jected LiDAR features:

TMI = argmaxT

I(p; lT ). (1)

Mutual information is expressed in terms of entropy.

I(p; lT ) = H(p) +H(lT )−H(p, lT ). (2)

where H(p) is the entropy of the optical features, H(lT )is the entropy of the LiDAR features, andH(p, lT ) is the JE.Mastin et al. [17] assume that the minimization of the JE isan approximation of the maximization of the mutual infor-mation according to Equation 2. They minimize Equation 3instead of maximizing MI over T for the registration.

H(p, lT ) ≈Nu∑i=1

Nv∑j=1

d(pi, lj ;T ) log(d(pi, lj ;T )), (3)

where d(•) denotes a joint histogram estimated of a den-sity and Nu and Nv denote the number of distinct bins foreach modality. In their case, Nu, Nv = 256.

They use a generic camera model, the finite projectivecamera as described in [12]. Under this camera model, apoint in space is mapped to the point on the image plane by

P = KR[I | −C]. (4)

Where C = [Cx, Cy, Cz]T is the camera center, I is the

identity matrix, and R is the camera rotation matrix. Thematrix K is the camera calibration matrix.

In this paper, we show that their assumption is invalid forour case. The small perturbations in camera poses may havelarger effect on the LiDAR rendering in the ground-leveldata. The entropy of a LiDAR image can not be regardedas approximately constant for small perturbations. This isdemonstrated by a perturbation analysis shown in Figure 2,which shows how the normalized MI and JE vary aroundthe initial registration in terms of altered camera poses. Theleft columns show the results for the normalized MI, andthe right columns show the results for the normalized JE.We use four representative scenes for this test as shown inFigure 2. Since the correct registration value should be nearthe initial registration, we set all parameters at their initialvalues and vary each parameter to view the shape of the costfunctions. The range of camera parameter perturbations are±2 units, meters for translation and degrees for orientation.The step size for the perturbation analysis is 0.1 units. Sowe have 40 measurements for each camera parameter per-turbation analysis, and the the number 0 in X axis corre-sponds to the initial registration, where camera pose correc-tions are set to zero. The data 1, 2, 3 ,4 in Figure 3 showthe results of the probe analysis, which corresponds to theimage 1, 2 ,3, 4 of Figure 2, respectively. The data 5 inFigure 3, which are indicated by solid black lines, are meancurves of the four curves showing a general trend. For allthe probe experiments, a global maximum within the testedcamera parameter range in terms of normalized MI can bealways found around the initial registration. However, thisis not the case for normalized JE, which a global minimacan not be found around the initial registration. The mini-mization of the JE does not work on our ground level mobileLiDAR data. Instead, we maximize the MI to obtain correctcamera solutions.

(a) Image 1 (b) Image 2

(c) Image 3 (d) Image 4

Figure 2: Probe analysis on four representative scenes

(a) X displacements MI (b) X displacements JE

(c) Y displacements MI (d) Y displacements JE

(e) Z displacements MI (f) Z displacements JE

(g) Roll displacements MI (h) Roll displacements JE

(i) Pitch displacements MI (j) Pitch displacements JE

(k) Yaw displacements MI (l) Yaw displacements JE

Figure 3: Probe analysis on camera translation and orienta-tion parameters for evaluation of normalized MI and JE

3.1. Coordinate Framework

Our optical images are panoramic images, which areoriginally taken from Ladybug III consisting of 6 Fisheyecameras. The 6 individual images are stitched via α blend-ing. For storage purpose, the stitched images are trans-formed to panoramic images via a Cylindrical EquidistantProjection as shown in the left image of Figure 4, whichis the original panoramic images. We map each panoramicimage onto a sphere and view it from the sphere center (thecenter of the camera), to generate spherical panoramas (lin-ear perspective images) as shown in the right image of Fig-ure 4.

Figure 4: Panoramic and spherical view(Left) A panoramic view; (Right) A spherical view

Both LiDAR and image data are geo-referenced. We firstconvert the geographic coordinates into Earth-Centered,Earth-Fixed (ECEF) coordinates, and then transform intolocal tangent plane (LTP) coordinates. All computationsare based on local tangent plane coordinates. Each LiDARpoint p = (x, y, z)T in LTP coordinates is converted tospherical coordinates (θ, ϕ) by Equation 5,

θ = arccos(z√

x2 + y2 + z2), ϕ = atan2(y, x), (5)

where θ is the inclination (θ ∈ [0, π]) , and ϕ is the azimuth(ϕ ∈ (−π, π]). Each point’s corresponding location in thepanoramic image (r, c) is computed by Equation 6,

r = int[θ

πH], c = int[(

ϕ

2π+ 0.5)W ], (6)

where H and W are the height and width of the panoramicimages respectively.

3.2. Mutual Information Registration

Mutual Information methods have been widely used forthe multi-modal registration problem in the medical imag-ing domain (e.g., registration of CT and MRI). Recentlythey also have been applied to the problem of registeringairborne LiDAR data with oblique aerial images [17]. Themutual information of two random variables X and Y canbe defined as

I(X;Y ) =

∫Y

∫X

p(x, y) log(p(x, y)

p1(x)p2(y))dxdy, (7)

where p(x, y) is the joint probability density function ofX and Y , and p1(x) and p2(y) are the marginal probabilitydensity functions of X and Y respectively. We assume thatan approximate camera pose is provided from the imper-fect GPS/IMU solution. The problem here is to estimate thecorrection of the relative camera pose between the LiDARsensor and the Ladybug camera. The spherical panorama ischosen as a fixed image because the camera view point hasto stay in the center of the sphere to generate perspectiveimages. Once the camera moves out of the sphere center,the spherical image will be distorted. The LiDAR image isselected as a moving image, where new LiDAR images aregenerated at each iteration during the optimization process.Both LiDAR and spherical images are rendered onto a planefrom the camera center using OpenGL for the MI evaluationunder a pinhole camera model. The perspective camera im-age is generated by rendering the spherical panorama with aview port of 940 by 452 pixels. The LiDAR data set is nor-mally very large. In our experiments, each scene contains 8million LiDAR points. To make 3D rendering efficient, wealso integrate the OpenGL rendering of the LiDAR featuresinto the registration pipeline to speed up the optimizationprocess.

We use three different representations of the LiDAR datawith spherical panoramas for evaluating mutual informa-tion. The first representation of LiDAR is the projectedLiDAR points with intensity information (see Figure 5b).We call it a LiDAR intensity image which looks similar toa gray-scale camera image. The second representation isthe projected LiDAR points without intensity information(see Figure 5c). The third is the depth map of the LiDARpoint cloud (see Figure 5d). The point cloud is renderedwith depth intensities, where brighter points indicate a fur-ther distance to the camera center. We use gray scale cameraimages instead of color images (see Figure 5a) for the mu-tual information computation. The Nelder-Mead downhillsimplex method [18] is used to optimize the cost functionas it does not require derivatives of the function.

4. ExperimentsFor the experiments, we made the algorithm automati-

cally run through an approximate 4 kilometer drive. Thedriving routine is shown with light blue line in Figure 6a.An illustration of the collected data is shown in Figure 6b,where the distance between each bubble is around 4 meters.The test data were collected in the northwestern suburban ofChicago, Illinois, which includes residential, urban streets,and highway scenes. The data is in binary format containingaround 4 GB LiDAR data (about 226 million points) and 1GB panoramic images (814 spherical panoramas). We usethe camera views perpendicular to or parallel to the vehicledriving direction to generate perspective images. For qual-itative analysis, we selected 10 representative urban scenes

(a) Camera gray-scale image (b) LiDAR intensity image

(c) LiDAR points (d) LiDAR depth map

Figure 5: Camera image and three representations of Li-DAR point clouds in a same scene

(a) Test data overview (b) An illustration of the data

Figure 6: Test data.

for the evaluation using the three different representationsof the LiDAR data described earlier.

We start with an approximate initial registration that isavailable from the GPS/IMU system. The initial camerapose corrections are set to zero. The optimization will com-pute the final camera corrections. The experiments for thequalitative analysis were performed on a laptop PC witha dual core 2.60GHz processor and 2 GB of RAM. TheNVIDIA Quadro NVS 135M video card was used. Theregistration algorithms were implemented in C++, and theimplementations of Mutual Information and Amoeba opti-mization were from ITK [1]. We adjust the tolerances onthe optimizer to define convergence. The tolerance on the 6parameters is 0.1 (the unit for translation parameters is me-ter and degree for orientation parameters). We also set thetolerance on the cost function value to define convergence.The metric returns the value of mutual information, whichwe set the tolerance to be 0.001 bits. The initial size of thesimplex is set to 1, and the maximum iteration number isset to 200. In our experiments, almost all registrations con-verged in less than 150 iterations.

(a) Optical image with correspondence points shown in red

(b) LIDAR intensity image with correspondence points shownin red

Figure 7: Optical image (a) and LIDAR image (b) with 15manually selected correspondence points

4.1. Performance Evaluation

To quantitatively measure the registration results, wecompare the registration accuracy in terms of pixel off-set between LiDAR and camera images before and afterthe registration. We manually selected 15 correspondencepoints in each spherical image and LiDAR intensity im-age. Figure 7 shows an example of a spherical image anda LIDAR intensity image marked with 15 correspondencepoints. Figure 8 shows computed Euclidean Distance his-togram of the correspondence points for all the 10 imagesbefore and after registration. The horizontal axis stands forthe pixel offsets, and the vertical axis stands for the fre-quency. Before the MI registration, most correspondencepoints have 2-3 pixel errors. After the registration, mostof the correspondence points are within 1 pixel. The pixeloffset histograms using other LiDAR representations aresimilar. Table 1 show the run time for the 4 representa-tive scenes. Testing on the 4km drive shows that using theLiDAR points without intensity normally runs fast with lessiterations. Using LiDAR points with intensity normally per-forms the most robustly followed by using LiDAR pointswithout intensity and using the LiDAR depth maps. Wealso study the convergence of the optimization using threedifferent measures of mutual information. Without loss ofgenerality, we choose the data shown in Figure 5 as an ex-ample. Figure 9 shows the sequence of metric values com-puted as the optimizer searched the parameter space usingthese three different representations of the LIDAR data. Themeasure initially increases overall with the number of iter-

ations. After about 50 iterations, the metric value reachessteady state without further noticeable convergence.

(a) Pixel offset histogram before MIregistration

(b) Pixel offset histogram after MI reg-istration

Figure 8: Histogram of Euclidean distance of pixel offsetfor before registration (a) and after registration (b). The his-tograms were generated with samples for all ten test images

Figure 9: Mutual information values produced during theregistration process

An example of the registration is shown in Figure 10.After MI registration, the misalignment is not noticeable.

Figure 10: An example of the registration results

MI Measure LiDAR in-tensity

LiDARwithoutintensity

LiDARdepth map

Image 1 0.93 0.50 1.08Image 2 0.87 0.55 0.88Image 3 0.70 0.4 1.06Image 4 0.73 0.38 0.87Mean 0.88 0.47 0.87

Table 1: Registration times in minutes for the 4 imags inFigure 2

4.2. Perturbation Analysis

We plot the normalized MI of Figure 5a in Figure 11using the three LiDAR attributes. Figure 11 (red-intensity,green-point only, yellow-depth map) demonstrates that eachcurve has a single peak over a subset of the displacement pa-rameters around the initial registration, which demonstratesthe effectiveness of maximization of MI for computing op-timal camera corrections.

(a) X displacements (b) Y displacements

(c) Z displacements (d) Roll displacements

(e) Pitch displacements (f) Yaw displacements

Figure 11: Plots of normalized mutual information

We also investigate the failure cases. In our experiments,the algorithm works well in feature-rich environments suchas residential areas, but often fails in scenes with sparse fea-tures or containing moving objects like cars, particularlyhighway scenes. In our case, the highway scenes mostlyfail. The partial overlap between LiDAR point clouds andcamera images is another reason. The LiDAR scanner onlycan reach up to 120 meters, while the camera can alwayshave a larger field of view than the LiDAR scanner. Figure12a and Figure 12b shows one typical failure case in a high-way scene. The cars in the camera image (Figure 12a) don’tappear in the LiDAR image (Figure 12b). The LiDAR im-age only partially covers the camera image, for instance, thetrees and buildings in the far distance in the camera imagedon’t appear in the LiDAR image. In [17], the authors claimthat they only use the image pixels with corresponding pro-jected LiDAR points for mutual information calculation andothers are considered background points and discarded. Wetried to discard the background points and only use overlap-ping regions for mutual information computation, but theresults were worse than using entire images for MI com-putation. When using entire images for MI computation,the background such as sky appears similar in both LiDARand camera images, which largely contributes the MI score.When using overlapping regions for MI computation, theLiDAR images contain no sky. Therefore the background isnot used in the MI computation, which affects the MI eval-uation. Failure cases are also due to overexposed images(Figure 12c and Figure 12d), particularly in the case wherethe vehicle drives through/out of a tunnel.

(a) Ladybug image (b) LiDAR image

(c) Overexposed Ladybug image I(d) Overexposed Ladybug imageII

Figure 12: Failure cases

4.3. Camera Pose Corrections

One of our interests is to investigate how camera poseerrors change during the data collection. To do so, we man-ually selected 100 successful registrations (using the regis-trations from camera views vertical to the vehicle driving

direction) by carefully examining the alignment of majorfeatures in the registered images, and plot the camera posecorrections shown in Figure 13. Figure 13a shows the cam-era translation corrections, and Figure 13b shows the cam-era orientation corrections. Our observation is that almostall the camera translation corrections are within 0.1 meter,while orientation corrections are within 1 degree.

(a) Camera translation corrections (b) Camera orientation corrections

Figure 13: Plots of camera pose corrections using 100 suc-cessful registrations

5. Conclusions and Future WorkIn this paper, we have investigated MI registration for

ground level LiDAR and images. The existing method [17]for registering airborne LiDAR with aerial oblique imagesdoes not work on the LiDAR and images collected fromthe mobile mapping system because the assumption used in[17] is violated in the case of mobile LiDAR data. Insteadof the minimization of the joint entropy, we use maximiza-tion of mutual information for computing optimal cameracorrections. The algorithms work with unstructured LiDARdata and perspective rectified panoramic images generatedby rendering panorama into an image plane using sphericviews. We tested the algorithm on various urban scenesusing three different representations of LiDAR data withcamera images for the mutual information calculation. Ourmutual registration algorithm automatically runs through alarge-scale mobile LiDAR and panoramic images collectedover a metropolitan scale. It is the first example we areaware of that tests mutual information registration in large-scale context. With the initiative of urban 3D modelingfrom location based service providers such as Nokia andGoogle, this work is particularly important for combiningground-level range and visual data for large-scale photo-realistic city modeling.

We generated perspective images from spherical imagesusing the view either perpendicular or parallel to the vehi-cle driving direction. Therefore, we just used 1/6 of theentire spherical image for the mutual information registra-tion, which does not efficiently use all the available infor-mation containing in the 360◦ panoramic images. For thefuture work, one possible approach is to project entire Li-

DAR points along with spherical images onto 6 cube facesusing a Quadrilateralized Spherical Cube mapping [19] orother linear projections. Because the sky and the ground donot provide much useful information, we actually just need4 faces for the mutual information registration. To speed upthe computation, a multi-resolution approach can be em-ployed by establishing image pyramids on both images.This coarse-to-fine strategy can improve the performanceof the registration algorithm and also increases robustnessby eliminating local optima at coarser level. One of the lim-itations of MI metric is that the intensity histograms containno spatial information. One possible direction is to incorpo-rate spatial context information into the metric to improvethe robustness of the similarity measure.

6. Acknowledement

The first author would like to thank Roman Ostrovskiyfor his help and discussion.

References[1] Insight segmentation and registration toolkit (itk).

http://www.itk.org/.[2] S. Barnea and S. Filin. Segmentation of terrestrial laser scan-

ning data by integrating range and image content. In TheProceedings of XXIth ISPRS Congress, 2008.

[3] S. Becker and N. Haala. Combined feature extraction for fa-cade reconstruction. In ISPRS Workshop on Laser Scanning,2007.

[4] M. Deveau, N. Paparoditis, M. Pierrot-deseilligny, X. Chen,and G. Thibault. Strategy for the extraction of 3d architec-tural objects from laser and image data acquired from thesame viewpoint. In International Archives of Photogramme-try, Remote Sensing and Spatial Information Sciences, vol-ume 36, 2005.

[5] P. Dias, V. Sequeira, F. Vaz, and J. G. Goncalves. Registra-tion and fusion of intensity and range data for 3d modellingof real world scenes. 3D Digital Imaging and Modeling, In-ternational Conference on, pages 418 – 425, 2003.

[6] J. Diebel and S. Thrun. An application of markov randomfields to range sensing. In Proceedings of Conference onNeural Information Processing Systems (NIPS), 2005.

[7] M. Ding, K. Lyngbaek, and A. Zakhor. Automatic registra-tion of aerial imagery with untextured 3d lidar models. InCVPR, 2008.

[8] J. Dolson, J. Baek, C. Plagemann, and S. Thrun. Upsamplingrange data in dynamic environments. Computer Vision andPattern Recognition, IEEE Computer Society Conference on,0:1141–1148, 2010.

[9] M. A. Fischler and R. C. Bolles. Readings in computer vi-sion: issues, problems, principles, and paradigms. chapterRandom sample consensus: a paradigm for model fittingwith applications to image analysis and automated cartog-raphy, pages 726–740. 1987.

[10] W. Forstner and E. Gulch. A fast operator for detection andprecise location of distinct points, corners and centres of cir-cular features. In ISPRS Intercommission Workshop Inter-laken, pages 281–305, 1987.

[11] D. Gonzalez Aguilera, P. Rodriguez Gonzalvez, andJ. Gomez Lahoz. An automatic procedure for co-registrationof terrestrial laser scanners and digital cameras. ISPRS Jour-nal of Photogrammetry and Remote Sensing, 64(3):308–316,May 2009.

[12] R. I. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press, ISBN:0521540518, second edition, 2004.

[13] L. Liu and I. Stamos. Automatic 3d to 2d registration for thephotorealistic rendering of urban scenes. In Proceedings ofthe 2005 IEEE Computer Society Conference on ComputerVision and Pattern Recognition (CVPR’05) - Volume 2 - Vol-ume 02, CVPR ’05, pages 137–143, 2005.

[14] L. Liu and I. Stamos. A systematic approach for 2d-image to3d-range registration in urban environments. In InternationalConference on Computer Vision, pages 1–8, 2007.

[15] D. G. Lowe. Three-dimensional object recognition fromsingle two-dimensional images. Artif. Intell., 31:355–395,March 1987.

[16] D. G. Lowe. Distinctive image features from scale-invariantkeypoints. Int. J. Comput. Vision, 60:91–110, November2004.

[17] A. Mastin, J. Kepner, and J. Fisher. Automatic registrationof lidar and optical images of urban scenes. Computer Vi-sion and Pattern Recognition, IEEE Computer Society Con-ference on, pages 2639–2646, 2009.

[18] J. A. Nelder and R. Mead. A simplex method for functionminimization. Computer Journal, 7:308–313, 1965.

[19] F. P. Chan and O’Neill. Feasibility study of a quadrilater-alized spherical cube earth data base. Computer SciencesCorp., EPRF Tech. Report 2-75, Prepared for the Environ-mental Prediction Research Facility, Monterey, California,1975.

[20] I. Stamos, L. Liu, C. Chen, G. Wolberg, G. Yu, and S. Zokai.Integrating automated range registration with multiview ge-ometry for the photorealistic modeling of large-scale scenes.Int. J. Comput. Vision, 78:237–260, July 2008.

[21] L. A. Torres-Mendez and G. Dudek. Reconstruction of 3dmodels from intensity images and partial depth. In Proceed-ings of the 19th national conference on Artifical intelligence,AAAI’04, pages 476–481, 2004.

[22] P. Viola and W. M. Wells, III. Alignment by maximizationof mutual information. International Journal of ComputerVision, 24:137–154, 1997.

[23] L. Wang and U. Neumann. A robust approach for automaticregistration of aerial images with untextured aerial lidar data.pages 2623–2630, 2009.

[24] Q. Yang, R. Yang, J. Davis, and D. Nister. Spatial-depthsuper resolution for range images. Computer Vision andPattern Recognition, IEEE Computer Society Conference on,0:1–8, 2007.