SCAN LINE OPTIMIZATION FOR TRI STEREO PLANETARY IMAGES

R.H.Bhalerao∗, S.S. Gedam and J. Joglekar

Geomatics Lab, Centre of Studies in Resources Engineering, Indian Institute of Technology BombayMumbai, India 400076

(raghavendra.bhalerao, jyotij, shirish)@iitb.ac.in

Commission III

KEY WORDS: Tri-stereo, Disparity Space Image,

ABSTRACT:

In this paper, we propose a new scan line optimization method for matching the triplet of images. In the present paper, the tripletsare initially matched using an area based local method. The cost is stored in a structure called as the Disparity Space Image (DSI).Using the global minimum of this cost the initial disparity is generated. Next the local minima are considered as potential matcheswhere global minimum gives erroneous results. These local minima are used for optimization of disparity. As the method is a scannedline optimization, it use popularly resampled images. The experiment is performed using Terrain Mapping Camera images from theChandrayaan-1 mission. In order to validate the result for accuracy, Lunar Orbiter Laser Altimeter dataset from Lunar ReconnaissanceOrbiter mission is used. The method is again verified using standard Middlebury stereo dataset with ground truth. From experiments,it has been observed that using optimization technique for triplets, the total number of correct matches has increased by 5-10% incomparison to direct methods. The method particularly gives good results at smooth regions, where dynamic programming and block-matching gives limited accuracy.

1. INTRODUCTION

Stereo image matching is a classical research problem for thecomputer vision and the phtogrammetry community. These stereomatching methods are broadly classified as spare and dense stereomatching. Sparse matching methods use distinctive features inthe image and match them, whereas as in dense matching algo-rithms the disparity is determined at each of the pixel location.Dense matching are again classified as Global and Local meth-ods. Local methods take into consideration of neighboring pixels,and defines a window or pixel based matching approach, whereasglobal methods minimize a global energy using an initial dispar-ity obtained from Local methods. DEM generation is an impor-tant process from satellite/aerial photogrammetry point of view.To generate DEM one of the steps is stereo matching. Commer-cially available softwares use least square matching. In additionto that, a lot of human effort and RPC parameters are requiredto correct the generated model. Initially, the software’s were ca-pable to generate DEM from pair of stereo images, but recentlysome of the software have the capability to handle triplets (SAT-PP, LPS). The exact workflow for DEM generation, which thesesoftware uses are are not publicly available for complete under-standing. Neither a brief description is given about removing in-correct matches nor the exact rules to fill the holes is provided.Therefore, at various places, generated DEM from these softwarerequires a lot of human effort for correction and hence these soft-ware are semi automatic. In the case of planetary images, we donot have very good ground control, due to which DEM generationusing conventional software is more challenging for planetary im-ages. Therefore in the present work an automatic stereo match-ing approach is used for disparity map generation, using tripletsand then automatically reject incorrect matches rigidly and finallymapping it to the ground using control parameters.

∗Corresponding author

2. RELATED WORK

Scanline Optimization (SO) is one of the global matching ap-proaches. The data structure that is used to store cost for scanlineoptimization is called as Disparity Space Image (DSI). This rep-resentation of the cost as of DSI finds its roots in the work byMarr and Poggio (Marr and Poggio, 1976), where they used co-operative algorithm to find the disparity. Later Cochran and G.Medioni (Cochran and Medioni, 1992) used same structure andcalled it as correlation array and used feature and area basedapproach to find the disparity. Intille and Bobick (Intille andBobick, 1994b) (Intille and Bobick, 1994a) used dynamic pro-gramming to find the disparity using DSI. They also gave math-ematical representation to DSI. After that many SO algorithmsare proposed based on DP. Heiko Hirschmuller (Hirschmuller,2008) (Hirschmuller, 2005) introduced a new method as SemiGlobal Matching which has been used widely for satellite andaerial images that isalso a SO method. SGM uses the DSI forthe entire image to aggregate the cost for a pixel. All the abovementioned methods use the stereo pair for experiment. Recentlytriplets are used using these methods. Heinrichs (Heinrichs etal., 2007) et al. used SGM method for triplets to determine thedisparity. Mozerov (Mozerov et al., 2009), introduced a globaloptimization based method using DSI for triplets to find the dis-parity. Based on above proposed method we use DSI, based ap-proach for tri-stereo matching. To constrain the search we aredefining some control points as initial matches as in (Intille andBobick, 1994b) (Intille and Bobick, 1994a) (Torr and Criminisi,2002) (Kim et al., 2005), which guide the disparity curve ob-tained by DP. Intille and Bobick used control points based onsome heuristics; Torr and Criminisi used edge and corner pointsand Kim et al. proposed a method of oriented spatial filters fordetermining control points. It has been observed that as globalminimum has the significant role in WTA strategy, but local min-ima also finds its importance to give correct disparity (Dima andLacroix, May 2002). Therefore, the proposed approach use initialpoints from global minimum and use weighted technique to fur-

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014ISPRS Technical Commission III Symposium, 5 – 7 September 2014, Zurich, Switzerland

This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-3-33-2014 33

ther use local minima to increase the number of correct disparitypoints to be used as control points.

3. CONTRIBUTION OF PRESENT WORK

The proposed work is inspired by the advantages of scan linestereo matching methods. The basic methods that fall in thiscategory are scan line optimization (SO), dynamic programming(DP) and semi-global-matching (SGM). SO and DP methods arevery similar. The basic problem is these methods is that thesemethods assumes monotonicity or ordering of disparity, basedon this, in decision making process there are only three possi-ble moves. If there is a large change in disparity, these methodsfails as the best path search chooses an incorrect move. Once anincorrect path is selected it takes several moves to come back tothecorrectpath. And all the disparities generated in between thesetwo moves are incorrect and gives the wrong result. In proposedwork based we are also searching for the best path, but this isconstrained by some control points. There are hard constraintsand soft constraints. The best path has to follow these controlpoints as used in (Torr and Criminisi, 2002) and constraints. Butthese control points are not corner points. The results are muchsimilar to SGM method but in proposed method the 8 or 16 direc-tion aggregation is not required as in SGM. If the slope is varyingvery rapidly or, there are uniform regions in the image the methodgives good results as generally at these locations global minimafails, and we have to consider the local minima.

4. RELATED THEORY

4.1 Cost Function

Cost function is measure of similarity between two images. HereZero Mean Sum of Squared Difference (ZSSD) is chosen as costfunction C, so as to compensate for changes in illumination (ra-diometric changes). Mathematically the relation is given as-

Clr(x, y) =∑j∈w

{(I1(x, y)− I1)− (I2(x, y + j)− I2)}2 (1)

As we have triplet of images we have 3 cost functions for three setof images Left-Center CLC , Center-Right CCR and Left-RightCLR. In present study we used CCR and CCL because centerview is common in this case. To consider the third cost functionCLR, we need to transform it with respect to central view in orderto compare. This transformation requires accurate estimate ofdepth (see equation 10), which itself is in error. Therefore this isconsidered for future work. Now using these cost function DSI isgenerated.

4.2 Disparity Space Image

DSI is a data structure. It is used to store various values of thecost function, for various pixels of a scan line, in a grid format.Figure 2 shows the DSI representation of the cost function theplot is Left scan line Vs right scan line and the disparity in diag-onally. As the disparity is limited to 0 to dmax, not all the pixelsare required to be mapped in this representation. The compressedversion of DSI is shown in figure 2, which is plotted as left scan-line Vs disparity. Disparity Space Image (DSI) representation ofthe cost function is one of the oldest ways to represent the costfunction as the data structure. It finds its roots in the work byMarr and Poggio long back in 1976. It is used to store various val-ues of the cost function for a particular scan-line in a grid formatand used to determine the disparity, using different approaches.

Figure 1: (A) Cost Function, (B) Color coded cost funciton rep-resentation of (A), (C) Disparity Space Image , (D) CompresedDisparity Space Image (arrow shows location of cost functionrepresented by (B))

(b) Disparity Space Image for Il Ir matching

(a) Scan line disparity

representation

(c) Compressed DSI for Il Ir matching

Il1

Il2

Il3

Il4

Il5

Il6

Ir1 Ir2 Ir3 Ir4 Ir5 Ir6

d2 d1 d0

0 5 7 9 4 3

3 2 4 6 1 0

5 0 2 4 1 2

7 2 0 2 3 4

8 3 1 1 4 5

2 3 5 7 2 1

1 6 8 10 5 4 Il

1 4 6 8 9 3 Ir1 Ir2 Ir3 Ir4 Ir5 Ir6

Ir

0 - 1 1 1 0

0 1 1 1 - 0

Disparity Ir-Il

Il1 Il2 Il3 Il4 Il5 Il6

Disparity Il-Ir

Ir1 Ir2 Ir3 Ir4 Ir5 Ir6

0 2 2 2 4 1

3 0 0 1 2

5 2 1 7

d0

d1

d2

Column global minimum

Row global minimum

CPl CPr

PMl

PMr D

DSI

CPsl CPsr

PMls

apm

(b) (a)

CPl CPr (Global Minimum)

CP–3 CP–2 CP–1 PMj CP1 CP2 CP3 co

st

PMl PMr (Local Minima)

-∆ε

+∆ε

j d

isp

arit

y

Figure 2: (a) Two scan lines Il{1, 4, 6, 8, 9, 3} andIr{1, 6, 8, 10, 5, 4} of image pair Il and Ir (b) DSIlr forIl −→ Ir matching (c) compressed DSIc

lr

(Heinrichs et al., 2007) used similar representation and solvedthe stereo matching using the Cooperative Algorithm for randomdot stereogram. The same structure has been termed as the Un-compressed Correlation Array in approach as in by (Cochran andMedioni, 1992). They used a composite approach of the featureand area based matching using DSI. (Intille and Bobick, 1994a,Intille and Bobick, 1994b) used the Dynamic Programming(DP)to determine disparity using DSI. They also explained the forma-tion and mathematical representation of DSI in a very lucid man-ner. Later on DSI has been extensively used involving DynamicProgramming approach for the stereo matching. For a stereo pair(Il− Ir) images ; in ideal case the size of DSI is (n×n), wherethe cost function C(i0, j, d) is given as |Il(i0, j)−Ir(i0, j +d)|.Mathematically DSI(i, j, d) is given as −

DSI(i0, j, d) =C(i0, j1, d1) C(i0, j2, d1) . . . C(i0, jn, d1)C(i0, j1, d2) C(i0, j2, d2) . . . C(i0, jn, d2)

: :. . . :

C(i0, j1, dn) C(i0, j2, dn) . . . C(i0, jn, dn)

(2)

In case of DSI the search range is for entire scan line, hence dranges from 1 to dn, but not all the values in DSI are used to de-termine the depth, as the search is limited by maximum disparitydm. Therefore it is expensive to store all of the cost values, fromcomputation as well as memory usage point of view. Thereforea compressed representation of DSI is used as cDSI . Thus thesize of cDSI is (dm × n). Mathematically cDSI is given as −

cDSI(i0, j, d) =C(i0, j1, d1) . . . C(i0, jn−1, dn−1) C(i0, jn, dn)C(i0, j1, d2) . . . C(i0, jn−1, dn) ×

: . . . × ×C(i0, j1, dm) . . . × ×

(3)

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014ISPRS Technical Commission III Symposium, 5 – 7 September 2014, Zurich, Switzerland

This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-3-33-2014 34

Figure 2 shows formation of DSI and cDSI from a scan linesof the stereo image pair (Il − Ir).

5. CONSTRAINTS

5.1 Control Points

In proposed method the best path is constrained by control points,these are global minimum and local minima.

5.1.1 Global minimum Let’s consider cost function C. As inWTA strategy the global argument of minima of the cost functionis defined as the disparity, mathematically which is given as -

D(x, y) = argmind

C(x, y, d) (4)

5.1.2 Local Minima It has been observed that due to largevariation in disparity and noise, many a time global minimum endwith erronous match but local minima gives correct match, henceapart from global minimum local minimum are used as controlpoints.All the control points are first filterd for consistency check. IFthey are consistence in forward and reverse matching and left-centre-right view then only considered further else rejected. Thesecoinsistency check is given as -

5.2 L R consistency in DSI

The argument of the global minima of columns in DSI is consid-ered as disparity. From above relation in terms of DSI it can beexpressed as -

Dv(x, y) = argmind

DSI(x, y, d) (5)

Due to this representation of the cost function in DSI, we haveargument of the global minima of rows also as the disparity, (fig-ure 2). So the next relation for disparity is give as-

Dh(x, y) = argminy

DSI(x, y, d) (6)

Physically the minima in vertical direction can be understood assearching LR matching and in horizontal direction its RL match-ing. Hence from above equations the LRC consistency chek givesfor the disparity to be accepted.

Dv(x, y) = Dv(x, y) (7)

5.3 CL CR consistency in DSI

Now as we have three views so the equation 7 can be expressedin terms of left-centre and centre-right as-

Dcl(x, y) = argmind

DSIcl(x, y, d) (8)

Dcr(x, y) = argmind

DSIcr(x, y, d) (9)

Equation 8 and 9 gives two disparity map from two differentviews. Therefore from two oservation we have two disparity map,the erreo between these two is fiven as −

Dcl(x, y) = Dcr(x, y) (10)

Table 1: TMC Data-set details

Major Location Mode Lines Date ofFeature acquisition

Aft 188828 2008 11 16Goldschmidt B 70.6◦N 6.7◦W Nadir 188708 2008 11 16

Fore 188608 2008 11 16Aft 161400 2009 04 18

Gassendi G 16.75◦S 44.67◦W Nadir 161396 2009 04 18Fore 161400 2009 04 18Aft 136164 2009 04 18

Marius D 11.41◦N 45.07◦W Nadir 136164 2009 04 18Fore 136164 2009 04 18

Table 2: LOLA Data set details

Crater Name of file resolutionMarius DLDEM 1024 00N 15N 300 330 (1/1024)◦/pixel

Gassendi G LDEM 1024 30S 15S 300 330 (1/1024)◦/pixelGoldschmidt LDEM 1024 60N 75N 330 360 B (1/1024)◦/pixel

6. DATA-SET

To generate 3D model the triplet stereo pairs are used and to ver-ify the results lidar data is used. Tri-stereo data used is from CH-1 missions Terrain Mapping Camera (TMC), which is acquiredfrom ISSDC (http://www.issdc. gov.in/CHBrowse/index.jsp). Thedetails of data acquired are detailed in table 1. The lidar datais acquired from Lunar Orbiter Laser Altimeter (LOLA) sensorof LRO mission (http://ode.rsl.wustl.edu/moon/). The details ofLOLA dataset is given in table 3. To verify the results on planer-ary images some more experiments are carried out on Middleburystereo dataset. te details are given in table 3. The dataset used isshown in figure 3.

7. PROPOSED WORK-FLOW

The images from CH1 TMC data set are used for experimenta-tion, the actual size is very large (1, 00, 000×4000 pixels) there-fore a small portion (2000 × 2000 pixels) near known feature isselected. Taking Nadir as reference view, corresponding imagesfrom Fore and Aft are obtained. Fore and Aft are, then resam-pled for quasi-epipolarity. Next, based on the outline of stereomatching (Scharstein and Szeliski, 2002) a cost function is de-fined. Using this cost function for a scan line, two DSI are gener-ated, one for Aft-Nadir view and other for Nadir-Fore view. Sub-sequently using proposed confidence measure incorrect matchesare rejected. Finally the LOLA dataset of LOC missoin is usedvalidation of the results thus obtained.

7.1 Quasi-Epipolar Resampling

TMC acquires the images in along track direction. In such case ofalong track stereo acquisition, epiploar resampling is basically amethod to make ‘across track’ parallax to ‘zero’ so that for deter-mining depth the corresponding pixel can be searched in ‘alongtrack’ direction only. Here a graphical approach is used to estab-lish the relations between different views. In order to get the in-sight of the geometric relations of stereo pair; first the images arematched using Scale Invariant Feature Transform (SIFT) (Lowe,2004). The matched SIFT features may have outliers, and for thisreason, to reject outliers Random Sample Consensus (RANSAC)algorithm (Fischler and Bolles, 1981) is applied. Based on theprocedure as detailed in (Bhalerao et al., 2013) the parameters

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014ISPRS Technical Commission III Symposium, 5 – 7 September 2014, Zurich, Switzerland

This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-3-33-2014 35

Table 3: Middleburry Stereo Dataset

Feature Name of file SizeIndoor lampshade1 1300× 1110

Figure 3: Input A-N-F view triplet images for Marius D craterand DEM, yellow line shows the scan line for which experimentis performed whose results are shown in figure 4

are obtained for epipolar resampling. Using these parameters Aft(and Fore) to are resampled to follow epiploar geometry w.r.t.Nadir.

7.2 DSI Generation

Using the above cost function as in equation DSI is generated.

7.3 Best Path Search

The proposed method is a global matching method that defines anenergy function and a penalty term. The initial conditions usedare

1. only a single local minima can be the correct disparity fora column or row. (This condition satisfies the unique matchproperty of stereo matching).

2. The translation between two disparities should be less than athreshold. (This condition satisfies the smoothness criterionof stereo matching).

The initial traversing direction of the best path can be any eithera row, or a column of the DSI, here path traversed is from thecolumn to column. This optimized path at a column first look forglobal minima if there is a global minima present at the location;this is the solution. If global minima is absent, it looks for globalminima of adjacent columns, one to left and one to right Cml andCmr or moves several columns if could not find it in immediateadjacent column. Using all local minima of the current column(Pm0), and neighbouring global minima Cml, Cmr; an optimiza-tion rule for considering a local minimum as desired solution isformulated. Weightages are given to all local minima inverselyproportional to their location from the selected global minimum.The best path is then selected based on highest weight from Cml,Cmr . The path is then traversed for the next column. The addi-tional rule for the path is to honour all the (Cm) but may omit alocal minimum from the set (Pm0) of a particular column. Theresult is thus a smooth profile.

Table 4: Correct matches

Dataset Scan line Correct MatchesWTA DP Proposed

Goldsscmidt B 1000 1685 1735 17831500 1227 1685 1840

Gassendi G 1000 1321 1670 17961500 1053 1227 1556

Marius D 1000 1143 1685 18401500 1242 1670 1796

Lamp 150 843 956 1053150 873 986 1071

Total pixels in scan line = 2150 pixels, for CH1 datasetTotal pixels in scan line = 1110 pixels, for Middleburry dataset

8. EXPERIMENTAL RESULTS

Using data from CH-1 mission TMC data-set the experiment isperformed for three known features as indicated in table 1. Ta-ble 4 gives the result obtained for one of the craters. For accuracyassessment obtained results are compared with the LOLA dataset. LOLA is laser altimetry dataset. It provides the location andelevation. The data from CH1 triplets only provides location anddisparity which can not be directly compared unless precisely lo-cated for location accuracy. Therefore, first the TMC data is cor-rected with Clementine data for planimetric error. Later basedon Clementine data the LOLA data is obtained for the same lo-cation. The resolution of LOLA is 29 meters and resolution ofTMC is 5 meters. Therefore, different profiles are compared foraccuracy assessment. It has been observed that a large number ofincorrect matches are removed using the defined criteria. Figure4 shows one of the results of the experimentation for Marius Dcrater and Middlebury image . The obtained result is comparedwith LOLA after transforming it by scaling and offsetting, it hasbeen observed more that 90% of points obtained are matched cor-rectly. The experiment is also performed on Middlebury dataset.As the proposed method, is based on scan-line hence to com-pare the results from proposed methodology has been comparedwith dynamic programming algorithm, along with the basic blockmatching. It has been observed; dynamic programming fails ifthe disparity is very large as once it choose an incorrect disparityit’s not easy for the algorithm to come hence to correct the path.The method gives the smoother result as compared to both thebasic block matching and dynamic programming. In the case ofcrater, the DP doesn’t gives good result at all.

Figure 4: Disparities generated by (A) Blockmatching (B) Dy-namic Programming (C) proposed method for Marius D crater

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014ISPRS Technical Commission III Symposium, 5 – 7 September 2014, Zurich, Switzerland

This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-3-33-2014 36

9. CONCLUSION AND FUTURE WORK

In this paper we used a method for triplet matching, which isbased on standard stereo matching techniques. We proposed amethod which uses information from triplet of images using aDSI to define rules to find out incorrect matches. In future workthe distribution of the matches will be considered and qualitativeanalysis will be done using ground truth from other sensor. Inpresent work the complete set DEM for a region is not investi-gated properly, instead profile is only investigated which will becarried in future work.

REFERENCES

Bhalerao, R., Gedam, S. and Almansa, A., 2013. Fast epipolar re-sampling of trinocular linear scanners images using chandrayaan-1 tmc dataset. In: 2nd IEEE International Conference on ImageInformation Processing, Shimla, India, pp. 56–60.

Cochran, S. and Medioni, G., 1992. 3-d surface description frombinocular stereo. IEEE Trans. PAMI 14(10), pp. 981–994.

Dima, C. and Lacroix, S., May 2002. Using multiple disparityhypotheses for improved indoor stereo. In: IEEE InternationalConference on Robotics and Automation), Vol. 4, IEEE, Wash-ington, DC, USA, pp. 3347–3353.

Fischler, M. A. and Bolles, R. C., 1981. Random sample consen-sus: a paradigm for model fitting with applications to image anal-ysis and automated cartography. Commun. ACM 24(6), pp. 381–395.

Heinrichs, M., Rodehorst, V. and Hellwich, O., 2007. EfficientSemi-Global Matching for Trinocular Stereo. In: Photogrammet-ric Image Anlysis, Vol. 36, Munich, Germany.

Hirschmuller, H., 2005. Accurate and Efficient Stereo Processingby Semi-Global Matching and Mutual Information. In: CVPR,Vol. 2, San Diego,USA, pp. 807–814.

Hirschmuller, H., 2008. Stereo processing by semiglobal match-ing and mutual information. PAMI 30(2), pp. 328–41.

Intille, S. and Bobick, A., 1994a. Incorporating intensity edgesin the recovery of occlusion regions. In: ICPR, Vol. 1, Jerusalem,Israel, pp. 674–677.

Intille, S. S. and Bobick, A. F., 1994b. Disparity-space imagesand large occlusion stereo. In: ECCV, LNCS, Vol. 801, SpringerBerlin Heidelberg, pp. 179–186.

Kim, J. C., Lee, K. M., Choi, B. T. and Lee, S. U., 2005. ADense Stereo Matching Using Two-Pass Dynamic Programmingwith Generalized Ground Control Points. In: CVPR 05, Vol. 2,Ieee, pp. 1075–1082.

Lowe, D. G., 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), pp. 91–110.

Marr, D. and Poggio, T., 1976. Cooperative computation of stereodisparity. Science 194(4262), pp. 283–287.

Mozerov, M., Gonzlez, J., Roca, F. X. and Villanueva, J. J., 2009.Trinocular stereo matching with copmposite disparity space im-age. In: ICIP, Cairo, Egypt, pp. 2089–2092.

Scharstein, D. and Szeliski, R., 2002. A taxonomy and evalua-tion of dense two-frame stereo correspondence algorithms. Int. J.Comput. Vision 47(1-3), pp. 7–42.

Torr, P. and Criminisi, A., 2002. Dense Stereo Using Pivoted Dy-namic. In: British Machine Vision Conference, pp. 39.1–39.10.

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014ISPRS Technical Commission III Symposium, 5 – 7 September 2014, Zurich, Switzerland

This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-3-33-2014 37