Cross-trees, edge and superpixel priors-based cost aggregationfor stereo matching

Feiyang Cheng a, Hong Zhang a, Mingui Sun b, Ding Yuan a,n

a Image Research Center, Beihang University, Xueyuan Rd., Haiidan Dist., 100191, Beijing, Chinab Department of Neurosurgery, University of Pittsburgh, Suite B-400, 200 Lothrop Street, PA15213, Pittsburgh, USA

a r t i c l e i n f o

Article history:Received 9 April 2014Received in revised form6 November 2014Accepted 5 January 2015Available online 14 January 2015

Keywords:Stereo matchingCost aggregationImage filteringSpanning trees

a b s t r a c t

In this paper, we propose a novel cross-trees structure to perform the non-local cost aggregation strategy,and the cross-trees structure consists of a horizontal-tree and a vertical-tree. Compared to other spanningtrees, the significant superiorities of the cross-trees are that the trees' constructions are efficient and thetrees are exactly unique since the constructions are independent on any local or global property of theimage itself. Additionally, two different priors: edge prior and superpixel prior, are proposed to tackle thefalse cost aggregations which cross the depth boundaries. Hence, our method contains two differentalgorithms in terms of cross-treesþprior. By traversing the two crossed trees successively, a fast non-localcost aggregation algorithm is performed twice to compute the aggregated cost volume. Performanceevaluation on the 27 Middlebury data sets shows that both our algorithms outperform the other two tree-based non-local methods, namely minimum spanning tree (MST) and segment-tree (ST).

& 2015 Elsevier Ltd. All rights reserved.

1. Introduction

Dense two-frame stereo matching has been extensively inves-tigated for decades as a traditional low-level vision task, since it iscrucial for many applications such as 3D reconstruction [1,2],image-based rendering [3,4] and anonymous driving [5]. Accord-ing to the analysis and taxonomy scheme proposed in [6], stereomatching algorithms can be categorized into two groups: localalgorithms and global algorithms. Stereo matching algorithms areoften implemented following a subset of the four steps or all:

1. Cost function/cost volume estimation.2. Cost aggregation within a support region.3. Disparity computation/optimization.4. Disparity refinement.

Global algorithms usually make explicit smoothness assumptions,and minimize a predefined energy function to obtain optimal results[7–9]. Despite the reliable matching results obtained, global algo-rithms are often time-consuming. All local algorithms compute thematching cost (step 1) firstly and then perform the cost aggregation

(step 2) to get a locally optimized cost volume [6,10–14]. We mainlyfocus on efficient and effective local and non-local methods in thispaper, and the readers are referred to a recent study for a compre-hensive study of the global methods [15].

To find a correspondence (x, x’), the problem of the localmethods can be concluded as a comparison of the similarity oftwo local patches which around x and x' respectively. The similarityof the two patches is computed by aggregating the costs of thepixels within the patches. Hence, the cost aggregation (step 2)procedure has important impacts on the accuracy and the efficiencyof a local algorithm. The cost aggregation of a pixel in traditionallocal algorithms is usually performed by averaging the costs of thepixel itself and all its neighboring pixels. Here, the implicit assump-tion is that all the pixels which lie in a special local support regionhave similar disparities, as shown in Fig. 1(a). Such local methodssuffer from well known ”edge fatten” effect once the local supportregions cover the depth boundaries. The problem can be explainedin the context of image filtering. For instance, the box filter alwaysblurs the edges of an image during the image denoising procedure.Hence, the problem of the cost aggregation step is how to chooseoptimal local support regions for each pixel. Various researcheshave been conducted to estimate optimal support regions forthe cost aggregation, such as various window-based methods[10,11] and adaptive support weights (ASW) methods (also knownas local filtering-based methods) [12–14] which have state of theart performance in the last years. However, the selected support

Contents lists available at ScienceDirect

journal homepage: www.elsevier.com/locate/pr

Pattern Recognition

http://dx.doi.org/10.1016/j.patcog.2015.01.0020031-3203/& 2015 Elsevier Ltd. All rights reserved.

n Corresponding author. Tel.:/fax: þ86 10 82338991.E-mail addresses: chfybuaa@gmail.com (F. Cheng),

dmrzhang@buaa.edu.cn (H. Zhang), drsun@pitt.edu (M. Sun),dyuan@buaa.edu.cn (D. Yuan).

Pattern Recognition 48 (2015) 2269–2278

regions of the ASW methods are often limited in a pre-definedwindow of fixed size. Due to this reason, this kind of methodscannot work well for the stereo images with large planar surfaces.

Recently, Yang proposed a non-local cost aggregation methodbased on a MST [16]. As shown in Fig. 1(b), a pixel is able to getsupport from all the other pixels of the image through uniquepaths on the tree structure. Different from aforementioned variouswindow-based methods and ASW methods, the cost aggregationwas performed over the whole image for each pixel to establishnon-local optimized results. Xing proposed a segment-tree struc-ture to perform the non-local cost aggregation strategy [17]. Thesework proved that the non-local cost aggregation methods outper-form the local ones much more.

Hence, this paper mainly focuses on the non-local cost aggre-gation procedure by comparing different tree construction tech-niques [16,17]. Section 2 is an overview of the previous work of thecost aggregation procedure. We briefly review the workflow of thenon-local framework and the non-local cost aggregation algorithmin Section 3.1 and then introduce the cross-trees and the two priorsin Section 3.2. A discussion of the strategies for constructingdifferent tree structures is also provided in Section 3.2. Experi-mental results and performance evaluations are shown in Section 4.A detailed analysis of the short points of the tree-based non-localcost aggregation is given in Section 5. Finally, we draw the conclu-sions and discuss the future work in Section 6.

2. Previous work

The cost aggregation procedure can be considered as two sub-problems: (1) how to estimate the optimal support regions; (2) howto aggregate the matching costs of the pixels within the estimatedsupport regions (usually, in terms of support weights). We reviewthe related work in this section according to the two sub-problems above.

2.1. Various window-based methods

Most early local methods aimed at estimating various windowsfor different pixels, from adaptive [10,11] to shiftable windows[18]. The optimal windows were often selected based on certainlocal properties to avoid covering disparity discontinuities. Fusielloet al. developed a multiple window approach by performing costaggregation in nine different window models and chose thewindow with the smallest aggregated cost as the optimal window.However, limited window models are not sufficient to representsupport regions with arbitrary shapes and sizes. Some researchers

proposed to use cross-based structures to represent various supportregions and developed several competitive algorithms [19,20].

Most of these methods considered the sub-problem (1) only,and gave all the pixels the same weight. It means that thesemethods were developed based on a simple smoothness assump-tion that all the pixels within the same support region have aconstant disparity. Such methods may result in over-smoothdisparity slices within smooth curved surfaces.

2.2. Adaptive support weights methods

One main resolution is the adaptive support weights (ASW)strategy, in which weighted supports decide whether the neigh-boring pixels contribute more or less to the center pixel [12,13].The support weights that adapt according to similarity and pro-ximity to the central pixel of the predefined large support windowactually control the real aggregation region and power. However,computing support weights iteratively for each central pixel is atime-consuming task. Many researchers indicated that such strat-egy can be approximately re-implemented by using local filtersuch as bilateral filter [21] and guided filter [14,22]. In this way,both the accuracy and the efficiency of the ASW algorithms havebeen improved. Hereto, the idea that the edge-preserving filterscan be employed to aggregate the matching costs and to preservethe depth edges simultaneously becomes clear gradually. How-ever, all the local filtering-based methods still establish locallyoptimized results within predefined windows which have a fixedsize. Detailed comparisons and discussions of the local filtering-based methods can be found in two recent reviews [23,24].

2.3. Tree-based non-local methods

As mentioned above, a MST-based non-local cost aggregationmethod was proposed recently [16]. In the same non-local frame-work, Mei et al. proposed to employ a ST instead of a MST tooptimize the non-local cost aggregation procedure [17]. Concep-tually, they segmented the image at first and then constructed asub-tree (i.e., a sub-MST) for each segment. Finally, a ST wasconstructed by linking these sub-trees for the non-local costaggregation. The main idea, using the segment prior to avoidconnecting two pixels which locate at the different sides of asegment boundary (i.e., potential depth boundaries), is similarwith other segment-based stereo matching methods [25,26]. Boththe MST method and the ST method outperform the local methodsin aggregation accuracy [14,21].

Actually, constructing a tree or a graph to improve the optimi-zation procedure is not new in many global methods such asDynamic Programming (DP) and Loop Belief Propagation (LBP).Veksler firstly employed DP on a tree instead of a scanline toenforce vertical consistency to establish truly global optimizationresults [27]. Cheng Lei et al. improved the tree-based DP methodby using a novel tree structure which they called region-tree [28].Zitnick et al. formulated a new MRF for global optimization byover-segmenting the images [4].

In conclusion, for both the non-local cost aggregation and theglobal optimization which are based on a tree structure, thecritical problem is the construction of the tree. In this paper, wefocus on the non-local cost aggregation methods only. Hence, wedescribe the problem in the context of the non-local cost aggrega-tion. To construct a tree from a graph, the important edges need tobe preserved and the edges crossing the depth boundaries must beremoved. By an important edge, we mean an edge whose twonodes are more likely to have the same disparity. Local criterions(i.e., color difference [16], distance [27], etc.), non-local properties(i.e., segmentation [17], over-segmentation [4], etc.) or a jointversion of the two can be used to decide whether an edge should

Fig. 1. Cost aggregation of the center pixel. (a) Local support region within thesquare frame: the center pixel gets support only from its neighboring pixels.(b) Tree: an unique path can be found between the center pixel and each pixel ofthe image. The dot line denotes that many pixels on the path are not shown here.

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–22782270

be selected to construct the tree. If an edge is selected, its twonodes are connected directly and they can get support from eachother during the cost aggregation procedure. Moreover, two nodeswhich are not connected directly by an edge can also get supportfrom each other through a unique path on the tree; see details inFig. 1(b). Once the tree is constructed, the support weight betweena pair of pixels is decided by the distance between them onthe tree.

3. Algorithm

Our work is directly motivated by the two non-local costaggregation methods recently [16,17] as well as the localfiltering-based methods [14,21] and a simple tree-based DP opti-mization method [29]. We employ two crossed trees, namelyhorizontal-tree and vertical-tree, and perform the non-local costaggregation algorithm proposed in [16] successively to get theaggregated cost volume. For convenience, we use the cross-trees todenote the two crossed trees in the residual part of this paper. Theconstructions of the cross-trees are simple, efficient and especiallyindependent on any local or non-local properties. We makeexplicit smoothness assumption in our method by truncating thedistance (i.e., color difference) between two neighboring pixels.However, performing the non-local cost aggregation on such treesdirectly will blur the depth boundaries. Hence, two differentpriors, namely edge prior and superpixel prior, are proposed toprevent the false smoothing at the depth boundaries. Based onthe two different priors, two different algorithms are proposedrespectively and each one has competitive performance comparedto other non-local cost aggregation methods. By performing thenon-local cost aggregation strategy on different trees, MST, ST andour method all have competitive rankings on the Middleburywebsite compared to the state of the art local cost aggregationmethods.

In this section, we review the non-local framework at first andthen propose our methods.

3.1. Non-local cost aggregation

The non-local framework for initial estimation of disparitymaps can be concluded in four steps as follows:

1. Cost function/cost volume estimation.2. Tree construction.3. Non-local cost aggregation on the constructed tree.4. Disparity estimation (WTA).

We mainly contribute to step 2 in this work, and Yang hasproposed an efficient algorithm for step 3 which we will briefreview in the remaining part of this section [16]. Steps 1 and 4 arecommonly used techniques in the context of stereo matching.

In the non-local framework, the reference image I is repre-sented as a 4-connected and undirected graph G¼ ðV ; EÞ. Here, apixel in I corresponds to a node in V and a pair of neighboringpixels are connected by an edge in E with the edge weightcomputed by using the following equation:

wðs; rÞ ¼wðr; sÞ ¼ j Is� Ir j ð1ÞHere, s and r are two neighboring pixels in I, Is and Ir represent

the intensity of s and r respectively. Veksler proposed a MIDDTtree by considering distance transform additionally to estimate theedge weights [27]. However, using sophisticated similarity costscontribute to improving the tree construction while at the price ofincreasing computational cost. Note that using various matchingcost functions to compute the edge weight is not appropriate here

[30–32]. When we compare two pixels within the stereo imagesrespectively, the assumption is that the support regions of the twopixels are also similar if the two pixels are the same point in 3Dspace. In this case, we can use various strategies to compare thetwo pixels by utilizing their neighboring pixels within the localsupport regions. It is totally reasonable. However, when wecompare the two neighboring pixels in the same image to decidewhether they can support each other, we assume that the twoneighboring pixels which have similar color are likely to havesimilar disparity. Here, there is no assumption that the two pixels'neighboring pixels are also similar because the image textures arenot repeated everywhere.

For the color images, the edge weight is the maximum wðs; rÞvalue estimated from three R-G-B channels. Using the max colordifference on each channel can guarantee that the two neighbor-ing pixels which have similar intensity on all the three channelsgive high support weights to each other. Using the max ratherthan other possibilities, such as the mean, is an empirical choiceaccording to the experiments.

A subset T of E will be extracted to construct a tree where aunique path Pðp; qÞ can be find between any pair of pixels ðp; qÞ in I.The distance Dðp; qÞ ¼Dðq; pÞ between p and q is defined as thesum of the edge weights along Pðp; qÞ. Thus the weighted supportSðp; qÞ which p and q give to each other is defined as (2) with auser-specified parameter σ:

Sðp; qÞ ¼ Sðq; pÞ ¼ exp �Dðp; qÞσ

� �ð2Þ

For the disparity level d, let Cd(p) denote the matching cost of p.It is straightforward to compute the final aggregated cost CA

dðpÞ of pby using the following equation:

CAdðpÞ ¼

XqA I

Sðp; qÞCdðqÞ ð3Þ

During the non-local cost aggregation, p gets weighted supportfrom all the pixels in I. It is different from the local filtering-basedmethods in which p gets weighted support from its neighboringpixels within a local support region only. Usually, the local supportregion is a predefined square window.

Computing Cd(p) for each pixel p by using (3) iteratively isreally time-consuming. Yang proved that the non-local costaggregation can be accomplished in exactly linear time [16]. Byperforming a leaf-to-root and a root-to-leaf cost aggregation onthe tree structure successively, the final aggregated cost for eachnode can be estimated (see Fig. 2 for details). In the leaf-to-rootpass, the intermediate aggregated cost CA↑

d ðqÞ of each node q can becomputed by using the following equation:

CA↑d ðqÞ ¼ CdðqÞþ

XParðq0 Þ ¼ q

Sðq; q0ÞCA↑d ðq0Þ ð4Þ

Here, Parðq0Þ denotes the collection of all the children nodes of q.Note that the final cost aggregation CA

dðpÞ equals to CA↑d ðpÞ for the

Fig. 2. Two-pass non-local cost aggregation. Here, the pixel p is treated as the rootnode of the tree. (a) Leaf-to-root pass: The intermediate aggregated cost of eachpixel is computed in a straightforward way and (b) root-to-leaf pass: the finalaggregated cost for each pixel can be computed by using the results of (a).

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–2278 2271

root node p in Fig. 2. Then for each leaf node q, the final aggregatedcost CA

dðqÞ can be computed in the root-to-leaf pass as follows:

CAdðqÞ ¼ SðParðqÞ; qÞ � CA

dðParðqÞÞþ½1�S2ðq; ParðqÞÞ� � CA↑d ðqÞ ð5Þ

For each disparity level, the computational complexity of the costaggregation is O(n), where n is the number of the pixels of thereference image. Hence, the total computational complexity of thenon-local cost aggregation algorithm is Oðn � lÞ, where l is the disparityrange. To refine the established disparity maps, a non-local post-processing technique based on the tree is also proposed in [16].

3.2. Cross-trees and priors

Our work exactly follows the non-local framework, and thesignificant difference is that we employ a different cross-treesstructure. Moreover, two different algorithms are proposed due tothe two different priors (edge prior and superpixel prior) incorpo-rated into the non-local framework. The weights of the edges in Gare also redefined according to the pixel similarity and the priors.

3.2.1. Cross-trees for non-local cost aggregationThe problem comes back to the construction of the tree. Yang

constructed a MST which has the minimum sum of the edge weightsamong all the spanning trees of G [16]. The intuition is that an edge isless likely to cross the depth boundaries if its two nodes (twoneighboring pixels in I) have higher pixel similarity. For the MSTmethod, there are two problems need to be emphasized on: (1) thetree structure is not unique since there are a lot of edges which havethe same weight; (2) the edges which lie in highly textured regionsare less likely to be selected though neighboring color variationenhances the pixels' distinction. Hence, only employing simple localproperty is not enough for constructing a unique optimal tree. Non-local segment prior was additionally employed in [17] to enhance thetree construction procedure. Though accuracy improvement wasachieved, it involved the segmentation task which is especially hardto tackle in some cases. Despite the image segmentation task itself,constructing a sub-MST in each segment may still suffer from theproblems of the MST construction if the segments are large.

Motivated by the problems discussed above, our tree construc-tion aims at being independent on the reference image (i.e., notrelated to pixel similarity), but consistent with the fact thatdisparity maps are mostly spatially smooth. Hence, we makeexplicit smoothness assumption as global methods during thetree construction procedure, and then incorporate priors into thenon-local framework to preserve the depth boundaries.

Instead of the pixel similarity, the proximity is referred only toconstruct the cross-trees. The structures of the cross-trees areshown in Fig. 3. We perform the non-local cost aggregationalgorithm on the two crossed trees successively. The final costaggregation process for a pixel p is illustrated in Fig. 4. To sum up,we perform a ‘window-based’ cost volume filtering in a non-localway. Different from the local filtering-based methods, the support‘window’ for each pixel is set to be the whole reference image.

As mentioned above, we make explicit smoothness assumptionby assigning each edge a truncated weight as (6) at first, and thenpreserve the depth boundaries by incorporating a prior into thenon-local framework later.

wðs; rÞ ¼minðj Is� Ir j ; τÞ ð6ÞHere, τ is the threshold of the color difference of two neighboringpixels s and r. Truncated weights of the edges are corresponding toour global smoothness assumption. Note that the proximity isimplicitly employed here since the distance is accumulated along apath on the tree. In other words, the further the two pixels inEuclidean space, the further the two pixels on the tree. Fig. 5shows the consistency of the two different spatial measurements.

3.2.2. Edge prior and superpixel priorAny pair of neighboring pixels is connected on the cross-treeswith

a truncated edge weight, and the non-local cost aggregation will blurthe depth boundaries because we assume disparity smoothnesseverywhere. Due to this reason, appropriate priors must be employedto indicate the potential locations of the depth boundaries. One of thecommonly used priors is the edge prior, as shown in Fig. 6(a).However, only a small part of the color edges belongs to the truedepth boundaries and assuming all the color edges as the depthboundaries will degrade the cost aggregation within highly texturedregions. Motivated by the categorization scheme proposed in [33,34],we propose to employ the superpixel prior in this work. By takingadvantage of the over-segmentation results, many false depthboundaries are removed as shown in Fig. 6(b).

Someone may wonder why we employ over-segmentationinstead of segmentation. We use the segmentation results as aprior to estimate the support regions during performing the costaggregation. As the units of local support regions, the segmentsneed to be compact and regular, especially in highly textured

Fig. 3. Illustration of the proposed cross-trees structure. Left: horizontal-tree; right:vertical-tree. The edges denoted as dot lines are the extra edges we add to connectdifferent rows or columns for constructing a tree. The weights of these edges areset to be a large number to prevent cost aggregation between different rows orcolumns in practical applications.

Fig. 4. Sequential non-local cost aggregation of a pixel p. The horizontal arrowsdenote the non-local cost aggregation on the horizontal-tree and the vertical arrowsdenote the non-local cost aggregation on the vertical-tree.

Fig. 5. The proximity in Euclidean space and the distance on the tree.

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–22782272

regions. From this perspective, segmentation algorithms cannotwork as well as over-segmentation algorithms. Additionally, thedepth boundaries are fully connected in this way since the super-pixels are compact and regular. In this paper, we utilize the SLICalgorithm [35] which have competitive performance on adherenceto potential disparity boundaries than other superpixel algorithms[36–38]. We have tried more complex constraints such as thesegmentation and the global edge constraint (GEC) proposed in[33,34], but the improvement was not obvious.

In our work, the weights of the edges are dependent on thepixel similarity (color difference) and the prior we employed. Theweight of an edge is rewritten as follows:

wðs; rÞ ¼j Is� Ir j if eðs; rÞ \ the priorminðj Is� Ir j ; τÞ others

(ð7Þ

Here, eðs; rÞ \ the prior means that the edge between s and rcrosses the prior. τ is the threshold of the edge weights if the edgesdo not cross the prior. τ is set to be 6/255 in all our experiments toenforce the cost aggregation between pixels at the same side ofthe prior. In this way, we aggregate the costs within planarsurfaces and preserve the depth boundaries when the prior exists.

Fig. 7 shows how the prior selects optimal support regions for thecenter pixel p. For an edge eðq3-q2Þ on a path Pðq3; pÞ, its weight isdefined as the color difference of the two neighboring pixels since itcrosses the proposed prior. In this case, the distance from q3 to p willbe raised sharply if the color difference of q3 and q2 is large. Hence,further pixels such as q3 will contribute much less to the center pixel p.By cutting the cost aggregation flow on different paths, our methodcan estimate support regions with arbitrary shapes and sizes, which isa challenge problem for traditional window-based local methods.

To a certain extent, our work is also motivated by the cross-basedstructures methods [19,20]. Instead of estimating the cross-basedstructures locally, we use the priors to locate potential local supportregions' boundaries and then estimate the support regions as cross-structures on the horizontal and the vertical scanlines implicitly bycutting the cost aggregation flow as shown in Fig. 7.

Note that for the superpixel prior, there are over-segmentboundaries even between neighboring pixels within low-texturedregions. Such over-segment boundaries will not terminate the costaggregations from further pixels on the path since the edge weights(equal to pixel similarity as defined in (7)) are still low.

3.3. Computational complexity

For all the tree-based non-local methods, we compare thecomputational complexities of the tree construction and the

non-local cost aggregation only because the other two steps areexactly the same for all the methods in the non-local framework.

For an N�N image, let n denote the number of the pixels in Iand m¼OðnÞ denote the number of the edges. For both the MSTand the ST constructions, the computational complexity of theedge weights estimation is O(m). The computational complexityis Oðlog 2 mþ2m log 2 nþnÞ for the MST construction, andOðmþ2mαðmÞÞ for the ST1 construction. Here, αðmÞ can beapproximated as a small constant for practical applications accord-ing to [17]. For ST2, the tree construction time is doubledcompared to ST1 because ST1 need to be performed at first toprovide initial depth results. Our cross-trees structure needs n�N

Fig. 6. Priors incorporated into the non-local framework. Left: edge prior; right: superpixel prior. The edges are detected by using Canny edge detector. The superpixels areestimated by using the SLIC algorithm [35].

Fig. 7. Non-local cost aggregation for pixel p on the cross-trees structure with aprior. Green arrows represent cost aggregation flow on horizontal scanlines and redarrows represent cost aggregation flow on the vertical scanlines. A cross-basedstructure which consists of these color arrows is the support region of pixel p. (Forinterpretation of the references to color in this figure caption, the reader is referredto the web version of this article.)

Table 1Computational complexities of the tree construction and the non-local costaggregation.

Methods Computational complexity

Tree construction Non-local cost aggregation

MST Oðmþ log 2 mþ2m log 2 nþnÞ Oðn � lÞST1 Oð2ðmþmαðmÞÞÞ Oðn � lÞST2 Oð4ðmþmαðmÞÞÞ Oð2ðn � lÞÞCross_E Oð2ðn�NÞÞ Oð2ðn � lÞÞCross_Sp Oð2ðn�NÞÞ Oð2ðn � lÞÞ

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–2278 2273

subtraction operations for constructing each tree and the totalcomputational complexity is Oð2ðn�NÞÞ.

The computational complexity of the non-local cost aggrega-tion ðOðn � lÞÞ is doubled in our algorithms and ST2 since we both

need to process two trees successively. We use Cross_E to denoteour cross-treesþedge prior-based algorithm and Cross_Sp todenote our cross-treesþsuperpixel prior-based algorithm. Table 1shows the comparison of the computational complexities of theentire five algorithms. For Cross_E and Cross_Sp, additional com-putations are needed to detect the color edges and to establishsuperpixels respectively. The exact execution time of each algo-rithm will be provided in the experiments (Section 4).

4. Experimental results

The performance of our method and the other two non-local costaggregation methods (MST and ST) is compared in this section. Bothqualitative evaluations and disparity maps are provided. Our methodcontains two algorithms: Cross_E and Cross_Sp. The ST method alsocontains two algorithms: initial ST-based algorithm ST1 and depthmap enhanced ST-based algorithm ST2.

Code: The results of ST1 and ST2 are estimated by using theCþþ code which has been published by the paper authors [17].We implement our algorithms and re-implement MST in the samecode framework. Hence, all the functions are the same in the fivealgorithms except the functions of the tree construction and somepre-processing steps. This is an equitable way for comparing thecost aggregation accuracy and the running time of the algorithms.

Data sets: We use 27 Middlebury data sets [39], including thefour commonly used data sets (Tsukuba, Venus, Teddy and Cones),to give a more reliable evaluation of the performance of the fivealgorithms. Moreover, we test MST, ST1 and Cross_E on manystereo video data sets from the Microsoft i2i dataset [40].

Parameter setting: For all the test data sets, the parameter ofboth our algorithms is constant: σ ¼ 0:05. σ is set empirically topermit a pixel to get weighted support from other pixels on the

Table 2Performance evaluation of the cost aggregation accuracy.

Data MST ST1 ST2 Cross_E Cross_Sp

Tsukuba 2:265 1:852 1:831 2:234 2:143

Venus 0:694 0:643 0:421 0:715 0:602

Teddy 7:282 7:674 7:171 7:825 7:653

Cones 3:824 3:553 3:221 3:925 3:232

Flowerpots 16:645 14:914 13:851 14:362 14:423

Baby1 5:575 4:522 4:261 4:633 4:633

Baby2 18:955 15:204 14:293 6:332 6:131

Baby3 5:133 5:072 4:891 5:234 5:275

Art 10:625 10:524 10:083 8:581 8:581

Aloe 4:675 4:404 4:373 4:022 3:521

Books 9:795 9:414 8:873 8:332 8:021

Cloth1 0:575 0:413 0:464 0:292 0:151

Cloth2 3:625 3:364 2:973 1:821 1:832

Cloth3 2:245 1:572 1:824 1:683 1:221

Cloth4 1:375 1:274 1:183 0:932 0:771

Dolls 5:885 5:173 5:504 4:431 4:462

Lampshade1 11:575 10:311 10:402 10:423 10:454

Lampshade2 14:742 21:654 21:745 14:721 14:763

Laundry 13:562 13:834 12:051 13:915 13:703

Moebius 8:395 8:194 7:751 8:093 7:892

Wood1 5:145 5:084 4:193 3:712 3:651

Wood2 3:995 3:064 2:053 0:801 0:832

Bowling1 22:95 18:884 16:593 15:511 15:522

Bowling2 12:875 11:134 10:073 8:792 8:731

Rocks1 2:605 2:343 2:414 1:481 1:612

Rocks2 1:985 1:613 1:744 1:271 1:352

Reindeer 9:155 7:734 6:513 5:672 5:391

Avg. error 7:635 7:164 6:693 5:912 5:801

Avg. rank 4:525 3:374 2:523 2:442 2:041

Fig. 8. The initial disparity maps without post-processing. The 1st row: Baby2; the 2nd row: Reindeer; the 3rd row: Lampshade1; the 4th row: Wood2. The bad pixels in thedisparity maps are marked red. See the regions of (e) and (f) within the yellow rectangular boxes, both our algorithms establish more smooth disparity maps within planarsurfaces. Details are best viewed in the electronic version. (a) The reference image, (b) MST, (c) ST1, (d) ST2, (e) Cross_E and (f) Cross_Sp. (For interpretation of the referencesto color in this figure caption, the reader is referred to the web version of this article.)

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–22782274

tree, and to cut the cost aggregation flow on a path at where itcrosses the prior we employed.

For both our algorithms, the priors define the support regionfor a pixel p and σ controls the support weights of p's neighboringpixels within the estimated support region. A larger σ means thatthe cost of pixel p will be smoothed powerfully, and suchparameter setting may result in over-smooth disparity maps. Onthe contrary, a smaller σmeans a less powerful filtering of the costof pixel p which will lead to noisy disparity maps.

For the Cross_Sp algorithm, the number of superpixels nsp is setto be n=ð10� 10Þ. Here, we suppose that the stereo images can beover-segmented into superpixels with an approximate size10�10. In other words, we suppose that the superpixels with anapproximate size 10�10 will not cover the depth boundariesunder this constraint for the test stereo images. A 10�10 super-pixel, within highly textured regions of the reference image,usually has a clear distinction to find reliable corresponding pointsin the other image for the pixels within it.

However, nsp should be changed to a larger value to segmentthe stereo images into smaller patches (superpixels) for stereo

images which contain tiny elements (less than 10�10 pixels). Inthis way, the superpixels will not cover the tiny elements'boundaries while degrade the distinctiveness. Hence, nsp shouldbe set as a trade-off between details preservation and superpixels'distinctiveness for matching.

For the MST, ST1 and ST2 algorithms, the parameters are set tobe the same with the settings in the original papers. The AD-Gradient cost function proposed in [14] is used for all the fivealgorithms to compute the cost volume.

Table 2 shows the quality evaluation of the initial disparitymaps established by using the five algorithms. We firstly performthe non-local cost aggregation algorithm on the different trees andthen establish the disparity maps with the WTA strategy. Tocompare the aggregation accuracy, no post-processing is per-formed in this step and only the non-occluded regions areevaluated. In Table 2, the normal numbers are the percentages ofthe bad pixels in the disparity maps with error threshold 1 and thesubscript numbers are the relative rank in each row. ST1 and ST2outperform MST for most of the test data sets as declared in theauthors' paper [17]. The initial disparity maps established by using

Fig. 9. Experimental results of the Middlebury data sets. The 1st row: the reference images; The 2nd row: the disparity maps established with ST2. The 3rd row: thedisparity maps established with Cross_SpþWMF. The 4th row: the disparity maps established with GF. The 5th row: the disparity maps established with MST.

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–2278 2275

the five algorithms are shown in Fig. 8. Both our algorithmsperform better than other methods within planar surfaces forthe reason that the truncated edge weights enforce strong costaggregations within these regions where no prior is detected orthe color difference is low.

Among the 27 test data sets, Cross_E ranks 1 for 8 stereo pairs,Cross_Sp ranks 1 for 10 stereo pairs and ST2 ranks 1 for 9 stereopairs. The average rank and the average error are computed in thelast two rows of Table 2. Both our algorithms are competitive andrank 1 and 2 respectively. Cross_Sp has both the best overallaccuracy and the best overall rank.

For the four standard Middlebury data sets, we test the fivealgorithms on a PC platform with Core i5 3.00 GHz CPU-8GB Mem-ory-64 bit OS. The average execution time for each algorithm is: MST-0.44 s, ST1-0.49 s, ST2-0.97 s, Cross_E-0.49 s, and Cross_Sp-0.69 s.Cross_E and ST1 have the similar efficiency with MST. Cross_Sp isslower than Cross_E due to the superpixel algorithm we employed[35]. ST2 is the slowest algorithm since it requires an initial estimationof the depth map and an enhanced construction of the tree.

We use the Cross_Sp algorithm to establish the initial disparitymaps for both the left and the right images of a stereo pair. After thecross-checking, we employ a constant time weighted median filter(WMF) as the post-processing technique [41]. The final disparity mapsof the four standard Middlebury data sets are shown in Fig. 9. Table 3is the performance evaluation on the Middlebury test bed. Comparedto the results which other authors submitted to the Middleburywebsite, our method ranks 2 among the four methods. GF had beenthe best local method when the original paper was published [14].Actually, the gaps between the average error rates of the algorithmsare really narrow and different post-processing techniques havesignificant effect on the final results. In this paper, we mainly focuson the cost aggregation accuracy (i.e., the initial disparity mapswithout post-processing).

Additional results of the Illkay and the Simon stereo video datasets from the Microsoft i2i dataset are shown in Fig. 8. We only

compare the MST, ST1 and Cross_E algorithms for video processingbecause they are much faster than Cross_Sp and ST2 and have thesimilar efficiency. Cross_E establishes the most smooth disparitymaps within the foregrounds and has the best performance nearthe foregrounds' depth boundaries (Fig. 10).

5. Discussion

A challenge problem of local stereo matching methods is locallyrepeated texture. In such case, a pixel of the reference image canoften find several correspondents in the other image because localmethods measure the similarity of a correspondence locally. In thissection, we show the performance of the non-local algorithms onthree challenge test data sets: Midd1, Midd2 and Plastic. For all theimages, there is a large repeated texture in the backgrounds orforegrounds. As shown in Table 4, the average error rates of thealgorithms are all over 30%. The bad results mainly locate inrepeated texture regions as shown in Fig. 11.

Actually, all tree-based non-local methods are proved to be stillestablishing locally optimized results within large planar surfaces dueto the Gaussian kernel of the support weights. As we have discussedin Section 3, the distance between two pixels increases progres-sively along a path. Fig. 12 shows that a leaf node contributes to the

Table 3Performance evaluation on the Middlebury stereo test bed.

Algorithms Avg. rank Tsukuba Venus Teddy Cones Avg. error

nonocc all disc nonocc all disc nonocc all disc nonocc all disc

ST2 [17] 34.5 1.25 1.68 6.69 0.20 0.30 1.77 6.00 11.9 15.0 2.77 8.82 7.81 5.35Cross_SpþWMF 41.2 1.68 1.99 7.82 0.22 0.32 2.84 6.23 11.7 14.8 2.52 7.71 7.50 5.44GF [14] 42.2 1.51 1.85 7.61 0.20 0.39 2.42 6.16 11.8 16.0 2.71 8.24 7.66 5.55MST [16] 45.4 1.47 1.85 7.88 0.25 0.42 2.60 6.01 11.6 14.3 2.87 8.45 8.10 5.48

Fig. 10. For the stereo video data sets, the disparity maps of a frame are established by using the MST, ST1 and Cross_E algorithms respectively. The 1st row: Ilkay; the 2ndrow: Simon. Cross_E performs much better than the other two algorithms in the foregrounds. The non-local post-processing technique proposed in [12] is employed in all thethree algorithms. (a) The reference image, (b) MST, (c) ST1 and (d) Cross_E.

Table 4Performance evaluation on three challenge data sets. The numbers are thepercentages of bad pixels in the non-occluded regions of the disparity maps witherror threshold 1.

Data MST ST1 ST2 Cross_E Cross_Sp

Midd1 26.05 31.37 34.37 35.66 35.41Midd2 45.14 28.67 35.45 31.68 31.71Plastic 39.76 40.82 35.84 36.87 36.71Avg. error 36.98 33.62 35.22 34.74 34.61

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–22782276

root node only when the distance between them is smaller than avalue which is related to the parameter σ. Hence for a given σ, themaximum size of the support regions is limited. It means that ‘non-local’methods still establish ‘local’ optimized results when there arelarge planar surfaces with repeated texture.

6. Conclusions and future work

We proposed a novel cross-trees structure for the non-local costaggregation. Based on two different priors, two algorithms areimplemented to address the stereo matching problem. Comparedto the other two top-ranked non-local cost aggregation methods,both our algorithms have better overall rank and overall accuracyfor initial disparity map estimation. Performance evaluation on theMiddlebury test bed also shows that our method is competitive. Thefuture work should emphasize on constructing more effective treesto resolve the challenge problem as we discussed above. New ideasfor performing the non-local cost aggregation are needed too.

Conflict of interest

None.

Acknowledgement

This work was supported by the National Natural ScienceFoundation of China Grant No. 61272351 and the National Insti-tutes of Health Grant Nos. R01CA165255 and R21CA172864 of theUnited States.

Appendix A. Supplementary data

Supplementary data associated with this article can be found inthe online version at http://dx.doi.org/10.1016/j.patcog.2015.01.002.

Fig. 11. The initial disparity maps without post-processing. The bad pixels in the disparity maps are marked red (see details in electronic version). The 1st row: Midd1; the2nd row:Midd2; the 3rd row: Plastic. (a) The reference image, (b) MST, (c) ST1, (d) ST2, (e) Cross_E and (f) Cross_Sp. (For interpretation of the references to color in this figurecaption, the reader is referred to the web version of this article.)

Fig. 12. Gaussian kernel of the support weights, σ ¼ 0:1. The horizontal axis shows the un-normalized distance between two nodes p and q on the constructed tree, and thevertical axis shows the values of support weights. As we can see, the root node can only get effective support weights from leaf nodes which are near enough (for example,Dðp; qÞo80; Sðp; qÞ40:05).

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–2278 2277

References

[1] M. Pollefeys, D. Nistér, J.M. Frahm, et al., Detailed real-time urban 3dreconstruction from video, Int. J. Comput. Vis. 78 (2–3) (2008) 143–167.

[2] A. Geiger, J. Ziegler, C. Stiller, Stereoscan: dense 3d reconstruction in real-time,in: Intelligent Vehicles Symposium, IEEE, 2011, pp. 963–968.

[3] C. Fehn, Depth-image-based rendering (dibr), compression, and transmissionfor a new approach on 3d-tv, in: Electronic Imaging, International Society forOptics and Photonics, 2004, pp. 93–104.

[4] C.L. Zitnick, S.B. Kang, Stereo for image-based rendering using image over-segmentation, Int. J. Comput. Vis. 75 (1) (2007) 49–65.

[5] S. Sengupta, E. Greveson, A. Shahrokni, P.H. Torr, Urban 3d semantic modellingusing stereo vision, in: International Conference on Robotics and Automation,IEEE, 2013, pp. 580–585.

[6] D. Scharstein, R. Szeliski, A taxonomy and evaluation of dense two-framestereo correspondence algorithms, Int. J. Comput. Vis. 47 (1) (2002) 7–42.

[7] Y. Boykov, O. Veksler, R. Zabih, Fast approximate energy minimization viagraph cuts, Pattern Anal. Mach. Intell. 23 (11) (2001) 1222–1239.

[8] V. Kolmogorov, R. Zabin, What energy functions can be minimized via graphcuts? Pattern Anal. Mach. Intell. 26 (2) (2004) 147–159.

[9] Q. Yang, L. Wang, N. Ahuja, A constant-space belief propagation algorithm forstereo matching, in: Computer Vision and Pattern Recognition, IEEE, 2010,pp. 1458–1465.

[10] T. Kanade, M. Okutomi, A stereo matching algorithm with an adaptivewindow: theory and experiment, Pattern Anal. Mach. Intell. 16 (9) (1994)920–932.

[11] O. Veksler, Fast variable window for stereo correspondence using integralimages, in: Computer Vision and Pattern Recognition, vol. 1, IEEE, 2003, pp.I–556.

[12] K.-J. Yoon, I.S. Kweon, Adaptive support-weight approach for correspondencesearch, Pattern Anal. Mach. Intell. 28 (4) (2006) 650–656.

[13] A. Hosni, M. Bleyer, M. Gelautz, C. Rhemann, Local stereo matching usinggeodesic support weights, in: International Conference on Image Processing,IEEE, 2009, pp. 2093–2096.

[14] C. Rhemann, A. Hosni, M. Bleyer, et al., Fast cost-volume filtering for visualcorrespondence and beyond, in: Computer Vision and Pattern Recognition,IEEE, 2011, pp. 3017–3024.

[15] J.H. Kappes, B. Andres, F.A. Hamprecht, et al., A comparative study of moderninference techniques for discrete energy minimization problems, in: ComputerVision and Pattern Recognition, IEEE, 2013, pp. 1328–1335.

[16] Q. Yang, A non-local cost aggregation method for stereo matching, in:Computer Vision and Pattern Recognition, IEEE, 2012, pp. 1402–1409.

[17] X. Mei, X. Sun, W. Dong, et al., Segment-tree based cost aggregation for stereomatching, in: Computer Vision and Pattern Recognition, IEEE, 2013, pp.313–320.

[18] M. Okutomi, Y. Katayama, S. Oka, A simple stereo algorithm to recover preciseobject boundaries and smooth surfaces, Int. J. Comput. Vis. 47 (1–3) (2002)261–273.

[19] K. Zhang, J. Lu, G. Lafruit, Cross-based local stereo matching using orthogonalintegral images, Circuits Syst. Video Technol. 19 (7) (2009) 1073–1079.

[20] X. Mei, X. Sun, M. Zhou, et al., On building an accurate stereo matching systemon graphics hardware, in: International Conference on Computer VisionWorkshops, IEEE, 2011, pp. 467–474.

[21] C. Richardt, D. Orr, I. Davies, et al., Real-time spatiotemporal stereo matchingusing the dual-cross-bilateral grid, in: European Conference on ComputerVision, Springer, 2010, pp. 510–523.

[22] K. He, J. Sun, X. Tang, Guided image filtering, in: European Conference onComputer Vision, Springer, 2010, pp. 1–14.

[23] A. Hosni, M. Gelautz, M. Bleyer, Accuracy-efficiency evaluation of adaptivesupport weight techniques for local stereo matching, in: Pattern Recognition,Elsevier, 2012, pp. 337–346.

[24] A. Hosni, M. Bleyer, M. Gelautz, Secrets of adaptive support weight techniquesfor local stereo matching, Comput. Vis. Image Underst. 117 (6) (2013) 620–632.

[25] A. Klaus, M. Sormann, K. Karner, Segment-based stereo matching using beliefpropagation and a self-adapting dissimilarity measure, in: InternationalConference on Pattern Recognition, vol. 3, IEEE, 2006, pp. 15–18.

[26] M. Bleyer, C. Rother, P. Kohli, et al., Object stereojoint stereo matching andobject segmentation, in: Computer Vision and Pattern Recognition, IEEE, 2011,pp. 3081–3088.

[27] O. Veksler, Stereo correspondence by dynamic programming on a tree, in:Computer Vision and Pattern Recognition, vol. 2, IEEE, 2005, pp. 384–390.

[28] C. Lei, J. Selzer, Y.-H. Yang, Region-tree based stereo using dynamic program-ming optimization, in: Computer Vision and Pattern Recognition, vol. 2, IEEE,2006, pp. 2378–2385.

[29] M. Bleyer, M. Gelautz, Simple but effective tree structures for dynamicprogramming-based stereo matching, in: VISAPP, 2008, pp. 415–422.

[30] S. Chambon, A. Crouzil, Similarity measures for image matching despiteocclusions in stereo vision, Pattern Recognit. 44 (9) (2011) 2063–2075.

[31] P. Aschwanden, W. Guggenbuhl, Experimental results from a comparativestudy on correlation-type registration algorithms, Robust Comput. Vis. (1992)268–289.

[32] M.Z. Brown, D. Burschka, G.D. Hager, Advances in computational stereo,Pattern Anal. Mach. Intell. 25 (8) (2003) 993–1008.

[33] F. Cheng, H. Zhang, D. Yuan, M. Sun, Stereo matching by using the global edgeconstraint, Neurocomputing 131 (2014) 217–226.

[34] H. Zhang, F. Cheng, D. Yuan, Y. Li, M. Sun, Stereo matching with global edgeconstraint and graph cuts, in: International Conference on Pattern Recogni-tion, IEEE, 2012, pp. 372–375.

[35] R. Achanta, A. Shaji, K. Smith, et al., SLIC superpixels compared to state-of-the-art superpixel methods, Pattern Anal. Mach. Intell. 34 (11) (2012) 2274–2282.

[36] A. Levinshtein, A. Stere, K.N. Kutulakos, D.J. Fleet, S.J. Dickinson, K. Siddiqi,Turbopixels: fast superpixels using geometric flows, Pattern Anal. Mach. Intell.31 (12) (2009) 2290–2297.

[37] A.P. Moore, S. Prince, J. Warrell, U. Mohammed, G. Jones, Superpixel lattices,in: Computer Vision and Pattern Recognition, IEEE, 2008, pp. 1–8.

[38] A.P. Moore, S.J. Prince, J. Warrell, U. Mohammed, G. Jones, Scene shape priorsfor superpixel segmentation, in: International Conference on ComputerVision, IEEE, 2009, pp. 771–778.

[39] Middlebury steteo matching test bed, http://vision.middlebury.edu/stereo/eval/, 2002.

[40] Microsoft i2i dataset, http://research.microsoft.com/en-us/projects/i2i/data.aspx, 2007.

[41] Z. Ma, K. He, Y. Wei, et al., Constant time weighted median filtering for stereomatching and beyond, in: International Conference on Computer Vision, 2013.

Feiyang Cheng received the B.S. degree in bio-medical engineering from Tianjin Medical University, China in 2010. He is currently a Ph.D. candidate in Image ResearchCenter, School of Astronautics at Beihang University, China. His research interests include stereo vision, image analysis and computational photograph.

Hong Zhang received her Ph.D. degree from Beijing Institute of Technology, China in 2002. She is currently a Professor of Beihang University. She was at the University ofPittsburgh as a visiting scholar from 2007 to 2008. Her research interests include activity recognition, image indexing, object detection and stereo vision.

Mingui Sun received his Ph.D. degree in 1989 from the University of Pittsburgh, USA where he is currently a Professor of Neurosurgery, Electrical and Computer engineering,and Bioengineering. His current research interests include advanced biomedical electronic devices, biomedical signal and data processing systems for diet and physicalactivity assessment.

Ding Yuan received her M.Phil. and Ph.D. degrees from the Chinese University of Hong Kong in 2004 and 2008 respectively. She is now an assistant professor of ImageResearch Center at Beihang University, China. Her research interests include stereo vision, 3D reconstruction, camera calibration, and camera's ego-motion estimation.

F. Cheng et al. / Pattern Recognition 48 (2015) 2269–22782278