NONPARAMETRIC TEXTURE AN ALYSIS WITHCOMPLEMENT ARY SPATIAL OPERA TORS

Matti Pietikäinen and Timo OjalaMachine Vision and Media Processing Unit, Infotech Oulu, University of Oulu

P.O. Box 4500, FIN-90401 Oulu, Finland{mkp,skidi}@ee.oulu.fi

Abstract. Recently, we have developed a nonparametric approach to texture analysis, in whichthe distributions of simple texture measures based on local binary patterns and signed gray leveldifferences are used to provide complementary information about the strurctural and stochasticproperties of image texture. Very good performance has been obtained in various texture clas-sification and segmentation problems. This paper overviews our approach and discusses rea-sons for its efficiency.

1. Introduction

Texture analysis is important in many applications of computer image analysis for classifica-tion, detection, or segmentation of images based on local spatial variations of intensity orcolor. Many different approaches to texture analysis have been proposed. Among the mostwidely used texture measures are those derived from gray level cooccurrence matrices or dif-ference histograms, “texture energy” measures obtained by local linear transforms, featuresbased on multi-channel Gabor filtering, and Markov random field models [4,5,29,3].

Recently, we have developed a nonparametric approach to texture analysis, in whichthe distri-butions of simple texture measures based on local binary patterns and signed gray level differ-ences are used to provide complementary information about the strurctural and stochasticproperties of image texture. Very good performance has been obtained in various texture clas-sification and segmentation problems, see e.g. [6,13-15,17-20,23-24].

This paper overviews our approach and discusses reasons for its efficiency.

2. Texture description with simple spatial operators

2.1 Local Binary Patterns

Ojala, Pietikäinen and Harwood [17] introduced the Local Binary Pattern (LBP) texture opera-tor shown in Fig. 1. The original 3x3 neighborhood is thresholded by the value of the centerpixel. The values of the pixels in the thresholded neighborhood are multiplied by the weightsgiven to the corresponding pixels. Finally, the values of the eight pixels are summed to obtain anumber for this neighborhood. The LBP histogram computed over a region is used for texturedescription. LBP provides us with knowledge about the spatial structure of the local image tex-ture.

LBP is invariant against any monotonic gray scale transformation. LBP does not address thecontrast of texture which is important in the discrimination of some textures. For this purpose,we can combine LBP with a simple contrast measure C as shown in Fig. 1 and consider jointoccurrences of LBP and C.

He and Wang [7] have introduced a Texture Spectrum operator, which is similar to LBP, but ituses three levels (i.e. two thresholds) instead of two levels used in LBP. This leads to a moreinefficient representation and implementation than with LBP (6561 bin values instead of 256),and according to our tests the three-level operator does not perform any better than LBP. TheTexture Spectrum operator usually also needs a user defined delta value for setting the thresh-old values, which makes it dependent on the gray scale variance.

2.2 Signed gray-level differences

Recently, Ojala et al. [20] showed that an approach based on multidimensional distributions ofsigned gray level differences of neighboring pixel values is very powerful for texture classifica-tion. The advantages of gray-level differences over the traditional cooccurrence method are:(1) the differences fall mainly within a narrower range than the gray levels, due to the high cor-relation between gray levels of adjacent pixels, consequently providing a more compact de-scription of texture; (2) the signed differences are not affected by changes in mean luminance.In comparison to the commonly used absolute differences, the signed differences contain moreinformation about image texture and consequently are more powerful.

In our experiments, the classification performance of two-, four-, and eight-dimensional differ-ence distributions has been evaluated. By computing cooccurring differences within 3x3-pixelsubimages,

we estimate following distributions,

The LBP operator is similar top8, when differences are expressed with two levels. If three lev-els are used,p8 resembles the Texture Spectrum operator.

The volume of the difference space for an image with G gray levels equals (2G-1)k, wherek=2,4,8, corresponding to the distribution we are estimating. If we would straightforwardly de-scribe the difference space with ak-dimensional histogram, we would obtain, even with mod-

g4 g2 g3

g5 g0 g1

g6 g7 g8

1 0 0

1 0

1 0 1

1 0 0

8 0

32 0 128

6 5 2

7 6 1

9 3 7

P1 P2 P3

P4 P0 P5

P6 P7 P8

example thresholded weightspixels

LBP = 1+8+32+128 =169C = (6+7+9+7)/4 - (5+2+1+3)/4 =4.5

Fig. 1. Computation of Local Binary Pattern (LBP) and contrast measure C.

p2 g1 g0– g2 g0–,( ) (1)

p4 g1 g0– g2 g0– g3 g0– g4 g0–, , ,( ) (2)

p8 g1 g0– g2 g0– … g8 g0–, , ,( ) (3)

est values ofG, very large histograms that are computationally expensive and suspect tostatistical unreliability. Instead of reducingG, for example, with simple requantization of eachcoordinate, we partition thek-dimensional difference space using vector quantization [18]. Forthis purpose we employ a codebook ofN k-dimensional codewords, which have indicesn =0,1,...,N-1. The codebook is trained with the optimized LVQ1 training algorithm [10], by se-lecting random vectors from each of the samples in the training set. Our method of vectorquantization is computationally much simpler than that used by Valkealahti and Oja [30].

We describe the difference information of a texture sample with a difference histogram. Themapping from the difference space to a difference histogram is straightforward. Given a partic-ular k-dimensional difference vector, the index of the nearest codeword corresponds to the binindex in the difference histogram. In other words, a codebook ofN codewords produces a his-togram ofN bins. The difference histogram of a texture sample is obtained by searching thenearest codeword to each vector present in the sample, and incrementing the bin denoted by theindex of this nearest codeword.

3. Experiments with texture classification

The 32 Brodatz textures used in the experiments are shown in Fig. 2 [2,30]. The images are256x256 pixels in size and they have 256 gray levels. Each image was divided into 16 disjoint64x64 samples, which were independently histogram-equalized to remove luminance differ-ences between textures. To make the classification problem more challenging and generic,three additional samples were generated from each sample: a sample rotated by 90 degrees, a64x64 scaled sample obtained from the 45x45 pixels in the middle of the ‘original’ sample,and a sample that was both rotated and scaled. Consequently, the classification problem in-volved a total of 2048 samples, 64 samples in each of the 32 texture categories [30].

The texture classifier was trained by randomly choosing, in each texture class, eight ‘original’samples, together with the corresponding 24 transformed samples, as models. The other half ofthe data, eight ‘original’ samples and the corresponding 24 transformed samples in each tex-

Fig. 2. Brodatz textures used in classification experiments.

ture class, was used for testing the classifier. In the classification phase a test sampleS was as-signed to the class of the modelM that maximized the log-likelihood measure

whereSn andMn correspond to the sample and model probabilities of binn, respectively.

We estimated distributionsp2, p4 andp8, by partitioning the difference space with a codebookof 384 codewords. The codebook was trained with the standard optimized LVQ1 training algo-rithm, by selecting 100 random vectors from each of the 1024 samples in the training set.

In experiments we obtained classification accuracies of 93.3% forp2, 95.7% forp4 and 96.8%for p8, respectively. These results are excellent considering the difficulty of the problem. Thesimple LBP operator achieved a remarkable accuracy of 94.4%, i.e. only 2.4% less than themuch more complex p8 operator.

For comparison, experiments with features based on standard GMRF models and Gabor filter-ing using different mask sizes, implemented as in [26], and two different classifiers were car-ried out. The best result for the GRMF approach was 68.2%, obtained with a standard 6th ordersymmetric mask and a multivariate Gaussian discriminant classifier. The best combination ofGMRF features obtained from features computed using all models from the 1st order to 6th or-der achieved 87.7%, providing an estimation for the best accuracy a method based multiresolu-tion GMRF might achieve. The best standard Gabor features, extracted with a filter bank ofthree different wavelengths and four different orientations in a 11x11 neighborhood, using a 3-NN classifier achieved an accuracy of 88.0%. The best combination of Gabor features obtainedfrom features computed using all mask sizes between 7x7 and 17x17 pixels achieved an accu-racy of 90.2%, indicating the best accuracy a multiscale Gabor approach might achieve.

These results are significantly poorer than those obtained with our methods, even though alarger neighborhood was used to compute these features instead of the 3x3 neighborhood usedin our approach. The rather poor results obtained especially with the GMRF method are partlycaused by the histogram equalization that was used to remove effects of unequal global bright-ness and contrast. Gray scale normalization is needed for a fair comparison of different meth-ods.

4. Experiments with unsupervised texture segmentation

Recently, an unsupervised texture segmentation algorithm using the LBP/C texture measureand nonparametric statistical test was developed by Ojala and Pietikäinen [15]. The methodhas performed very well in experiments. It is not sensitive to the selection of parameter values,does not require any prior knowledge about the number of textures or regions in the image asmost existing approaches do, and seems to provide significantly better results than existing ap-proaches. The method can be easily generalized, e.g., to utilize other texture features, multi-scale information, color features, and combinations of multiple features.

The segmentation method consists of three phases: hierarchical splitting, agglomerative merg-ing and pixelwise classification. First, hierarchical splitting is used to divide the image into re-gions of roughly uniform texture. Then, an agglomerative merging procedure merges similaradjacent regions until a stopping criterion is met. At this point, we have obtained rough esti-

L S M,( ) Sn Mnln

n 1=

N

∑= (4)

mates of the different textured regions present in the image, and we complete the analysis by apixelwise classification to improve the localization. Fig. 3 illustrates the steps of the segmenta-tion algorithm on a 512 x 512 mosaic containing five different Brodatz textures.

We have applied our segmentation method to a variety of texture mosaics with excellent re-sults [15,8].

The segmentation of a natural scene taken from [21] is shown in Fig. 4 [15]. The textures ofnatural scenes are generally more nonuniform than the homogeneous textures of the test mosa-ics. Also, in natural scenes adjacent textured regions are not necessarily separated by well-de-fined boundaries, but the spatial pattern smoothly changes from one texture to another. Further,we have to observe the infinite scale of texture differences present in natural scenes; choosingthe right scale is a very subjective matter. For these reasons there is often no ‘correct’ segmen-tation for a natural scene, as is the case with texture mosaics.

hierarchical agglomerative pixelwiseclassificationsplitting merging

(a) (b) (c) (d)

Fig 3. Main sequence of the segmentation algorithm.

Fig. 4. Segmentation of a natural scene.

The invariance of the LBP/C transform to average gray level shows in the bottom part of theimage, where the sea is interpreted as a single region despite the shadows. The result obtainedis very satisfactory, considering that important color or gray scale information is not utilized inthe segmentation.

5. Why is our approach so powerful?

The role of local features (primitives, textural elements, textons) and their differences in preat-tentive texture discrimination has been shown in psychophysical studies [1,9]. Our approachfits well to these findings: simple feature detectors are used to measure perceptually significanttextural properties and the segmentation is performed by comparing textural similarities ofneighboring regions on the basis of densities of different features.

In our approach, the distributions of texture measures based on local binary patterns and signedgray level differences are used to provide complementary information about the strurctural andstochastic properties of image texture, making it very efficient for various types of micro- andmacrotextures.

LBP encodes various simple feature detectors (edge, curve/line, corner, curve/line end, spot) atdifferent orientations in a single operator. An important point is that for each pixel the outputof the best-matching feature detector is only utilized. A conventional texture analysis methodbased on linear filtering (e.g., Laws’ masks) creates a feature vector for each pixel containinginformation both from good and bad local pattern matches, making this kind of approach lessefficient. The LBP features can be exracted at multiple scales by using successively reducedresolution versions of the image, for example.

The LBP histogam contains the density of each feature over a given region. The texture primi-tives detected by LBP could be linked to more complex textural elements using principles ofproximity and good continuation, but our exellent results suggest that this is usually not so im-portant. Coarseness, edge orientation and contrast are perceptually very important texturalproperties [28,11]. The LBP histograms contain information about edge orientation andcoarseness (e.g. edges per unit area). LBP does not address the contrast of texture. For this pur-pose, we can combine LBP with a simple contrast measure C. This measure is invariant withrespect to average gray level, which is very important considering, for example, the needs oftexture segmentation (see Fig. 4).

LBP is invariant against any monotonic gray scale transformation, which is a necessary proper-ty in many applications involving uneven illumination or great within-class variability; see, forexample, our results in metal strip inspection [22] and in the classification of tilted 3D textures[27]. The basic LBP method is sensitive to texture orientation, but this property appears to bevery useful in texture segmentation e.g. for finding accurate boundaries between neighboringregions. A rotation invariant version of LBP is proposed in [13,24], but recently we have sig-nificantly improved it by concentrating only on edge-like patterns encoded in LBP [16].

The signed gray level difference method is closely related to the widely used cooccurencemethod. It is invariant with respect to average gray level, while the cooccurrence approach isnot. The method is highly efficient in applications involving stochastic textures, but it also dis-criminates more sturctured textures very well. The difference histograms provide informationabout the coarseness and contrast of texture. For a coarse texture the difference histogram willpeak near zero, but for a more fine-grained texture it will spread out more uniformly. The

eight-dimensional signed gray level difference operator can be regarded as a generalization ofthe LBP operator, capturing information about structural and stochastic aspects of texture.

The importance of the spatial pattern and contrast information in texture discrimination is alsosupported by our earlier findings. A joint pair of a rotation-invariant version of LBP (ROTLBP)and a local variance measure (VAR) provided the best performance in our experiments with ro-tation-invariant texture classification [24]. By replacing ROTLBP with a new adaptive LBPthat only concentrates on edge-like patterns, the performance of this kind of feature pair can befurther improved significantly [16]. In addition, the original LBP measure and a measure basedon absolute gray level differences were the best features for two different test image sets in ourexperiments with multichannel texture description, providing together nearly as good perform-ance as an “optimal” combination of all 12 measures used in the study [14].

The choice of highly discriminating texture features is very important for a success in texturesegmentation. The features should easily discriminate various types of textures and the win-dow size used for computing textural features should be small enough to be useful for smallimage regions and to provide small error rates at region boundaries. The LBP, LBP/C andsigned gray level difference features perform well also for small image regions (e.g., 16 x 16pixels) unlike many other approaches.

Local texture similarity can be measured very efficiently and reliably with nonparametric sta-tistical test as in our segmentation procedure. As a major advantage, these methods do not re-quire the specification of a suitable vector-space metric. Instead, similarity is defined directlyvia the respective feature distributions [8].

Our results also suggest that a texture analysis process can be divided into more or less inde-pendent texture channels measuring, for example, pattern shape and gray-level difference sta-tistics properties. These features can be extracted at multiple resolutions. The featuredistributions computed over a region are used for texture description. This approach leads tocomputationally simple implementations. For example, the sum of the log-likelihood measurescomputed for a LBP histogram and for a 1-dimensional signed difference histogram (p1) pro-vided an accuracy of 95.9% in our classification problem, which is only 0.9% less than ob-tained with the 8-dimensional distributionp8. It would also be possible to use single features orjoint features one by one, by e.g. first comparing the similarity of regions with respect to spa-tial pattern and then with respect to gray level difference statistics. In addition to these, someother features measuring more global information about texture might be useful, such as perio-dicity [12] or color.

Rao and Lohse [25] identified three orthogonal dimensions for texture to be repetitive vs. non-repetitive; high-contrast and non-directional vs. low-contrast and directional; granular, coarseand low-complexity vs. non-granular, fine and high-complexity. The suggested feature setshould be efficient for discriminating textures of all these dimensions.

6. Conclusion

An efficient nonparametric approach to texture analysis using simple spatial operators has beendeveloped. The distributions of texture measures based on local binary patterns and signedgray level differences are used to provide complementary information about the strurctural andstochastic properties of image texture. Local texture similarity is measured very efficiently andreliably with nonparametric statistical tests. Our methodology resembles some well-known

psychophysical models on human texture perception. The approach has provided excellent re-sults in various texture classification and segmentation problems, and leads to computationallysimple implementations. Therefore, our results should be of great interest both from theoreticaland practical viewpoints.

Acknowledgements

The financial support provided by the Academy of Finland and the Technology DevelopmentCenter is gratefully acknowledged.

References

[1] J. Beck, S. Prazdny and A. Rosenfeld, “A theory of textural segmentation,” in Human andMachine Vision, eds. J. Beck, B. Hope and A. Rosenfeld, Academic, New York, 1983.

[2] P. Brodatz,Textures: A Photographic Album for Artists and Designers, Dover Publications,New York, 1966.

[3] R. Chellappa, R.L Kashyap and B.S. Manjunath, “Model-based texture segmentation andclassification,“ inHandbook of Pattern Recognition and Computer Vision, Second Edition,eds. C.H. Chen, L.F. Pau, P.S.P. Wang, World Scientific, Singapore, 1999, pp. 249-282.

[4] L. Van Gool, P. Dewaele and A. Oosterlinck, “Texture analysis anno 1983,” ComputerVision, Graphics, and Image Processing 29 (1985) 336-357.

[5] R.M. Haralick and L.G. Shapiro,Computer and Robot Vision, Vol. 1, Addison-Wesley,1992.

[6] D. Harwood, T. Ojala, M. Pietikäinen, S. Kelman and L. Davis, “Texture classification bycenter-symmetric autocorrelation, using Kullback discrimination of distributions,” PatternRecognition Letters 16 (1995) 1-10.

[7] D.C. He and L. Wang, “Texture unit, texture spectrum and texture analysis,” IEEE Trans.Geosci. Remote Sensing 28 (1990) 509-512.

[8] T. Hofmann, J. Puzicha and J .M. Buhmann, “Unsupervised texture segmentation in adeterministic annealing framework,” IEEE Transactions on Pattern Analysis and MachineIntelligence 20 (1998) 803-818.

[9] B. Julesz and J.R. Bergen, “Textons, the fundamental elements in preaattentive vision andperception of textures,“The Bell System Technical Journal 62 (1983) 1619-1645.

[10] T. Kohonen, J. Kangas, J. Laaksonen and K. Torkkola, “LVQ_PAK: A program packagefor the correct application of learning vector quantization algorithms,” Proc. InternationalJoint Conference on Neural Networks, Baltimore, 1992, pp. 1725-1730.

[11] M. Levine,Vision in Man and Machine, McGraw-Hill, 1985.[12] F. Liu and R.W. Picard, “Periocity, directionality, and randomness: Wold features for

image modeling and retrieval,” IEEE Transactions on Pattern Analysis and Machine Intelli-gence 18 (1996) 722-733.

[13] T. Ojala,Nonparametric texture Analysis Using Simple Spatial Operators, with Applica-tions in Visual Inspection, Acta Universitatis Ouluensis, C 105, 1997.

[14] T. Ojala and M. Pietikäinen, “Nonparametric multichannel texture description with simplespatial operators,“Proc. 14th Int. Conf. on Pattern Recognition, Brisbane, Australia, 1998,pp. 1952-1056.

[15] T. Ojala and M. Pietikäinen, “Unsupervised texture segmentation using feature distribu-tions,” Pattern Recognition 32 (1999) 477-486.

[16] T. Ojala and M.Pietikäinen, “An adaptive LBP texture operator” (1999), in preparation.[17] T. Ojala, M. Pietikäinen and D. Harwood, “A comparative study of texture measures with

classification based on feature distributions,” Pattern Recognition 29 (1996) 51-59.

[18] T. Ojala, M. Pietikäinen and J. Kyllönen, “Gray level cooccurrence histograms via learn-ing vector quantization,” Proc. 11th Scandinavian Conference on Image Analysis, Kanger-lussuaq, Greenland, 1999.

[19] T. Ojala, M. Pietikäinen and J. Nisula, “Determining composition of grain mixtures bytexture classification based on feature distributions,” Int. J. Pattern Recogn. Artif. Intell. 10(1996) 73-82.

[20] T. Ojala, K. Valkealahti, E. Oja and M. Pietikäinen, ”Texture discrimination with multidi-mensional distributions of signed gray level differences,” Submitted for review, 1999.

[21] D.K. Panjwani and G. Healey, “Markov random field models for unsupervised segmenta-tion of textured color images,” IEEE Transactions on Pattern Analysis and Machine Intelli-gence 17 (1995) 939-954.

[22] M. Pietikäinen, T. Ojala, J. Nisula and J. Heikkinen, “Experiments with two industrialproblems using texture classification based on feature distributions,” SPIE 2354 IntelligentRobots and Computer Vision XIII: 3D Vision, Product Inspection and Active Vision, 1994,pp. 197-204.

[23] M. Pietikäinen, T. Ojala and O. Silven, “Approaches to texture-based classification, seg-mentation and surface inspection,“ inHandbook of Pattern Recognition and ComputerVision, Second Edition, eds. C.H. Chen, L.F. Pau, P.S.P. Wang, World Scientific, Singapore,1999, pp. 711-736.

[24] M. Pietikäinen, T. Ojala and Z. Xu, “Rotation-invariant texture classification using featuredistributions,” Pattern Recognition (1999), in press.

[25] A.R. Rao and G.L. Lohse, “Towards a texture naming system: identifying relevant dimen-sions of texture,” Proc. IEEE Conf. Visualization, San Jose, Calif., 1993, 220-227.

[26] G. Smith and I. Burns, “Measuring texture classification algorithms,” Pattern RecognitionLetters 18 (1997) 1495-1501. http://www.cssip.elec.uq.edu.au/~guy/meastex/meastex.html.

[27] M. Soriano, T. Ojala and M. Pietikäinen, “Robustness of Local Binary Pattern (LBP)operators to tilt-compensated textures”,Proc. Workshop on Texture Analysis in MachineVision, Oulu, Finland, 1999.

[28] H. Tamura, S. Mori and T. Yamawaki, “Textural features corresponding to visual percep-tion,” IEEE Transactions on Systems, Man, and Cybernetics 8 (1978) 460-473.

[29] M. Tuceryan and A.K. Jain, “Texture analysis,” in Handbook of Pattern Recognition andComputer Vision, Second Edition, eds. C.H. Chen, L.F. Pau, P.S.P. Wang, World Scientific,Singapore, 1999, pp. 207-248.

[30] K. Valkealahti and E. Oja, “Reduced multidimensional cooccurrence histograms in tex-ture classification,“ IEEE Transactions on Pattern Analysis and Machine Intelligence 20(1998) 90-94.