A Two Level Approach for Scene Recognition

Le Lu Kentaro Toyama Gregory D. Hager

Computer Science Department Microsoft Research Computer Science DepartmentJohns Hopkins University One Microsoft Way Johns Hopkins University

Baltimore, MD 21218 Redmond, WA 98052 Baltimore, MD 21218

AbstractClassifying pictures into one of several semantic cat-

egories is a classical image understanding problem. Inthis paper, we present a stratified approach to both binary(outdoor-indoor) and multiple category of scene classifica-tion. We first learn mixture models for 20 basic classes oflocal image content based on color and texture information.Once trained, these models are applied to a test image, andproduce 20 probability density response maps (PDRM) in-dicating the likelihood that each image region was producedby each class. We then extract some very simple featuresfrom those PDRMs, and use them to train a bagged LDAclassifier for 10 scene categories. For this process, no ex-plicit region segmentation or spatial context model are com-puted.

To test this classification system, we created a labeleddatabase of 1500 photos taken under very different envi-ronment and lighting conditions, using different cameras,and from 43 persons over 5 years. The classification rateof outdoor-indoor classification is93.8%, and the classifi-cation rate for 10 scene categories is90.1%. As a byprod-uct, local image patches can be contextually labeled intothe 20 basic material classes by using Loopy Belief Propa-gation [33] as an anisotropic filter on PDRMs, producingan image-level segmentation if desired.

1 IntroductionClassifying pictures into semantic types of scenes [24,

26, 22] is a classical image understanding problem whichrequires the effective interaction of high level semantic in-formation and low level image observations. Our goal isto build a very practical prototype for scene classificationof typical consumer photos, along the lines of the Kodaksystem [22]. Thus, we are interested in systems that are ac-curate, efficient, and which can work with a wide range ofphotos and photographic quality.

Given the extremely large within-category variations intypical photographs, it is usually simpler and thus easier tobreak the problem of scene classification into a two-stepprocess. In this paper, we first train local, image patch

based color-texture Gaussian Mixture models (GMM) todetect each of 20 materials in a local image patch. Thesemodels are used to scan an image and generate 20 local re-sponses for each pixel. Each response map, called a Prob-ability Density Response Map (PDRM), can be taken as areal-valued image indicating the relative likelihood of eachmaterial at each image location. We then compute momentsfrom the response maps and form a feature vector for eachphoto. By employing the random subspace method [12, 28]and bootstrapping [31], we obtain a set of LDA scene clas-sifiers over these feature vectors. These classification re-sults are combined into the final decision through bagging[2]. After learning the local and global models, a typical1200 × 800 image can be classified in less than 1 secondwith our unoptimized Matlab implementation. Thereforethere is a potential to develop a real-time scene classifierupon our approach. A complete diagram of our approach isshown in Figure 1.

There are several related efforts in this area. Luo et al.[19, 22] propose a bottom-up approach to first find and labelwell-segmented image regions, such as water, beach, sky,and then to learn the spatial contextual model among re-gions. A Bayesian network codes these relational depen-dencies. By comparison, we do not perform an explicitspatial segmentation, and we use relatively simple (LDA-based) classification methods. Perona et al. [8, 30] presenta constellation model of clustered feature components forobject recognition. Their method works well for detectingsingle objects, but strongly depends on the performance andreliability of the interest detector [13]. In the case of sceneclassification, we need to model more than one class of ma-terial, where classes are non-structural and do not have sig-nificant features (such as foliage, rock and et al.) [13]. Thismotivates our use of a GMM on the feature space. In orderto maintain good stability, we estimate the GMM in a lin-ear subspace computed by LDA. These density models arequite flexible and can be used to model a wide variety ofimage patterns with a good compromise between discrimi-nation and smoothness.

Kumar et al. [14, 15] propose the use of Markov random

1

field (MRF)-based spatial contextual models to detect man-made buildings in a natural landscape. They build a multi-scale color and textual descriptor to capture the local depen-dence among building and non-building image blocks anduse MRF to model the prior of block labels. In our work,we have found that simple local labeling suffices to gener-ate good classification results; indeed regularization usingloopy belief propagation method [33] yields no significantimprovement in performance. Thus, we claim that there isno need to segment image regions explicitly for scene clas-sification as other authors have done [22, 19, 15].

Linear discriminant analysis (LDA) is an optimizationmethod to compute linear combinations of features thathave more power to separate different classes. For texturemodeling, Zhu et al [35] pursue features to find the mar-ginal distributions which are also the linear combinationsof the basic filter banks, but they use a much more com-plex method (Monte Carlo Markov Chain) to stochasticallysearch the space of linear coefficients. In our case, the goalis not to build a generative model for photos belonging todifferent scenes, but simply to discriminate among them.We show a simple method such as LDA, if designed prop-erly, can be very effective and efficient to build a useful clas-sifier for complex scenes.

We organize the rest of the paper as follows. In sec-tion 2, we present the local image-level processing usedto create PDRMs. In section 3, we describe how PDRMsare processed to perform scene classification. Experimen-tal results and analysis on the performance of patch basedmaterial detector and image based scene classification on adatabase of 1500 personal photos taken by 43 users usingtraditional or digital cameras over the last 5 years are givenin section 4. Finally we summarize the paper and discussthe future work in section 5.

2 Local Image-Level ProcessingThe role of image-level processing is to roughly classify

local image content at each location in the image. The gen-eral approach is to compute feature vectors of both colorand texture, and then develop classifiers for these features.In our current implementation, we have chosen to performsupervised feature classification. Although arguably lesspractical than corresponding unsupervised methods, super-vised classification permits us to control the structure of therepresentations built at this level, and thereby to better un-derstand the relationship between low-level representationsand overall system performance.

In this step, we compute 20 data driven probabilistic den-sity models to describe the color-texture properties of imagepatches of 20 predefined materials1. These 20 categories

1The vocabulary of materials to be detected is designed by consideringtheir popularity in the usual family photos. This definition is, of course,not unique or optimized.

are: building, blue sky, bush, other (mostly trained with hu-man clothes), cloudy sky, dirt, mammal, pavement, pebble,rock, sand, skin, tree, water, shining sky, grass, snow, car-pet, wall and furniture.

To prepare the training data, we manually crop image re-gions for each material in our database, and randomly drawdozens of 25 by 25 pixel patches from each rectangle. Al-together, we have 2000 image patches for each material.Some examples of the cropped images and sampled imagepatches are shown in Figure 2. For simplicity, we do notprecisely follow the material boundaries in the photos whilecropping. Some outlier features are thus included in thetraining patches. Fortunately these outliers are smoothednicely by learning continuous mixture density models.

Multi-scale image representation and automatic scale se-lection problem has been a topic of intense discussion overthe last decade [17, 20, 13, 6, 14]. In general, the approachof most authors has been to first normalize images with re-spect to the estimated scale of local image regions beforelearning. However it is not a trivial problem to reliably re-cover the local image scales for a collection of 1500 familyphotos. We instead choose to train the GMM using the rawimage patches extracted directly from the original pictures.For the labeled image patches with closer and coarser views,their complex color-texture distributions can will be approx-imated by a multi-modal Gaussian mixture model duringclustering.

2.1 Color-Texture Descriptor for Image PatchesOur first problem is to extract a good color-texture de-

scriptor which effectively allows us to distinguish the ap-pearance of different materials. In the domain of color, ex-perimental evaluation of several color models has not indi-cated significant performance differences among color rep-resentations. As a result, we simply represent the color ofan image patch as the mean color in RGB space.

There are also several methods to extract texture featurevectors for image patches. Here we consider two: filterbanks, and the Haralick texture descriptor. Filter banks havebeen widely used for 2 and 3 dimensional texture recogni-tion. [16, 5, 27]. We apply theLeung-Malik (LM) filterbank [16] which consists of 48 isotropic and anisotropicfilters with 6 directions, 3 scales and 2 phases. Thus, eachpatch is represented by a 48 component feature vector.

The Haralick texture descriptor [10] is designed for im-age classification and has been adopted in the area of im-age retrieval [1]. Haralick texture measurements are de-rived from the Gray Level Co-occurrence Matrix (GLCM).GLCM is also called theGrey Tone Spatial DependencyMatrix which is a tabulation of how often different combi-nations of pixel brightness values (grey levels) occur in animage region. GLCM texture considers the relation betweentwo pixels at a time, called the reference and the neighborpixel. Their spatial relation can be decided by two fac-

2

image patches for blue sky

image patches for tree

image patches for pavement

Labeled Image Patches for each of 20 materials

Patch-based Discriminative Gaussian Mixture Density Model

for each of 20 materials

Patc

h B

ased C

olo

r-Textu

re

Featu

re E

xtra

ctio

n

LD

A P

roje

ctio

n fo

r Mate

rial

Cla

sses

GMM for pavement

GMM for tree

GMM for blue sky

PDRM for pavement

PDRM for tree

PDRM for blue sky

Probability Density Response Maps for each of 20 materials

Mom

ents

Featu

re E

xtra

ctio

n a

nd

Vecto

rizatio

n o

f Each P

DR

M

LD

A P

roje

ctio

n fo

r Scene

Cate

gorie

s (B

oots

trappin

g+

Random

S

ubspace S

am

plin

g)

Baggin

g o

f LD

A C

lassifie

rs fo

r S

cene C

ate

gorie

s

Patch-Level Processing Image-Level Procesing

Figure 1: The diagram of our two level approach for scene recognition. The dashed line boxes are the input data or output learnedmodels; the solid line boxes represent the functions of our algorithm.

Figure 2:(a, c, e, g) Examples of cropped subimages of building, building under closer view, human skin, and grass respectively. (b, d, f,h) Examples of image patches of these materials including local patches sampled from the above subimages. Each local image patch is 25by 25 pixels.

tors, the orientation and offset. Given any image patch, wesearch all the pixel pairs satisfying a certain spatial relationand record their second order gray level distributions witha 2 dimensional histogram indexed by their brightness val-ues2. Haralick also designed 14 different texture features[10] based on the GLCM. We selected 5 texture featuresincluding dissimilarity, Angular Second Moment (ASM),mean, standard deviation (STD) and correction. Definitionsfor these can be found in Appendix A.

There is no general argument that the filter bank featuresor Haralick feature is a better texture descriptor. We eval-uate their texture discrimination performances experimen-

2The reference and neighbor pixel intensities normally need to be quan-tized into 16 or less levels instead of 256 which results in not too sparseGLCM.

tally in section 4 and find Haralick features generally per-form better.2.2 Discriminative Mixture Density Models for 20

MaterialsThe color and texture features for 2000 image patches

form, in principle, an empirical model for each material.However, classifying new patches against the raw featureswould require the solution to a high-dimensional nearest-neighbor problem, and the result would be sensitive to noiseand outliers. Instead, we compute a continuous membershipfunction using a Gaussian mixture model.

Although we have 2000 training samples, our featurevectors have 40 dimensions, so the training set is still toosparse to learn a good mixture model without dimensionalreduction. Because one of our purposes is to maximize the

3

discrimination among different materials, Linear Discrim-inant Analysis (LDA) [31] was chosen to project the datainto a subspace where each class is well separated. TheLDA computation is reviewed in appendix B.

When each class has a Gaussian density with a commoncovariance matrix, LDA is the optimal transform to sepa-rate data from different classes. Unfortunately the materialcolor-texture distributions all have multiple modes becausethe training image patches are sampled from a large varietyof photos. Therefore we have two options: employ LDA todiscriminate among 20 material classes; or use LDA to sep-arate all the modes of materials. Although the latter seemscloser to the model for which LDA was designed, we foundits material classification rate is worse because the optimalseparation among the multiple modes within the same ma-terial class is irrelevant. Therefore we choose the former.

The LDA computation provides a projection of the origi-nal feature space into a lower-dimensional feature spaceZ.We assume that the color-texture features of each materialclass is described by a finite mixture distribution onZ ofthe form

P (z|c) =gc∑

k=1

πckG(z; µc

k, Σck), c = 1, 2, ..., 20 (1)

where theπck are the mixing proportions (

∑gc

k=1 πck = 1)

and G(z;µck,Σc

k) is a multivariate Gaussian function de-pending on a parameter vectorθc

k. The number of mix-turesgc and the model parameters{πc

k, θck} for each ma-

terial classc are initialized by spectral clustering [21] andlearned in an iterative Expectation-Maximization manner[31, 7] wheregc ranged from 4 to 8 depending on the mate-rial class. As a summary, discriminative Gaussian mixturemodels are obtained by applying LDA across the materialclasses and learning the GMM within each material class,respectively.

3 Global Image ProcessingOnce we obtain 20 Gaussian mixture models

{πik, P (z; θi

k), i = 1, 2, ..., 20} for 20 material classes,we can evaluate the membership density values of imagepatches for each material class. For any given photo, wescan local image patches, extract their color-texture featurevector, normalize each of its components from 0 to 1 [1],project it to the lower dimensional subspaceZ computedby LDA, and finally compute the density value given byequation (1) for all 20 material classes. The result is 20real-valued grid maps3 representing membership supportfor each of the 20 classes. An example is shown in Figure3. Two examples of the local patch labeling for indoor andoutdoor photos are shown in Figure 4.

Our next goal is to classify the photos into one of ten

3The size of the map depends on the original photo size and the patches’spatial sampling intervals.

Skin

Skin

Bush

Furniture

Wall&Curtain

Pebbel

Blue Sky

Water

Pavement

Sand

Rock

Bush

Grass

Other

Building

(a) (b)

Figure 4: (a) The local patch material labeling results of an in-door photo. (b) The local patch material labeling results of anoutdoor photo. Loopy belief propagation is used for enhancement.The colored dots represent the material label and the boundariesare manually overlayed for illustration purpose only.

categories: cityscape, landscape, mountain, beach, snow,other outdoors, portrait, party, still life and other indoor. Inorder to classify photos, we must still reduce the dimensionof the PDRMs to a manageable size. To do this, we computethe zeroth, first, and second order moments of each PDRM.Intuitively, the zeroth moment describes the prevalence of agiven material class in an image; the first moment describeswhere it occurs, and the second moment its spatial ”spread”.The moment features from the 20 PDRMs are combined ina global feature vectorY.

Using the scene category labels of the training photos,we now compute the LDA transform that attempts to sep-arate the training feature vectors of different categories.For the indoor-outdoor recognition, the LDA projected sub-space has only one dimension. As a typical pattern classifi-cation problem, we can find the optimal decision boundaryfrom the training data and apply it to the other testing data.Finding decision boundaries for 10 scene category recog-nition is more complex. In practice, it is very difficult totrain a GMM classifier because of the data is too sparse overthe 10 categories. As a result, we have used both the near-est neighbor and Kmeans [31] classifiers for this decisionproblem.

We have found that the standard method for creating anLDA classifier works well for indoor-outdoor scene clas-sification, but the classification results for 10 scene cate-gories is not good enough to constitute a practical proto-type. To improve the classification rate, we have imple-mented variations on random subspace generation [12, 28]and bootstrapping [31] to create multiple LDA classifiers.These classifiers are combined using bagging [2]. Recallthat LDA is a two step process that first computes the singu-lar value decomposition (SVD) [9] of the within-class scat-ter matrix SW , then, after normalization, computes SVDon the between-class scatter matrixS

′B . After the first step,

SW is divided into the principal subspaceSP of the nonzeroeigenvaluesΛP and their associated eigenvectorsUP , andthe null subspaceSN with the zero eigenvaluesΛN and cor-responding eigenvectorsUN . In the traditional LDA trans-form, only SP is used for the whitening ofSW and nor-

4

Figure 3:(a) Photo1459#. (b) Its confidence map. (c, d, e, f, g) Its support maps of blue sky, cloud sky, water, building and skin. Onlythe material classes with the significant membership support are shown.

malization ofSB while SN is discarded (see equation 10 inAppendix B). Chen et al. [4] have found that the null sub-spaceSN satisfyingUT

P SW UP = 0 also contains importantdiscriminatory information. Here we make use of this ob-servation by uniformly sampling an eigenvector matrixUr

from {UP ∪UN} and use it in place ofU in the initial LDAprojection step. Several projections (including the originalLDA projection matrix) are thus created.

In the second step of LDA, the subsetVP of the fulleigenvector matrixV with the largest eigenvalues, nor-mally replacesV in equation (10). It is also possible thatthere is useful discriminative information in the subspace{V − VP }. Therefore we employ a similar sampling strat-egy as [28] in the context of PCA by first sampling a smallsubset of eigenvectorsVr of {V − VP }, then replacingVwith the joint subspace{VP ∪ Vr} in equation 10.

Finally we also perform bootstrapping [31] by samplingsubjects of the training set and creating LDA classifiersfor these subsets. By the above three random samplingprocesses, we learn a large set of LDA subspaces and classi-fiers which we combine using the majority voting (bagging)methods [2]. In Section 4, we show the bagged recognitionrates of 20 classifiers from bootstrapping replicates and 20from random subspace sampling.

4 ExperimentsOur photo collection currently consists of 540 indoor

and 860 outdoor customer photos. We randomly select halfof them as the training data and use other photos as thetesting data. We have also intentionally minimized redun-dancy when collecting photos, i.e., only one photo is se-lected when there are several similar pictures.

We first address the problem of the image patch basedcolor-texture feature description and classification. Com-parison of the recognition rates of 1200 testing imagepatches for each material class for different color-texturedescriptors, different numbers of training patches and dif-ferent classifiers is provided in Figure 6 (a,b). In partic-ular, we have also benchmarked the LDA+GMM modelagainst a brute-force nearest neighbor classifier. Letxj andzj represent an image patch feature vector before and af-ter the LDA projection, respectively. The nearest neighborclassifier computes the class label of a testing patchj asthe label of that training patchl such that‖xj − xl‖ =mini{‖xj − xi‖} wherei ranges over the training imagepatches of all material classes. The GMM classifier simply

building

blue sky

bush

other

c-sky

dirt

mammal

pavement

pebble

rock

sand

skin

tree

water

s-sky

grass

snow

carpet

furniture

wall

Figure 5: The pairwise confusion matrix of 20 materialclasses. The indexing order of the confusion matrix isshown on the left of the matrix. The indexing order is sym-metrical.

chooses the maximal class density, i.e. the classc∗ such thatP (zj |c∗) = maxc=1,2,...,20{P (zj |c)}.

Comparing the plots shown in Figure 6, the classifierbased on the Maximum Likelihood of GMM density func-tions outperforms the Nearest Neighbor classifier, thus val-idating the use of the LDA+GMM method. We also com-pared the recognition rates of 4 different feature combina-tions and found that the Haralick texture descriptor com-bined with the mean color of the image patch yields the bestresults. Finally, in Figure 6 (b), we see that the LDA+GMMmethod improves the recognition rate significantly when in-creasing the training image patch from 500, becoming sta-ble after 2000 patches.

Figure 5 shows the confusion rate using the GMM clas-sifiers learned from 2000 training image patches per class.The size of the white rectangle in each grid is proportionalto the pairwise recognition error ratio. The largest andsmallest confusion rates are23.6% and0.24%, respectively.From Figure 5, we see that pebble, rock and sand classesare well separated which shows that our patch-level learn-ing process achieves a good balance of Haralick textureand color cues by finding differences of the material classeswith the similar color. There is significant confusion amonggrass, bush and tree due to their similar color and texturedistribution. For some material classes, such as furniture,carpet, and other, the overall confusion rates are also high.

For global classification, we have found that first order

5

(a) (b)

Figure 7:(a) An misclassified indoor photo. (b) An misclassifiedoutdoor photo.

moment features of PRDMs are useful in outdoor scenes,but reduce the recognition rate for indoor scenes. Thismakes sense since in most outdoor scenes spatial contex-tual constraints, for instance the sky above grass, are usefulcues. This naturally suggests a hierarchical classificationscheme (first determine indoor/outdoor followed by catego-rization), however we have not yet pursued this approach.Thus, we confine ourselves to zeroth order moments for theremainder of this paper.

Our global image moment features after LDA projectionare very easy to visualize in the indoor/outdoor case as theybecome points in a 1-dimensional LDA subspace (6 (c)).In this case, the 1-D indoor-outdoor decision boundary issimply determined by fitting a scaled exponential functionto each of the indoor or outdoor histogram distributions andcalculating the point of intersection.

We show the recognition results of our method in Fig-ure 6 (d), compared with the direct low-level color or tex-ture based scene recognition methods4 without LDA learn-ing as the baselines. Our indoor-outdoor recognition rateis 93.8%, which is comparable or slightly better than theKodak’s recently published classification system [22], al-though our approach is tested on a40% larger photo data-base. It is interesting that the bagging algorithm does notsignificantly improve the recognition performance of forindoor-outdoor classification. The likely explanation is thatthe individual indoor-outdoor LDA classifiers have nearlyachieved the best possible recognition rate. Figure 7 shows2 examples of misclassified photos. The first photo con-sists of a person sitting indoors, but in front of a curtainof tree leaves. In the second, the playground is incorrectlyclassified as ”carpet” not ”dirt”. The appearance of peopleand animals are irrelevant for indoor-outdoor classification— their associated moment features are assigned with nearzero weights.

As shown in Figure 6 (e), feature points of some scenecategories are well separated from others and thus easy to

4We divide each image as a 9 by 9 grid, and extract the mean coloror the DOG (Derivative of Gaussian) filtered texture features within eachgrid. Each photo is then formulated as a feature vector by combining cuesin all grids. A nearest neighbor classifier is later employed for recognitionbased on the feature vectors’ distances of the training and testing photos.

be recognized in a certain LDA subspace, while some cat-egories are not. Fortunately, Figure 6 (f) demonstrates thatthe individual LDA classifiers capture the complimentarydiscriminative information in different random subspaces.Finally, it results that the combined (nearest neighbor andKmeans) classifiers both show improved performances of6 − 10% on average. As a comparison, Boutell et al. [3]achieve less than80% classification accuracy for 923 im-ages in 5 categories. In their work, model-based graphmatching techniques are used to learn the explicit sceneconfiguration consisting of semantic image regions.

5 Conclusions & DiscussionThis paper makes three contributions. First, we propose

an efficient, yet effective, approach for scene recognition forboth indoor-outdoor and multiple photo categories. In prac-tice, this approach can handle the photos’ spatial complexityboth in the local patch-level and the global image-level suc-cessfully. All the training and testing processes are basedupon a challenging photo database. Second, we describea combination of LDA and Gaussian mixture models thatachieves a good balance of discrimination and smoothness.Finally, we study the use of moment features of PDRMs asan effective image-level representation for scene classifica-tion, and the bagging [2] method to combine the individ-ual scene classifiers obtained by the random subspace algo-rithm [12]. The bagging method has shown success in ourexperiments, especially for 10 category scene recognition.

Although we have used supervised methods to create thelocal image patch classifiers, a practical system would likelearn at least some of these classifiers using unsupervisedmethods. However we believe that the supervised materialdetectors provide the best scene recognition performance,and as such provide a “benchmark” against which unsuper-vised methods can be evaluated. In future work, we intendto investigate unsupervised clustering methods for low-levelimage patch classification. In particular, we plan to applyour unsupervised, iterative LDA-GMM algorithm [18]. Wealso plan to investigate a hybrid approach where classifiedimages are used as labeled data to compute an initial LDAprojection, which is then subsequently refined with new, un-labeled images using iterative LDA-GMM. Finally, becauseLDA is only optimal when each class has a Gaussian den-sity with a common covariance matrix, the non-parametricdiscriminant analysis (proposed in [34]) will be tested asa means to generalize our approach to a more comprehen-sive image database which may contain thousands of vari-ous kinds of photos.

AppendicesA The GLCM

Let us denote GLCM aN×N matrixPi,j whereN is thequantized level of pixel intensity andi, j = 0, 1, ..., N − 1.The diagonal elements(i = j) all represent pixel pairs with

6

500 1000 1500 2000 25000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Comparison of image patch recogniton of 4 features via Near−Neighbor Classifier

The number of training image patches

Rec

ogni

tion

rate

Filter BanksHaralickFilter Banks+ColorHaralick+Color

500 1000 1500 2000 25000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Comparison of image patch recogniton of 4 features via GMM Classifier

Rec

ogni

tion

rate

The number of training image patches

Filter BanksHaralickFilter Banks+ColorHaralick+Color

−6 −5 −4 −3 −2 −1 0 1 2 30

2

4

6

8

10

12

14

16

18

20

The Histogram of the LDA Features of the Trainging Indoor and Outdoor Photos in a One Dimensional Subspace

The Bins of the LDA Features

The

Num

ber

of P

hoto

s

indooroutdoor

(a) (b) (c)

0 5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Comparison of Indoor−outdoor Scene Recognition

The Number of Classifiers

Rec

ogni

tion

Rat

e

Bagged LDA ClassifierRandom LDA ClassifiersColor based NN ClassifierTexture based NN Classifier

−8−6

−4−2

02

46

−6

−4

−2

0

2

4

6

−20

−10

0

10

Dimension One

The First 3 Dimensions of the Random Subspace LDA Features of 10 Scene Categories

Dimension Two

Dim

ensi

on T

hree

in1in2in3in4ot1ot2ot3ot4ot5ot6

0 5 10 15 20 25 30 35 400.7

0.72

0.74

0.76

0.78

0.8

0.82

0.84

0.86

0.88

0.9

Comparison of 10 Category Scene Recognition

The Number of Classifiers

Rec

ogni

tion

Rat

e

Bagged LDA Classifier with NNRandom LDA Classifiers with NNBagged LDA Classifier with KmeansRandom LDA Classifier with Kmeans

(d) (e) (f)Figure 6: (a) Comparison of the image patch based recognition of 4 kinds of features (filter banks feature, Haralick texture feature andtheir joint features with color) via Nearest-Neighbor Classifier. (b) Comparison of the image patch based recognition of 4 kinds of featuresvia GMM Classifier. (c)The 1D feature histogram distributions of indoor-outdoor photos after LDA projection. (d) The comparison ofindoor-outdoor recognition rates of 4 methods. (e) The first 3D feature point distributions of 10 category photos after LDA projection. (f)The comparison of 10 categories recognition rates of 4 methods.

no grey level difference; while the off-diagonal cells(i 6= j)represent pixel pairs with dissimilarity|i − j| increasinglinearly away from the diagonal. Therefore we havedissimilarity =

∑N−1i,j=1 (P (i, j)× |i− j|). Furthermore

ASM =∑N−1

i,j=1 P (i, j)2 measures the uniformity of the

distribution of GLCM. µi =∑N−1

i,j=1 (P (i, j)× i|)and µj =

∑N−1i,j=1 (P (i, j)× j|) are the means

of the reference pixels or neighbor pixels. Sim-

ilarly, σi =√∑N−1

i,j=1 (P (i, j)× (i− µi)2) and

σj =√∑N−1

i,j=1 (P (i, j)× (j − µj)2) are the re-spective standard deviations, andcorrelation =∑N−1

i,j=1 (P (i, j)× (i− µi)(j − µj))/(σi × σj). Ifthe above means and standard deviations are calcu-lated from symmetrical GLCM,µ = µi = µj andσ = σi = σj . Finally the output of 5 Haralick featuresare {dissimilarity, ASM,µ, σ, correlation} for eachGLCM 5.

5Note that we choose the pair of reference and neighbor pixels accord-ing to 4 directions (45 degree each) and 1 or 2 pixel offsets. Thereforewe have 8 GLCMs for any image patch which results in a 40 componentfeature vector.

B LDA

The following objective function

J(ω) =ωT SBω

ωT SW ω(2)

is maximized by solving a generalized eigenvector equation

SBω = λSW ω (3)

where

SW =1M

C∑

i=1

M∑

j=1

zij(Xj −mi)(Xj −mi)T (4)

SB =C∑

i=1

Mi

M(mi −m)(mi −m)T (5)

Denote that SB and SW are respectively named thebetween-class or within-class scatter matrix,xj is a featurevector,mi is the mean of classi andm is the global meanof the dataX, i = 1...C is a class number (C is the totalnumber of classes) and the binary membership function

zij ={

1, if xj ∈ class i0, otherwise

(6)

The LDA algorithm firstly perform the singular value de-

7

composition (SVD) ofSW

SW = UΛUT (7)

then transformSB into

S′B = Λ−

12 UT SBUΛ−

12 (8)

and compute the eigenvectors of

S′BV = VΛ (9)

whereΛ is the diagonal matrix of eigenvalues ofS′B . The

optimal feature vectorsZ are therefore

Z = AT X (10)

through the projected transformAT = VT Λ−12 UT . For

dimension reduction, only the subset of eigenvectorsV andU with large eigenvalues are used in the transform. Thedimension of the LDA projected subspace is at mostC − 1.

References[1] S. Aksoy and R. Haralick, Feature Normalization and

Likelihood-Based Similarity Measures for Image Retrieval,Pattern Recognition Letters, 22(5):563-582, 2001.

[2] L. Breiman, Bagging Predictors, Machine Learning,24(2):123-140, 1996.

[3] M. Boutell, J. Luo and C. Brown, Learning spatial configu-ration models using modified Dirichlet priors,Workshop onStatistical Relational Learning, 2004.

[4] L. Chen, H. Liao, M. Ko, J. Lin, and G. Yu, A new LDA-basedface recognition system which can solve the small sample sizeproblem,Journal of Pattern Recognition, 33(10):1713-1726,2000.

[5] O. Cula and K. Dana, Compact representation of bidirectionaltexture functions,CVPRI:1041-1047, 2001.

[6] J. De Bonet, P. Viola, A non-parametric multi-scale statisticalmodel for natural images,NIPS, 1997.

[7] A. Dempster, N. Laird and D. Rubin, Maximum likelihoodfrom incomplete data via the EM algorithm,Journal of theRoyal Statistical Society, Series B, 39:1-38, 1977.

[8] R. Fergus, P. Perona and A. Zisserman, Object class recogni-tion by unsupervised scale-invariant learning,CVPR, 2003.

[9] G. Golub and C. Van Loan,Matrix Computations, JohnsHopkins University Press, 1996.

[10] R. Haralick, K. Shanmugam, and I. Dinstein, Texture fea-tures for image classification.IEEE Trans. on System, Manand Cybernatic, 1973.

[11] X. He, R. Zemel and M. Carreira-Perpinan, Multiscale Con-ditional Random Fields for Image Labeling,CVPR, 2004.

[12] T. K. Ho, The random subspace method for constructingdecision forests,IEEE Trans. on PAMI, 20(8):832-844, 1998.

[13] T. Kadir and M. Brady, Scale, saliency and image descrip-tion, IJCV, 45(2):83105, 2001.

[14] S. Kumar and M. Hebert, Man-made structure detection innatural images using a causal multiscale random field,CVPR,1:119-126, 2003.

[15] S. Kumar, A. C. Loui and M. Hebert, An Observation-Constrained Generative Approach for Probabilistic Classi-fication of Image Regions, Image and Vision Computing,21:87-97, 2003.

[16] T. Leung and J. Malik, Representing and recognizing the vi-sual appearance of materials using three-dimensional textons,IJCV, 2001.

[17] T. Lindeberg, Principles for automatic scale selection,Hand-book on Computer Vision and Applications, 2:239–274, Aca-demic Press, Boston, 1999.

[18] Authors, Authors’ paper, to appearNIPS, 2004.

[19] J. Luo, A. Singhal, S. Etz, and R. Gray, A computationalapproach to determination of main subject regions in pho-tographic images,Image Vision Computing, 22(3):227-241,2004.

[20] J. Malik, S. Belongie, T. Leung and J. Shi Contour and tex-ture analysis for image segmentation,Int Journal of Com-puter Vision, 43(1):7-27, 2001.

[21] A. Ng, M. Jordan and Y. Weiss, On Spectral Clustering:Analysis and an algorithm,NIPS, 2001.

[22] N. Serrano, A. Savakis and J. Luo, Improved scene classi-fication using efficient low-level features and semantic cues,Pattern Recognition37(9):1773-1784, 2004.

[23] A. Singhal, J. Luo and W. Zhu, Probabilistic Spatial ContextModels for Scene Content Understanding,CVPR, 2003.

[24] M. Szummer and R. W. Picard, Indoor-outdoor image clas-sification, IEEE Int. Workshop Content-Based Access ImageVideo Databases, 1998.

[25] A. Torralba, Contextual priming for object detection,IntJournal of Computer Vision, 53(2):169-191, 2003.

[26] A. Vailaya, M. Figueiredo, A. Jain and H.-J. Zhang, Imageclassification for content-based indexing,IEEE Trans. ImageProcessing, 10(1):117-130, 2001.

[27] M. Varma and A. Zisserman, Classifying images of ma-terials: achieving viewpoint and illumination independence,ECCV, 2002.

[28] X. Wang and X. Tang, Random sampling LDA for facerecognition,CVPR, 2004.

[29] X. Wang and X. Tang, Dual-space linear discriminant analy-sis for face recognition,CVPR, 2004.

[30] M. Weber, M. Welling and P. Perona, unsupervised learningof models for recognition,ECCV, 2000.

[31] A. Webb, Statistical Pattern Recognition, John Wiley &Sons, 2002.

8

[32] Y. Wu, Q. Tian, T. Huang, Discriminant-EM algorithm withapplication to image retrieval,CVPR, I:222-227, 2000.

[33] J. Yedidia, W. T. Freeman and Y. Weiss, Understanding be-lief propagation and its generalizations,IJCAI, 2001.

[34] M. Zhu and T. Hastie, Feature extraction for non-parametricdiscriminant analysis,JCGS, 12(1):101-120, 2003.

[35] S.C. Zhu, Y.N. Wu, and D. Mumford, Minimax entropy prin-ciple and its applications to texture modeling,Neural Com-putation, 9:1627-1660, 1997.

9