05.Tr862.Probabilistic Modeling for Semantic Scene Classification

8/14/2019 05.Tr862.Probabilistic Modeling for Semantic Scene Classification

1/115

1

Probabilistic Modeling for Semantic Scene Classification

Matthew R. Boutell

URCS Technical Report 862

May, 2005

This thesis was proposed in April, 2003. The dissertation of the thesis will

be published in May, 2005. While the dissertation subsumes and modifies much

of the material in this proposal, I have made it available (as URCS TR 862)

as a historical supplement to document the details and results of the MASSES

(Material and Spatial Experimental Scenes) prototype.


2/115

Probabilistic Modeling for SemanticScene Classification

by

Matthew R. Boutell

Thesis Proposal

for the Degree

Doctor of Philosophy

Supervised by

Christopher M. Brown

Department of Computer Science

The College

Arts and Sciences

University of Rochester

Rochester, New York

2005


3/115

ii

Abstract

Scene classification, the automatic categorization of images into semantic classes

such as beach, field, or party, is useful in applications such as content-based im-

age organization and context-sensitive digital enhancement. Most current scene-

classification systems use low-level features and pattern recognition techniques;

they achieve some success on limited domains.

Several contemporary classifiers, including some developed in Rochester, in-

corporate semantic material and object detectors. Classification performance im-

proves because because the gap between the features and the image semantics is

narrowed. We propose that spatial relationships between the objects or materials

can help by distinguishing between certain types of scenes and by mitigating the

effects of detector failures. While past work on spatial modeling has used logic-

or rule-based models, we propose a probabilistic framework to handle the loose

spatial relationships that exist in many scene types.

To this end, we have developed MASSES, an experimental testbed that can

generate virtual scenes. MASSES can be used to experiment with different spatial

models, different detector characteristics, and different learning parameters. Using

a tree-structured Bayesian network for inference on a series of simulated natural

scenes, we have shown that the presence of key materials can effectively distinguishcertain scene types. However, spatial relationships are needed to disambiguate

other types of scenes, achieving a gain of 7% in one case.

However, our simple Bayes net is not expressive enough to model the faulty

detection at the level of individual regions. As future work, we propose first to


4/115

iii

evaluate full (DAG) Bayesian networks and Markov Random Fields as potential

probabilistic frameworks. We then plan to extend the chosen framework for our

problem. Finally, we will compare our results on real and simulated sets of images

with those obtained by other systems using spatial features represented implicitly.


5/115

iv

Table of Contents

Abstract ii

List of Tables vii

List of Figures viii

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 The Problem of Scene Classification . . . . . . . . . . . . . . . . . 3

1.2.1 Scene Classification vs. Full-scale Image Understanding . . 4

1.2.2 Scene Classification vs. Not Object Recognition . . . . . . 5

1.3 Past Work in Scene Classification . . . . . . . . . . . . . . . . . . 6

1.4 Statement of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.4.1 Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.5 Summary of Preliminary Work . . . . . . . . . . . . . . . . . . . . 8

1.6 Organization of Proposal . . . . . . . . . . . . . . . . . . . . . . . 9

2 Related Work 10

2.1 Design Space of Scene Classification . . . . . . . . . . . . . . . . . 11

2.1.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11


6/115

v

2.1.2 Learning and Inference Engines . . . . . . . . . . . . . . . 11

2.2 Scene Classification Systems . . . . . . . . . . . . . . . . . . . . . 13

2.2.1 Low-level Features and Implicit Spatial Relationships . . . 13

2.2.2 Low-level Features and Explicit Spatial Relationships . . . 16

2.2.3 Mid-level Features and Implicit Spatial Relationships . . . 19

2.2.4 Semantic Features without Spatial Relationships . . . . . . 20

2.2.5 Semantic Features and Explicit Spatial Relationships . . . 21

2.2.6 Summary of Scene Classification Systems . . . . . . . . . . 22

2.3 Options for Computing Spatial Relationships . . . . . . . . . . . . 23

2.3.1 Computing Qualitative Spatial Relationships . . . . . . . . 23

2.3.2 Computing Quantitative Spatial Relationships . . . . . . . 26

2.4 Probabilistic Graphical Models . . . . . . . . . . . . . . . . . . . 27

2.4.1 Bayesian Networks . . . . . . . . . . . . . . . . . . . . . . 29

2.4.2 Markov Random Fields . . . . . . . . . . . . . . . . . . . . 33

2.4.3 Relative Merits . . . . . . . . . . . . . . . . . . . . . . . . 42

3 Methodology 44

3.1 Statistics of Natural Images . . . . . . . . . . . . . . . . . . . . . 45

3.1.1 Ground Truth Collection Process . . . . . . . . . . . . . . 46

3.1.2 Scene Prototypes . . . . . . . . . . . . . . . . . . . . . . . 48

3.1.3 Spatial Relationships . . . . . . . . . . . . . . . . . . . . . 48

3.1.4 Scene-specific Spatial Relationship Statistics . . . . . . . . 51

3.2 Experimental Environment . . . . . . . . . . . . . . . . . . . . . . 51

3.2.1 Advantages of Working in Simulation . . . . . . . . . . . . 52

3.2.2 MASSES Prototype . . . . . . . . . . . . . . . . . . . . . . 53


7/115

vi

3.2.3 Simulating Faulty Detectors . . . . . . . . . . . . . . . . . 58

3.2.4 Background Regions . . . . . . . . . . . . . . . . . . . . . 64

4 Experimental Results 66

4.1 Best-case Detection in MASSES . . . . . . . . . . . . . . . . . . . 66

4.2 Best-case Detection on Beach Photographs . . . . . . . . . . . . . 68

4.3 Faulty Detection on Beach Photographs . . . . . . . . . . . . . . 70

5 Proposed Research 73

5.1 Research Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5.1.1 Why a New Inference Mechanism? . . . . . . . . . . . . . 76

5.1.2 Bayes Nets Vs. MRFs . . . . . . . . . . . . . . . . . . . . 77

5.1.3 Extend the Chosen Framework . . . . . . . . . . . . . . . 79

5.1.4 Analyze the Effect of Detector Quality . . . . . . . . . . . 82

5.1.5 Evaluate Explicit Spatial Relationships . . . . . . . . . . . 82

5.1.6 Evaluate Semantic Features . . . . . . . . . . . . . . . . . 83

5.1.7 Generalize to Other Scene Classes . . . . . . . . . . . . . . 83

5.1.8 Explore Potential Long-term Directions . . . . . . . . . . . 84

5.2 Issues Not Addressed in This Thesis . . . . . . . . . . . . . . . . . 85

5.3 Research Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

6 Acknowledgments 87

Bibliography 88

A Natural Scene Statistics 98

B Detector Characteristics 103


8/115

vii

List of Tables

2.1 Options for features to use in scene classification. . . . . . . . . . 12

2.2 Potential classifiers to use in scene classification. . . . . . . . . . . 14

2.3 Related work in scene classification, organized by feature type and

use of spatial information. . . . . . . . . . . . . . . . . . . . . . . 15

3.1 Scene class definitions. . . . . . . . . . . . . . . . . . . . . . . . . 47

3.2 Distribution resulting from offline sampling procedure. . . . . . . 64

4.1 MASSES with best-case material detection: Accuracy with and

without spatial information. . . . . . . . . . . . . . . . . . . . . . 67

4.2 MASSES with best-case material detection: Accuracy with and

without spatial information. . . . . . . . . . . . . . . . . . . . . . 69

4.3 MASSES with faulty material detection: Accuracy with and with-

out spatial information. . . . . . . . . . . . . . . . . . . . . . . . . 70


9/115

viii

List of Figures

1.1 Content-ignorant color balancing can destroys the brilliance of sun-

set images, such as those pictured, which have the same global color

distribution as indoor, incandescent-illuminated images. . . . . . . 3

2.1 A Bayes Net with a loop. . . . . . . . . . . . . . . . . . . . . . . . 32

2.2 Portion of a typical two-layer MRF. In low-level computer vision

problems, the top layer (black) represents the external evidence of

the observed image while the bottom layer (white) expresses the a

prioriknowledge about relationships between parts of the underly-

ing scene. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.1 Ground-truth labeling of a beach scene. Sky, water, and sand re-

gions are clearly shown. . . . . . . . . . . . . . . . . . . . . . . . 46

3.2 Prototypical beach scenes. (a) A simple beach scene without back-

ground objects. (b) Because we make no attempt to detect it, we

consider the sailboat to be background. (c) A more complicated

scene: a developed beachfront. (d) A scene from a more distant

field-of-view. (e) A crowded beach. . . . . . . . . . . . . . . . . . 49


10/115

ix

3.3 Prototypical urban scenes. (a) The most common urban scene,

containing sky, buildings, and roads. (b),(c) The sky is not simply

above the buildings in these images. (d) Roads are not necessary.

(e) Perspective views induce varied spatial relationships. (f) Close

views can preclude the presence of sky. . . . . . . . . . . . . . . . 50

3.4 An example of spatial relationships involving a split region. . . . . 50

3.5 The MASSES environment. Statistics from labeled scenes are used

to bootstrap the generative model, which can then produce new

virtual scenes for training or testing the inference module. . . . . 54

3.6 Single-level Bayesian network used for MASSES . . . . . . . . . . 54

3.7 Sampling the scene type yields class C. Then we sample to find

the materials present in the image, in this case, M1, M3, and M4.

Finally, we sample to find the relationships between each pair of

these material regions. . . . . . . . . . . . . . . . . . . . . . . . . 57

3.8 Bayesian network subgraph showing relationship between regions

and detectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.1 Images incorrectly classified due to spatial relationships. . . . . . 69

4.2 Images incorrectly classified using faulty detectors and no spatial

relationships. The actual materials are shown the top row; the

detected materials are shown below each. . . . . . . . . . . . . . . 70

4.3 Images incorrectly classified using faulty detectors. . . . . . . . . 71

4.4 Images incorrectly classified using faulty detectors. . . . . . . . . 72

5.1 Classifier input (labeled regions) and output (classification and con-

fidence). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

5.2 An image in which a region is mis-detected, creating contradictory

spatial relationships in the material-based inference scheme. . . . 76


11/115

x

5.3 Proposed DAG Bayesian network. Note that separate material and

region layers are needed. . . . . . . . . . . . . . . . . . . . . . . . 77


12/115

1

1 Introduction

Semantic scene classification, the process of categorizing images into semantic

classes such as beaches, sunsets or parties is a useful endeavor. As humans, we

can quickly determine the classification of a scene, even without recognizing every

one of the details present. Even the gist of a scene is worth much in terms of

communication.

1.1 Motivation

Automatic semantic classification of digital images finds many applications. We

describe two major ones briefly: content-based image organization and retrieval

(CBIR) and digital enhancement.

With digital libraries growing in size so quickly, accurate and efficient tech-

niques for CBIR become more and more important. Many current systems allow a

user to specify an image and then search for images similar to it, where similarity

is often defined only by color or texture properties. Because a score is computed

on each image in the potentially-large database, it is somewhat inefficient (though

individual calculations vary in complexity).

Furthermore, this so-called query by example has often proven to be return


13/115

2

inadequate results [68]. Sometimes the match between the retrieved and the query

images is hard to understand, while other times, the match is understandable, but

contains no semantic value. For instance, with simple color features, a query for a

rose can return a picture of a man wearing a red shirt, especially if the background

colors are similar as well.

Knowledge of the semantic category of a scene helps narrow the search space

dramatically [37]. If the categories of the query image and the database images

have been assigned (either manually or by an algorithm), they can be exploited

to improve both efficiency and accuracy. For example, knowing what constitutes

a party scene allows us to consider only potential party scenes in our search and

thus helps to answer the query find photos of Marys birthday party. This way,

search time is reduced, the hit rate is higher, and the false alarm rate is expected

to be lower. Visual examples can be found in [76].

Knowledge about the scene category can find also application in digital en-

hancement [73]. Digital photofinishing processes involve three steps: digitizing

the image if necessary (if the original source was film), applying enhancement

algorithms, and outputting the image in either hardcopy or electronic form. En-hancement consists primarily of color balancing, exposure enhancement, and noise

reduction. Currently, enhancement is generic (i.e. without knowledge of the scene

content). Unfortunately, while a balancing algorithm might enhance the quality

of some classes of pictures, it degrades others.

Take color balancing as an example. Photographs captured under incandescent

lighting without flash tend to be yellowish in color. Color balancing removes

the yellow cast. However, applying the same color balancing to a sunset image(containing the same overall yellow color) destroys the desired brilliance (Figure

1.1).

Other images that are negatively affected by color balancing are those con-

taining skin-type colors. Correctly balanced skin colors are important to human


14/115

3

Figure 1.1: Content-ignorant color balancing can destroys the brilliance of sunset

images, such as those pictured, which have the same global color distribution as

indoor, incandescent-illuminated images.

perception [64], and it is important to balance them. However, causing non-skin

objects with similar colors to look like skin is a conspicuous error.

Rather than applying generic color balancing and exposure adjustment to all

images, knowledge of the scenes semantic classification allows us to customize

them to the scene. Following the example above, we could retain or boost sun-

set scenes brilliant colors while reducing a tungsten-illuminated indoor scenes

yellowish cast.

1.2 The Problem of Scene Classification

On one hand, isnt scene classification preceded by image understanding, the holy

grail of vision? What makes us think we can achieve results? On the other hand,

isnt scene classification just an extension of object recognition, for which many

techniques have been proposed with varying success? How is scene classification

different from these two related fields?


15/115

4

1.2.1 Scene Classification vs. Full-scale Image Understand-

ing

As usually defined, image understanding is the process of converting pixels to

predicates: (iconic) image representations to another (symbolic) form of knowl-

edge [2]. Image understanding is the highest (most abstract) processing level in

computer vision [71], as opposed to image processing techniques, which convert

one image representation to another. (For instance, using a mask to convert raw

pixels to an edge image is much more concrete than identifying the expression

on a persons face in the image!) Lower-level image processing techniques such

as segmentation are used to create regions that can then be identified as objects.

Various control strategies are used to order the processing steps and can vary [3].

The end result desired is for the vision to support high-level reasoning about the

objects and their relationships to meet a goal.

While image understanding in unconstrained environments is still very much

an open problem [71; 77], much progress is currently being made in scene clas-

sification. Because scenes can often be classified without full knowledge of every

object in the image, the goal is not as ambitious. For instance, if a person recog-

nizes trees at the top of a photo, grass on the bottom, and people in the middle,

he may hypothesize that he is looking at a park scene, even if he cannot see every

detail in the image. Or on a different level, if there are lots of sharp vertical and

horizontal edges, he may be looking at an urban scene.

It may be possible in some cases to use low-level information, such as color

or texture, to classify some scene types accurately. In other cases, perhaps ob-

ject recognition is necessary, but not necessarily of every object in the scene.

In general, classification seems to be an easier problem than unconstrained im-

age understanding; early results have confirmed this for certain scene types in

constrained environments [74; 77]. Scene classification is a subset of the image


16/115

5

understanding problem, and can be used to ease other image understanding tasks

[75]. For example, knowing that a scene is of a beach constrains where in the

scene one should look for people.

Obtaining image understanding in unconstrained environments is a lofty goal,

and one worthy of pursuit. However, given the state of image understanding, we

see semantic scene classification as a necessary stepping-stone in pursuit of the

grail.

1.2.2 Scene Classification vs. Not Object Recognition

However, scene classification is a different beast than object recognition. Detection

of rigid objects can rely upon geometrical relationships within the objects, and

various techniques [21; 63] can be used to achieve invariance to affine transforms

and changes in scene luminance. Detection of non-rigid objects is less constrained

physically, since the relationships are looser [12]. Scene classification is even less

constrained, since the components of a scene are varied. For instance, while

humans might find it easy to recognize a scene of a childs birthday party, the

objects and people that populate the scene can vary widely, and the cues that

determine the birthday scene class (such as special decorations, articles marked

with the age of the child, and facial expressions on the attenders) can be subtle.

Even the more obvious cues, like a birthday cake, may be difficult to determine.

Again, the areas of scene classification and object recognition are related;

knowing the identity of some of the scenes objects will certainly help to classify the

scene, while knowing the scene type affects the expected likelihood and location

of the objects it contains.


17/115

6

1.3 Past Work in Scene Classification

Most of the current systems primarily use low-level features to classify scenes

and achieve some success on constrained problems. These systems tend to be

exemplar-based, in which features are extracted from images, and pattern recog-

nition techniques are used to learn the statistics of a training set and to classify

novel test images. Very few systems are model-based, in which the expected

configuration of the objects in the scenes is specified by a human expert.

1.4 Statement of Thesis

The limited success of scene classification systems using low-level features forces us

to look for other solutions. Currently, good semantic material detectors and object

recognizers are available [70; 38; 63] and have begun to be successfully applied to

scene classification [37]. However, the presence or absence of certain objects is

not always enough to determine a scene type. Furthermore, object detection is

still developing and is far from perfect. Faulty detection causes brittle rule-based

systems to break.

Our central claim is that Spatial modeling of semantic objects and materials

can be used to disambiguate certain scene types as well as mitigate the effects of

faulty detectors. Furthermore, an appropriate probabilistic inference mechanism

must be developed to handle the loose spatial structure found in real images.

Current research into spatial modeling relies on (fuzzy) logic and subgraph

matching [44; 83]. While we have found no research that incorporates spatial

modeling in a probabilistic framework, we argue that a probabilistic approach

would be more appropriate. First, logic (even fuzzy variants) is not equipped to

handle exceptions efficiently [50], a concern we address in more detail in Section

2.4. Second, semantic material detectors often yield belief in the materials. While


18/115

7

it is not obvious how to use belief values, it seems desirable to exploit the uncer-

tainty in calculating the overall belief in each scene type. A nice side effect of true

belief-based calculation is the ease in which a dont know option can be added

to the classifier: simply threshold the final belief value.

The first interesting problem is the appropriate choice of a probabilistic frame-

work. Both Bayesian networks [66] and Markov Random Fields [30; 16; 15] have

been applied to other vision problems in the past.

We also propose to investigate the effects of spatial modeling. In our experi-

mentation, we plan to compare the following:

1) Baseline (no spatial relationships). Use naive Bayes classification rules usingthe presence or absence of materials only.

2a) Qualitative spatial relationships. Incorporate relations such as above or

beside between regions. This would be appropriate for use in a Bayesian Network

framework.

2b) Quantitative spatial relationships. Use distance and direction between

regions. This may potentially be more accurate, due to the increase in the infor-

mation used, but requires more training data. These may work particularly well

within a Markov Random Field framework.

One of our major foreseen contributions will be to validate the hypothesized

gain due to spatial modeling.

1.4.1 Philosophy

The success of our approach seems to hinge on the strength of the underlying

detectors. Consider two scenarios. First, if the detectors are reasonably accurate,

then we can expect to overcome some faults using spatial relationships. However,

if they are extremely weak, we would be left with a very ambitious goal: from a

very pessimistic view of the world (loose spatial structure and weak detectors),


19/115

8

pull order from the chaos and accurately apply a discrete label to a configuration

of objects.

In this latter case, prior research seems to confirm that the task does not sound

promising. For instance, Selinger found that if an object could be recognized

with moderate success using a single camera view, additional views could improve

recognition substantially. However, if the single view gave weak detection, then

multiple views could not redeem the problem. She states [62] (p. 106):

The result is in concert with general expertise in the field of recog-

nition concerning the difficulty of leveraging multiple sources of weak

evidence into strong hypotheses.

Therefore, while we cannot expect to use our technique to classify scenes for

which the detectors are completely inaccurate, we stand a reasonable chance if

improving accuracy if the detectors are reasonably strong themselves.

1.5 Summary of Preliminary Work

We have performed our experiments in a simulated abstract world. The materials

and spatial relationships used are based on statistics captured from real scenes.

This provides us with a rich test bed, in which we can develop algorithms, compare

approaches, quickly experiment with parameters, and explore what-if situations.

With a prototype scene simulator we developed, using a single-level, tree-

structured Bayesian network for inference on a series of simulated natural scenes,

we have shown that the presence of key materials can effectively distinguish certain

scene types. However, spatial relationships are needed to disambiguate other types

of scenes, achieving a gain of 7% in one case.


20/115

9

However, when simulating faulty detectors, we found that the network is not

expressive enough to capture the necessary information, actually leading to lower

accuracy when spatial relationships were used.

1.6 Organization of Proposal

Chapter 2 contains an overview of the relevant literature in scene classification,

spatial modeling, and probabilistic frameworks. In Chapter 3, we describe our

methodology, both for the detector and the simulator. Chapter 4 is a summary of

our experiments and results (using best-case and faulty detectors). We conclude

in Chapter 5, in which we describe our research plan and proposed contributions.


21/115

10

2 Related Work

Scene classification is a young, emerging field. The first section of this chapter

is taken in large part from our earlier review of the state of the art in scene

classification [6]; because this thesis is a work in progress, there is much overlap

between the two. Here we focus our attention on systems using approaches directly

related to our proposed thesis. Readers desiring a more comprehensive survey or

more detail are referred to the original review.

All systems classifying scenes must extract appropriate features and use some

sort of learning or inference engine to classify the image. We start by outlining

the options available for features and classifiers. We then present a number of

systems which we have deemed to be good representations of the field.

We augment our review of the literature by discussing two computational

models of spatial relationships and then discussing in detail two graphical models

we could use for probabilistic inference: Bayesian Networks and Markov Random

Fields, each of which will be explored in the thesis.


22/115

11

2.1 Design Space of Scene Classification

The literature reveals two approaches to scene classification: exemplar-based and

model-based. On one hand, exemplar-based approaches use pattern recognition

techniques on vectors of low-level image features (such as color, texture, or edges)

or semantic features (such as sky, faces or grass). The exemplars are thought

to fall into clusters, which can then be used to classify novel test images, using

an appropriate distance metric. Most systems use an exemplar-based approach,

perhaps due to recent advances in pattern recognition techniques. On the other

hand, model-based approaches are designed using expert knowledge of the scene

such as the expected configuration of a scene. A scenes configuration is the layout

(relative location and sizes) of its objects. While it seems as though this should

be very important, very little research has been done in this area.

In either case, appropriate features must be extracted for accurate classifica-

tion. What makes a certain feature appropriate for a given task? For pattern

classification, one wants the inter-class distance to maximized and the intra-class

distances to be minimized. Many choices are intuitive, e.g. edge features should

help separate city and landscape scenes [78].

2.1.1 Features

In our review [6], we described features we found in similar systems, or which

we thought could be potentially useful. Table 2.1 is a summary of that set of

descriptions.

2.1.2 Learning and Inference Engines

Pattern recognition systems classify samples represented by feature vectors (see a

good review in [28]). Features are extracted from each of a set of training images,


23/115

12

Table 2.1: Options for features to use in scene classification.

Feature Description

Color Histograms [72], Coherence vectors [49], Moments [79]

Texture [51] Wavelets[42; 65], MSAR [73], Fractal dimension [71]

Filter Output Fourier & discrete cosine transforms [46; 73; 74; 75; 77],

Gabor [59], Spatio-temporal [53]

Edges Direction histograms [77], Direction coherence vectors

Context Patch Dominant edge with neighboring edges [63]

Object Geometry Area, Eccentricity, Orientation [10]

Object Detection Output from belief-based material detectors[37; 66],

rigid object detectors [63], face detectors [60]

IU Output Output of other image understanding systems

e.g., Main Subject Detection [66]

Context Within images (scale, focus, pose) [75]

Between images (adjacent images on film or video)

Mid-level Spatial envelope features [47]

Meta-data Time-stamp, Flash firing, Focal length, text [36; 24]

Statistical Measures Dimensionality reduction [18; 58]


24/115

13

or exemplars. In most classifiers, a statistical inference engine then extracts infor-

mation from the processed training data. Finally, to classify a novel test image,

this type of system extracts the same features from the test image and compares

them to those in the training set [18]. This exemplar-based approach is used by

most of the current systems.

The classifiers used in these type of systems differ in how they extract infor-

mation from the training data. In Table 2.2, we present a summary of the major

systems used in the realm of scene classification.

2.2 Scene Classification Systems

As stated, many of the systems proposed in the literature for scene classification

are exemplar-based, but a few are model-based, relying on expert knowledge to

model scene types, usually in terms of the expected configuration of objects in

the scene. In this section, we describe briefly some of these systems and point

out some of their limitations en route to differentiating our proposed method. We

organize the systems by feature type and in the use of spatial information, asshown in Table 2.3. Features are grouped into low-level, mid-level, and high-level

(semantic) features, while spatial information is grouped into those that model

the spatial relationships explicitly in the inference stage and those that do not.

2.2.1 Low-level Features and Implicit Spatial Relationships

A number of researchers have used low-level features sampled at regular spatial

locations (e.g. blocks in a rectangular grid). In this way, spatial features are

encoded implicitly, since the features computed on each location are mapped to

fixed dimensions in the feature space.


25/115

14

Table 2.2: Potential classifiers to use in scene classification.

Classifier Description

1-Nearest-Neighbor Classifies test sample with same class as the exemplar

(1NN) closest to it in the feature space.

K-Nearest-Neighbor Generalization of 1NN in which the sample is given

(kNN) [18] the label of the majority of the k closest exemplars.

Learning Vector A representative set of exemplars, called a codebook,

Quantization (LVQ) is extracted. The codebook size and learning rate

[31; 32] must be chosen in advance.

Maximum a Posteriori Combines the class likelihoods (which must be

(MAP) [77] modeled, e.g., with a mixture of Gaussians) with

class priors using Bayes rule.

Support Vector Find an optimal hyperplane separating two classes.

Machine (SVM) Maps data into higher dimensions, using a kernel

[8; 61] function,to increase separability. The kernel and

associated parameters must be chosen in advance.

Artificial Neural Function approximators in which the inputs are

Networks (ANN) [1] mapped, through a series of linear combinations

and non-linear activation functions to outputs.

The weights are learned using a technique

called backpropagation.


26/115

15

Table 2.3: Related work in scene classification, organized by feature type and use

of spatial information.

Spatial Information

Feature Type Implicit/None Explicit

Low-level Vailaya, et al. Lipson, et al.

Oliva, et al. Ratan & Grimson

Szummer & Picard Smith & Li

Serrano, et al.

Paek & Chang

Carson, et al.

Wang, et al.

Mid-level Oliva, et al.

High-level Luo, et al. Mulhem, et al.

(Semantic) Song & Zhang Proposed Method

The problems addressed include indoor vs. outdoor classification (Szummer

and Picard [73], Paek and Chang [48], and Serrano et al. [65]), outdoor scene

classification (Vailaya et al. [77]), and image orientation detection [79; 80] 1.

The indoor vs. outdoor classifiers accuracy approaches 90% on tough (e.g.,

consumer) image sets. On the outdoor scene classification problem, mid-90%

accuracy is reported. This may be due to the use of constrained data sets (e.g.

from the Corel stock photo library), because on less constrained (e.g., consumer)

image sets, we found the results to be lower. The generalizability of the technique

is also called into question by the discrepancies in the numbers reported for image

orientation detection by some of the same researchers [79; 80].

1While image orientation detection is a different level of semantic classification, many of the

techniques used are similar.


27/115

16

Pseudo-Object Features

The Blobworldsystem, developed at Berkeley, was developed primarily for content-

based indexing and retrieval. However, it is used for scene classification problem

in [9]. The researchers segment the image and use statistics computed for each re-

gion (e.g., color, texture, location with respect to a 33 grid) without performing

object recognition. Admittedly, this is a more general approach for scene types

containing no recognizable objects. However, we can hope for more using object

recognition. Finally, a maximum likelihood classifier performs the classification.

Wangs SIMPLIcity (Semantics-sensitive Integrated Matching for Picture LI-

braries) system [80] also uses segmentation to match pseudo-objects. The system

uses a fuzzy method called Integrated Region Matching to effectively compen-

sate for potentially poor segmentation, allowing a region in one image to match

with several in another image. However, spatial relationships between regions are

not used and the framework is used only for CBIR, not scene classification.

2.2.2 Low-level Features and Explicit Spatial Relationships

The systems above either ignore spatial information or encode it implicitly using

a feature vector. However, other bodies of research imply that explicitly-encoded

spatial information is valuable and should be encoded explicitly and used by the

inference engine. In this section, we review this body of research, describing a

number of systems using spatial information to model the expected configuration

of the scene.

Configural Recognition

Lipson, Grimson, and Sinha at MIT use an approach they call configural recog-

nition [34; 35], using relative spatial and color relationships between pixels in low

resolution images to match the images with class models.


28/115

17

The specific features extracted are very simple. The image is smoothed and

subsampled at a low resolution (ranging from 8 8 to 32 32). Each pixel

represents the average color of a block in the original image; no segmentation

is performed. For each pixel, only its luminance, RGB values, and position are

extracted.

The hand-crafted models are also extremely simple. For example, a template

for a snowy mountain image is a blue region over a white region over a dark

region; one for a field image is a large bluish region over a large greener region.

In general, the model contains relative x- and y-coordinates, relative R-, G-, B-,

and luminance values, and relative sizes of regions in the image.

The matching process uses the relative values of the colors in an attempt to

achieve illumination invariance. Furthermore, using relative positions mimics the

performance of a deformable template: as the model is compared to the image,

the model can be deformed by moving the patch around so that it best matches

the image. A model-image match occurs if any one configuration of the model

matches the image. However, this criterion may be extended to include the degree

of deformation and multiple matches depending on how well the model is expectedto match the scene.

Classification is binary for each classifier. On a test set containing 700 pro-

fessional images (the Corel Fields, Sunsets and Sunrises, Glaciers and Mountains,

Coasts, California Coasts, Waterfalls, and Lakes and Rivers CDs), the authors

report recall using four classifiers: fields (80%), snowy mountains (75%), snowy

mountains with lakes (67%), and waterfalls (33%). Unfortunately, exact precision

numbers cannot be calculated from the results given.

The strength of the system lies in the flexibility of the template, in terms of

both luminance and position. However, one limitation the authors state is that

each class model captured only a narrow band of images within the class and that

multiple models were needed to span a class.


29/115

18

Learning the Model Parameters In a follow-up study by Ratan and Grimson

[54], they also used the same model, but learned the model parameters from

exemplars. They reported similar results to the hand-crafted models used by

Lipson. However, the method was computationally expensive [83].

Combining Configurations with Statistical Learning In another varia-

tion on the previous research, Yu and Grimson adapt the configural approach to

a statistical, feature-vector based approach, treating configurations like words ap-

pearing in a document [83]. Set representations, e.g. attributed graphs, contain

parts and relations. In this framework, the configurations of relative brightness,

positions, and sizes are subgraphs. However, inference is computationally costly.

Vector representations allow for efficient learning of visual concepts (using

the rich theory of supervised learning). Encoding configural information in the

features overcomes the limited ability of vector representations to preserve relevant

information about spatial layout. [83]

Within a CBIR framework with two query images, configurations as extracted

as follows. Because configurations contained in both images are most informative,

an extension of the maximum clique method is used to extract common subgraphs

from the two images. The essence of the method is that configurations are grown

from the best matching pairs (e.g., highly contrasting regions) in each image.

During the query process, the common configurations are broken into smaller

parts and converted to a vector format, in which feature i corresponds to the

probability that sub-configuration i is present in the image.

A naive (i.e., single-level, tree-structured) Bayesian network is trained on-linefor image retrieval. A set of query images is used for training, with likelihood

parameters estimated by EM. Database images are then retrieved in order of

their posterior probability.

On a subset of 1000 Corel images, a single waterfall query is shown to have


30/115

19

better retrieval performance than other measures such as color histograms, wavelet

coefficients, and Gabor filter outputs.

Note that the spatial information is explicitly encoded in the features, but is

used directly in the inference process.

In the subgraph extraction process above, if extracting a common configuration

from more than two images is desired, one can use Hong, et al.s method [26].

Composite Region Templates (CRT)

CRTs are configurations of segmented image regions [69]. The configurations are

limited to those occurring in the vertical direction: each vertical column is stored

as a region string and statistics are computed for various sequences occurring in

the strings. While an interesting approach, one unfortunate limitation of their

experimental work is that the size of the training and testing sets were both

extremely limited.

2.2.3 Mid-level Features and Implicit Spatial Relationships

Oliva and Torralba [46; 47] propose what they call a scene-centered description

of images. They use an underlying framework of low-level features (multiscale

Gabor filters), coupled with supervised learning to estimate the spatial envelope

properties of a scene. They classify images with respect to verticalness (vertical

vs. horizontal), naturalness (vs. man-made), openness (presence of a horizon

line), roughness (fractal complexity), busyness (sense of clutter in man-made

scenes), expansion (perspective in man-made scenes), ruggedness (deviationfrom the horizon in natural scenes), and depth range.

Images are then projected into this 8-dimensional space in which the dimen-

sions correspond to the spatial envelope features. They measure their success first

on individual dimensions through a ranking experiment. They then claim that


31/115

20

their features are highly correlated with the semantic categories of the images

(e.g., highway scenes are open and exhibit high expansion), demonstrating

some success on their set of images. It is unclear how their results generalize.

One observation they make is that their scene-centered approach is comple-

mentary to an object-centered approach like ours.

2.2.4 Semantic Features without Spatial Relationships

Semantic Features for Indoor Vs. Outdoor Classification

Luo and Savakis extended the method of [65] by incorporating semantic mate-

rial detection [37]. A Bayesian Network was trained for inference, with evidence

coming from low-level (color, texture) features and semantic (sky, grass) features.

Detected semantic features (which are not completely accurate) produced a gain

of over 2% and best-case (100% accurate) semantics gave a gain of almost 8%

over low-level features alone. The network used conditional probabilities of the

form P(sky present|outdoor). While this work showed the advantage of using

semantic material detection for certain types of scene classification, it stopped

short of using spatial relationships.

Semantic Features for Image Retrieval

Song and Zhang investigate the use of semantic features within the context of

image retrieval [70]. Their results are impressive, showing that semantic features

greatly outperform typical low-level features, including color histograms, color

coherence vectors, and wavelet texture for retrieval.

They use the illumination topology of images (using a variant of contour trees)

to identify image regions and combine this with other features to classify the

regions into the semantic categories such as sky, water, trees, waves, placid water,

lawn, and snow.


32/115

21

While they do not apply their work directly to scene classification, their success

with semantic features confirms our hypothesis that they help bridge the semantic

gap between pixel-representations and high-level understanding.

2.2.5 Semantic Features and Explicit Spatial Relationships

Mulhem, Leow, and Lee [44] present a novel variation of fuzzy conceptual graphs

for use in scene classification. Conceptual graphs are used for representing knowl-

edge in logic-based applications, since they can be converted to expressions of

first-order logic. Fuzzy conceptual graphs extend this by adding a method of

handling uncertainty.

A fuzzy conceptual graph is composed of three elements: a set of concepts

(e.g., mountain or tree), a set of relations (e.g., smaller than or above),

and a set of relation attributes (e.g., ratio of two sizes). Any of these elements

which contain multiple possibilities is called fuzzy, while one which does not is

called crisp.

Model graphs for prototypical scenes are hand-crafted, and contain crisp con-

cepts and fuzzy relations and attributes. For example, a mountain-over-lake

scene must contain a mountain and water, but the spatial relations are not guar-

anteed to hold. A fuzzy relation such as smaller than may hold most of the

time, but not always.

Image graphs contain fuzzy concepts and crisp relations and attributes. This

is intuitive: while a material detector calculates the boundaries of objects and

can therefore calculate relations (e.g. to the left of) between them, they can be

uncertain as to the actual classification of the material (consider the difficulty of

distinguishing between cloudy sky and snow, or of rock and sand). The ability to

handle uncertainty on the part of the material detectors is an advantage of this

framework.


33/115

22

Two subgraphs are matched using graph projection, a mapping such that each

part of a subgraph of the model graph exists in the image graph, and a metric

for linearly combining the strength of match between concepts, relations, and

attributes. A subgraph isomorphism algorithm is used to find the subgraph of the

model that matches best the image.

The basic idea of the algorithms is to decompose the model and image into

arches (two concepts connected by a relation), seed a subgraph with the best

matching pair of arches, and incrementally add other model arches that match

well.

They found that the image matching metric worked well on a small database oftwo hundred images and four scene models (of mountain/lake scenes) generated by

hand. Fuzzy classification of materials was done using color histograms and Gabor

texture features. The method of generating confidence levels of the classification

is not specified.

While the results look promising for mountain/lake scenes, it remains to be

seen how well this approach will scale to a larger number of scene types.

2.2.6 Summary of Scene Classification Systems

Referring back to the summary of prior work in semantic scene classification given

in Table 2.3, we see that our work is closest to that of Mulhem, et al., but differs in

one key aspect: while theirs is logic-based, our proposed method is founded upon

probability theory, leading to principled methods of handling variability in scene

configurations. Our proposed method also learns the model parameters from a

set of training data, while theirs are fixed.


34/115

23

2.3 Options for Computing Spatial Relationships

If we are to utilize spatial relationships between regions, we need a method for

computing and encoding these relationships. We discuss both qualitative and

quantitative spatial relationships.

2.3.1 Computing Qualitative Spatial Relationships

We start by considering three models of computing qualitative spatial relation-

ships: Attentional Vector Sum, a biologically-inspired model; Weighted Walk-

throughs, a model developed for occluded regions; and a hybrid model producedfor efficient computation.

Attentional Vector Sum

Regier and Carlson [55] propose a computational model of spatial relations based

on human perception. They consider up, down, left, and right, to be symmetric,

and so focus their work on the above relation.

They call the reference object the landmark and the located object the tra-

jector. For example, the ball (trajector) is above the table (landmark). The

model is designed to handle 2D landmarks, but only point trajectors. However,

the researchers state that they are in the process of extending the model.

Four models are compared:

1. Bounding box (BB). A is above B if it is higher than the landmarks highest

point and between its leftmost and rightmost points. The strength of the

match varies depending on the height and how centered it is above the ob-

ject; three parameters govern how quickly the match drops off, via sigmoidal

functions.


35/115

24

2. Proximal and Center-of-Mass (PC). Here, the projection is defined based

on the angle formed by the y-axis and the line connecting the trajector to

the landmark. Connecting it to the closest point on the landmark gives

the proximal angle and to the centroid gives the center-of-mass angle. This

model has four parameters: a gain, slope, and y-intercept of the piecewise

function for the goodness of the angle, and the relative weight given to the

components corresponding to the two angles.

3. Hybrid Model (PC-BB). This model extends the PC model by adding the

BB models height term. The height term gives the presence of a grazing

line at the top of the landmark, an effect that was observed experimentally.The model has four parameters: the slope, y-intercept, and relative weight

of the PC model plus the gain on the height functions sigmoid.

4. Attentional Vector Sum (AVS). This model incorporates two human percep-

tual elements:

Attention. Visual search for a target in a field of distractors is slow

when targets differ from distractors only in the spatial relation amongtheir elements (i.e. they do not exhibit pop-up). Therefore, they

require attention.

Vector sum. Studies of orientation cells in the monkey cortex show

that directions were modeled by a vector sum of the cells.

In the AVS model, the angle between the landmark and the trajector is

calculated as the weighted sum of angles between the points in the landmark

area and the trajector. The weights in the sum are related to attention. The

center of attention on the landmark is the point closest to the trajector; its

angle receives the most weight. As the landmark points get further from

the center of attention, they are weighted less, dropping off exponentially.


36/115

25

Lastly, the BB models height function is used again (for which they can

give no physiological or perceptual basis, but only because it was observed

experimentally). The model has four parameters: the width of the beam of

attention, the slope and y-intercept of the linear function relating angle to

match strength (as in the PC model) and the gain on the height functions

sigmoid.

Optimal parameters for each model were found by fitting the model with an-

other researchers data set. A series of experiments was then performed to dis-

tinguish between the models. The AVS model fit each experiments data most

consistently.

The AVS method also gives a measure of how above one region is compared

to another. This measure may potentially be used to our advantage.

However, as given, AVS may be too computationally expensive. Where ni

is # points in each region i, finding points on perimeter of each is O(n1 + n2),

giving p perimeter points. Finding closest point between landmark and trajector,

yields O(p) distances. Integrating over each region yields O(n1 n2) distances.

However, if using a larger step size would not substantially reduce accuracy, we

could reduce the computation significantly.

Weighted Walkthroughs

Berretti et al. [5] developed a technique named weighted walkthroughs to cal-

culate spatial relations. The method is designed to compare segmented regions

created by color backpropagation, and therefore has the advantage of handlinglandmarks or trajectors that are made of multiple regions. This may be impor-

tant in natural images, where large regions are sometimes occluded.

The method is straightforward: consider two regions A and B. All pairs of

points (a, b) in the set S = {(a, b)|a A, b B} are compared (a walkthrough


37/115

26

each of the regions). For some pairs, A will lie northeast of B, for others A will

lie SE, etc. for the four quadrants. The fraction of pairs contained in each of the

four quadrants are computed: four weights: wNE, wNW, wSE, and wSW.

Finally, these can be converted to above/below, left/right, and diagonality by

computing four features: above = wNE+wNW, right = wNE+wSE, and diagonality

= wNE + wSW.

One advantage of this method is its ability to handle 2D, occluded ((i.e.,

disconnected) landmarks and trajectors.

Hybrid Approach

In their research, Luo and Zhu [40] use a hybrid approach, combining the bounding

box and weighted walkthrough methods. The method was designed for modeling

spatial relations between materials in natural scenes, and so favors above/below

calculations. It skips weighted walkthroughs when object bounding boxes do not

overlap. It does not handle some obscure cases correctly, but is fast and correct

when used in practice.

The final decision of the spatial relationship model would be most appropriate

for my work depends in large part on whether the AVS method can be extended

to 2D trajectors while being made computationally tractable. One answer may

be to work on a higher conceptual level than individual pixels.

2.3.2 Computing Quantitative Spatial Relationships

While computationally more expensive and possibly too sensitive, more detailed

spatial information may be necessary to distinguish some scene types. Rather

than just encoding the direction (such as above) in our knowledge framework,

we could incorporate a direction and a distance. For example, Rimey encoded

spatial relationships using an expected area net [56].


38/115

27

In the limit, we may wish to model the class-conditional spatial distributions as

continuous variables. There is a body of literature addressing the issue of efficient

inference of continuous variables in graphical models (e.g. Bayes Nets), if the

distributions are assumed Gaussian (e.g., [33]). Felzenszwalb and Huttenlocher

[19] used Gaussian models in their object recognition system, which we will review

in the next section.

Another option, if a lattice-structured Markov Random Field framework is

used (also discussed in the next section), would be to use the spatial relationships

that arise from the lattice structure.

2.4 Probabilistic Graphical Models

Early research involving spatial relationships between objects used logic [2]. This

approach was not particularly successful on natural scenes: while logic is certain,

life is uncertain. In an attempt to overcome this limitation, more recent work has

extended the framework to use fuzzy logic [44].

However, Pearl [50] argues that logic cannot be extended to the uncertaintyof life, where many rules have exceptions. The rules of Boolean logic contain no

method of combining exceptions. Furthermore, logic interactions occur in stages,

allowing for efficient computation. We would like to handle uncertain evidence

incrementally as well. But unless one makes strict independence assumptions, this

is impossible with logicand computing the effect of evidence in one global step

is impossible.

Logic-based (syntactic or rule-based) systems combine beliefs numerically. The

uncertainty of a formula is calculated as a combination of the uncertainties of the

sub-formulas. Computationally, this approach mirrors the process of logical infer-

ence, leading to an efficient, modular scheme. Rules can be combined regardless

of other rules and regardless of how the rule was derived. Semantically, these


39/115

28

assumptions are too strong, except under the strongest of independence assump-

tions. They cause the following problems semantically.

1. Bidirectional inferences. Semantically, if we have the two statements, firesmoke

and smoke, then fire should be more plausible. However, in a rule-based sys-

tem, this would introduce a loop.

2. Limits of modularity. Consider the rules alarm burglar and alarm

earthquake. Due to the modular nature of logic, if alarm becomes more

plausible, then burgular should become more plausible as well. However,

using plausible reasoning, if we add evidence for earthquake, then alarm

becomes more plausible and burglar becomes less plausible, which corre-

sponds with human intuition.

3. Correlated evidence. The rules of logic cannot handle multiple pieces of

evidence originating from a single source. As an example, one should not

independently increase the belief of an event based on many local news

stories that merely echo the Associated Press.

Some attempts have been made to overcome this last limitation, such as bounds

propagation or user-defined combination; however, each approach introduces fur-

ther difficulties.

We are fully aware that there is not universal agreement with Pearl philo-

sophically regarding the superiority of probability over logic. (Witness the heated

rebuttals to Cheesemans argument for probability [14] by the logic community!)

Still, we think his arguments are sound.

Specifically, Pearl argues for a graphical model-based approach founded on

probability calculus. While he elaborated on Bayesian Networks in [50], we also

consider Markov Random Fields (MRF), another probabilistic graphical model


40/115

29

that has been used primarily for low-level vision problems (finding boundaries,

growing regions), but has recently been used for object detection.

In general, graphical probability models provide a distinct advantage in prob-

lems of inference and learning, that of statistical independence assumptions. In

a graphical model, nodes represent random variables and edges represent depen-

dencies between those variables. Ideally, nodes are connected by an edge if and

only if their variables are directly dependent; however, many models only capture

one direction.

Sparse graphs, in particular, benefit from the message-passing algorithms used

to propagate evidence around the network. While the calculation of a joint proba-bility distribution takes exponential space (and marginals are difficult to calculate)

in general, these calculations are much cheaper in certain types of graphs, as we

will see.

2.4.1 Bayesian Networks

Bayesian (or belief) networks are used to model causal probabilistic relationships

[13] between a system of random variables. The causal relationships are repre-

sented by a directed acyclic graph (DAG) in which each link connects a cause (the

parent node) to an effect (the child node). The strength of the link between

the two is represented as the conditional probability of the child given the parent.

The directed nature of the graph allows conditional independence to be specified;

in particular, a node is conditionally independent of all of its non-successors, given

its parent(s).

The independence assumptions allow the joint probability distribution of all

of the variables in the system to be specified in a simplified manner, particularly

if the graph is sparse.

Specifically, the network consists of four parts, as follows [66]:


41/115

30

Prior probabilities are the initial beliefs about the root node(s) in the net-

work when no evidence is presented.

Each node has a conditional probability matrix (CPM) associated with it,representing the causality between the node and its parents. These can be

assigned by an expert or learned from data.

Evidence is the input presented to the network. Nodes can be instantiated

(by setting the belief in one of its hypotheses to 1) or set to fractional

(uncertain) beliefs (via virtual evidence [50]).

Posteriors are the output of the network. Their value is calculated fromthe product of priors and likelihoods arising from the evidence (as in Bayes

Rule).

The expressive power, inference schemes and associated computational com-

plexity all depend greatly on the density and topology of the graph. We discuss

three categories: tree, poly-tree, and general DAG.

Trees

If the graph is tree-structured, with each node having exactly one parent node,

each nodes exact posterior belief can be calculated quickly and in a distributed

fashion using a simple message-passing scheme. Feedback is avoided by separating

causal and diagnostic (evidential) support for each variable using top-down and

bottom-up propagation of messages, respectively.

The message-passing algorithm for tree-structured Bayesian networks is simple

and allows for inference in polynomial time. However, its expressive power is

somewhat limited because each effect can have only a single cause. In human

reasoning, effects can have multiple potential causes that are weighed against one

another as independent variables [50].


42/115

31

Causal Polytrees

A polytree is a singly-connected graph (one whose underlying undirected graph

is acyclic). Polytrees are a generalization of trees that allow for effects to have

multiple causes.

The message-passing schemes for trees generalize to polytrees, and exact pos-

terior beliefs can be calculated. One drawback is that each variable is conditioned

on the combination of its parents values. Estimating the values in the condi-

tional probability matrix may be difficult because its size is exponential in the

number of parent nodes. Large numbers of parents for a node can induce con-

siderable computational complexity, since the message involves a summation over

each combination of parent values.

Models for multicausal interactions, such as the noisy-OR gate, have been

developed to solve this problem. They are modeled after human reasoning and

reduce the complexity of the messages from a node to O(p), linear in the number

of its parents. The messages in the noisy-OR gate model can be computed in

closed form (see [50]).

A nice summary of the inference processes for trees and polytrees given in [50]

can be found in [66].

General Directed Acyclic Graphs

The most general case is a DAG that contains undirected loops. While a DAG

cannot contain a directed cycle, its underlying undirected graph may contain a

cycle, as shown in Figure 2.1.

Loops cause problems for Bayesian networks, both architectural and semantic.

First, the message passing algorithm fails, since messages may cycle around the

loop. Second, the posterior probabilities may not be correct, since the conditional

independence assumption is violated. In Figure 2.1, variables B and C may be


43/115

32

A

B

D

C

Figure 2.1: A Bayes Net with a loop.

conditionally independent given their common parent A, but messages passed

through D from B will also (incorrectly) affect the belief in C.

There exist a number of methods for coping with loops [50]. Two methods,

clusteringand conditioning, are tractable only for sparse graphs. Another method,

stochastic simulation, involves sampling the Bayesian network. We use a simple

top-down version, called logic sampling, as a generative model and describe it in

Section 3.2.2. However, it is inefficient in the face of instantiated evidence, since

it involves rejecting each sample that does not agree with the evidence.

Finally, the methods ofbelief propagation and generalized belief propagation, in

which the loops are simply ignored, has been applied with success in many cases

[82] and is worth further investigation. We discuss these methods in the context

of MRFs in the next section.

Applications of Bayesian Networks

In computer vision, Bayesian networks have been used in many applications in-

cluding indoor vs. outdoor image classification [37; 48], main subject detection

[66], and control of selective perception [57]. An advantage of Bayesian networks


44/115

33

is that they are able to fuse different types of sensory data (e.g. low-level and

semantic features) in a well-founded manner.

2.4.2 Markov Random Fields

Markov Random Fields (MRFs), or Markov Networks, model a set of random

variables as nodes in a graph. Dependencies between variables are represented

by undirected arcs between the corresponding nodes in the graph. The topology

of the network explicitly identifies independence assumptions absence of an

arc between two nodes indicates that the nodes are assumed to be conditionally

independent given their nbrs. MRFs are used extensively for problems in low-

level computer vision and statistical physics. MRFs provide a framework to infer

underlying global structure from local observations.

We now discuss the basic concepts of MRFs, drawing from our in-house review

[7] of the typical treatments in the literature [30; 16; 22; 15].

Random Field A set of random variables X = {xi}.

Graphical Model A random field X may be represented as a graphical model

G = (Q, E) composed of a set of nodes Q and edges E connecting pairs

of nodes. A node i Q represents the random variable xi X. An

edge (i, j) E connecting nodes i and j indicates a statistical dependency

between random variables xi and xj. More importantly, the lack of an edge

between two graph nodes indicates an assumption of independence between

the nodes given their neighbors.

Configuration For a random field X of size n, a configuration of X assigns

a value (x1 = 1, x2 = 2, . . . xn = n) to each random variable xi X.

P() is the probability density function over the set = {} of all possible

configurations of X.


45/115

34

Neighborhood Relationship We define a neighborhood relationship N on a

field X as follows. Let E be a set of ordered pairs representing connections

(typically probabilistic dependencies) between elements of X. Then for any

xi, xj X, xj Ni (xi, xj) E.

Markov Property [30]

For a variable xi in a random field X, if xi satisfies the Markov Property,

P(xi = i|xj = j, j = i) = P(xi = i|xj = j, j Ni)

The probabilities in Equation 2.4.2 are called local characteristics and intu-

itively describe a locality condition, namely that the value of any variable

in the field depends only on its neighbors.

Positivity Condition The positivity condition states that for every configura-

tion , P(x = ) > 0.

Markov Random Field Any random field satisfying both the Markov property

and the positivity condition. Also called a Markov Network.

Two-Layer MRF [23; 16; 22]

Two-Layer describes the network topology of the MRF. The top layer

represents the input, or evidence, while the bottom layer represents the

relationships between neighboring nodes (Figure 2.2).

In typical computer vision problems, inter level links between the top and

bottom layers enforce compatibility between image evidence and the un-

derlying scene. Intra-level links in the top layer of the MRF leverage a

prioriknowledge about relationships between parts of the underlying scene

to enforce consistency between neighboring nodes in the underlying scene

[16]..


46/115

35

Figure 2.2: Portion of a typical two-layer MRF. In low-level computer vision prob-

lems, the top layer (black) represents the external evidence of the observed image

while the bottom layer (white) expresses the a prioriknowledge about relationships

between parts of the underlying scene.

Pairwise MRF [23; 16; 82]

In a pairwise MRF, the joint distribution over the MRF is captured by a set

of compatibility functions that describe the statistical relationships between

pairs of random variables in the MRF. For inferential purposes, this means

that the graphical model representing the MRF has no cliques larger than

size two.

Compatibility Functions The statistical dependency between the two random

variables xi, xj in a random field is characterized by a compatibility function

i,j(i, j) that scores every possible pair of hypotheses (xi = i, xj = j).

As an example, consider a link (i, j) in a graphical model G connecting

nodes i and j. If there are three possible outcomes for xi and two possible

outcomes for xj, the compatibility function relating i and j is a 32 matrix,

M = [mij].

Depending upon the problem, compatibilities may be characterized by either

the joint distribution of the two variables2. For some problems for which

2Or equivalently by both conditional distributions (p(x,y) may be obtained from p(x|y) and

p(y|x).


47/115

36

the joint is unobtainable, a single conditional distribution suffices (e.g.for a

problem for which p(x|y) is known, but p(y|x) cannot be computed).

Inference

In typical low-level computer vision applications of MRFs, what is desired from

the inference procedure is the MAP estimate of the true scene (the labeling), given

the observed data (the image). We have identified two complementary approaches

in the literature for calculating the MAP estimate: deterministic techniques and

Monte Carlo techniques (described later in this section).

We start by reviewing two deterministic techniques: Belief Propagation and

Highest Confidence First. The Highest Confidence First algorithm finds local

maxima of the posterior distribution by using the principle of least commitment

[43], while belief propagation is an inexact inference procedure using message-

passing algorithms successfully in loopy networks by simply ignoring the loops.

Highest Confidence First (HCF) [16]

The HCF algorithm is used for MAP estimation, finding local maxima ofthe posterior distribution. It is a deterministic procedure founded on the

principle of least commitment. Scene nodes connected to image nodes with

the strongest external evidence (i.e. a hypothesis with a large ratio of the

maximum-likelihood hypothesis to the others) are committed first, since

they are unlikely to change (based on compatibility with neighbors). Nodes

with weak evidence commit later and are based primarily on their compat-

ibility with their committed neighbors.

Using edge-modeling MRF as an example, large intensity gradients might

constitute strong evidence in some of the networks nodes. The nodes with

strong evidence should influence scene nodes with weaker evidence (via edge

continuity constraints) more than the other way around.


48/115

37

Belief Propagation [82; 22; 29]

The Belief Propagation (BP) algorithm is a message-passing algorithm for

probabilistic networks. It is a generalization of a number of inference al-gorithms such as the forward-backward algorithm, the Viterbi algorithm,

Pearls algorithm for Bayesian polytrees, and the Kalman filter.

At each iteration, each node computes a belief, which is a marginal of the

joint probability. The belief is a function of local compatibilities, i(xi)

(e.g. local evidence nodes, which are constant, can subsumed) and incoming

messages, mji from neighboring nodes:

bi(xi) = ki(xi)

jN(i)

mji(xi)

The messages, mji, are computed from a function of compatibilities of

the messages sender and recipient nodes and previous messages from the

senders other neighbors:

mij(xj) =xi

(xi, yi)(xi, xj)

kN(i)j

mki(xi)

Intuitively, the incoming messages represent combined evidence that has

already propagated through the network.

In the rare case that the graph contains no loops, it can be shown that

the marginals are exact. However, some experimental work suggests that

at least for certain problems, that the approximations are good even in the

typical loopy networks, as the evidence is double-counted [81].

One can calculate the MAP estimate at each node by replacing the summa-

tions in the messages with max.

Generalized belief propagation [82]


49/115

38

Generalized belief propagation (GBP) uses messages from clusters of nodes

to other clusters of nodes. Since these messages are expected to be more

informative, the performance is also expected to increase.

GBP is theoretically justified: in fact, GBP is related to Kikuchi approxi-

mations in a manner analogous to BP and Bethe free energy: a set of beliefs

gives a GBP fixed point in a graph if and only if the beliefs are local station-

ary points of the Kikuchi free energy (for details of free energy minimization

techniques, see [82]).

GBP has been found to perform much better than BP on graphs with short

loops. The drawback is that the complexity is exponential in the clustersize, but again, if the graph has short loops (and thus necessitates only

small clusters), the increased complexity can be minimal.

Pearls clustering algorithm is a special case of GBP, with clusters chosen

to overlap in a fixed manner that are usually large. They obtain increased

accuracy, but at increased complexity.

An advantage of GBP is that it can be used to vary the cluster size in order

to make a trade-off between accuracy and complexity.

Inference on Tree-Structured MRFs

Felzenszwalb and Huttenlocher [19] use tree-structured MRFs for recognition of

objects such as faces and people. They model their objects as a collection of

parts appearing in a particular spatial arrangement. Their premise is that in a

part-based approach, recognition of individual parts is difficult without context,and needs spatial context for more accurate performance.

They model the expected part locations using a tree-structured two-layer

MRF. In the scene layer, the nodes represent parts and the connections repre-


50/115

39

sent general spatial relationships between the parts. However, rather than using

the typical square lattice, they use a tree.

Inference in the MRF is both exact and efficient, due to the tree structure.

Their MAP estimation algorithm is based on dynamic programming and is very

similar in flavor to the Viterbi algorithm for Hidden Markov Models. In fact, the

brief literature in the field on using Hidden Markov Models for object and people

detection [20] might be better cast in an MRF framework.

Monte Carlo Methods

Our treatment is taken in large part from [7; 45; 41; 23].

Monte Carlo methods are used for sampling. The goal is to characterize a

distribution using a set of well-chosen samples. These can be used for approximate

MAP estimation, computing expectations, etc., and are especially helpful when

the expectations cannot be calculated analytically.

How the representative samples are drawn and weighted depends on the Monte

Carlo method used. One must keep in mind that the number of iterations of the

various algorithms that are needed to obtain independent samples may be large.

Monte Carlo Integration Monte Carlo integration is used to compute expec-

tations of functions over probability distributions. Let p(x) be a probability

distribution and a(x) be a function of interest. We assume that

a(x)p(x)

cannot be evaluated analytically, but p(x) is easy to sample from (e.g., Gaus-

sian). To compute a Monte Carlo estimate of

a(x)p(x), we first create a

representative sample of Xs from p(x). There will be many Xs from theregions of high probability density for p(x) (intuitively, the Xs that should

be common in the real world). We then calculate a(x) for each x in the set.

The average value of a(x) closely approximates the expectation.

A key insight into this concept can be stated as follows:


51/115

40

The expectation of a function of random variables depends not

only on the functions value, but on how often the random vari-

ables take certain values!

The drawback is that our assumption that p(x) is easy is often not valid;

p(x) is often not easy to sample from, and so we need a search mechanism

to draw good samples. Furthermore, we must be careful that this search

mechanism does not bias the results.

Monte Carlo Markov Chain (MCMC) Methods [45; 41]

In Monte Carlo Markov Chain (MCMC) methods, the samples are drawn

from the end of a random walk.

A Markov chain is a series of random variables, X0, X1, . . . , X t in which a

locality condition is satisfied, that is,

P(X(t + 1)|Xt, X(t 1), . . . , X 0) = [P(X(t + 1)|Xt)

The chain can be specified using the initial probability, p0(x) = P(X0) and

the transition probabilities, p(X(t + 1)|Xt). The transition probability of

moving from state x to state y at time t is denoted Tt(x, y) , which can be

summarized in a transition matrix, Tt.

We consider homogenous Markov chains, those in which the transition prob-

abilities are constant, i.e. Tt = T for all T.

We take an initial distribution across the state space (which, in the case of

MRFs, is the set of possible configurations of the individual variables).

This distribution is multiplied by a matrix of transition probabilities re-

peatedly, each iteration yielding a new distribution. The theory of Markov

chains gaurantees that the distribution will converge to the true distribution

if the chain is ergodic (always converging to the same distribution).


52/115

41

In practice, one generates a sample from the initial probability distribution

(e.g., uniform) and then moves through the state space stochastically; a

random walk guided by the values in the transition matrix. Since the distri-

bution is guaranteed to converge, at the end of the random walk, the sample

should be from the actual distribution.

The number of steps needed before convergence is reached is bounded by

theoretical results, but varies in practice.

There are a number MCMC algorithms:

Gibbs Sampler. [23; 45; 41]

In the Gibbs sampling algorithm, each step of the random walk is taken

along one dimension, conditioned on the present values of the other

dimensions. In a MRF problem, it is assumed that the conditional

probabilities are known, since they are local (by the Markov property).

This method was developed in the field of physics, but was first applied

to low-level computer vision problems by Geman and Geman on the

problem of image restoration [23]. Geman and Geman furthermore

combined Gibbs sampling with simulated annealing to obtain not just

a sample, but the MAP estimate of their distribution.

Their application of Gibbs sampling is also called stochastic relaxation

(so as to differentiate it from deterministic relation techniques).

Metropolis Sampler. [45; 41]

In each iteration of the Metropolis algorithm, one makes a small change

from the current state, and accepts the change based on how good

(probabilistically) the new state is compared to the old one.

Metropolis-Hastings Sampler. [41]

Generalization of the Metropolis algorithm.


53/115

42

Importance Sampling [45; 41]

To sample from a distribution f(x), sample from a simpler distribution,

g(x), and weight based on the ratio between the original distribution and

g(x). One caveat is that g(x) should have heavy tails, e.g. Cauchy, because

g(x) should not equal zero where the original distribution has non-zero prob-

ability.

Rejection Sampling [45; 41]

To sample from a distribution f(x), sample from another, similar distribu-

tion g(x), which is bounded by a constant multiple, c, of the true distribu-

tion. Generate a point x from g(x) and accept the point with probability

f(x)/cg(x), repeating until a point

05.Tr862.Probabilistic Modeling for Semantic Scene Classification

Documents

Transcript of 05.Tr862.Probabilistic Modeling for Semantic Scene Classification