Robust Moving Object Detection & Categorization

using self-improving classifiers

Omar Javed, Saad Ali & Mubarak Shah

Moving Object Detection & Categorization

Goal Detect moving objects in images and

classify them into categories, e.g., humans or vehicles.

Motivation Most monitoring and video

understanding systems require knowledge of, location and type of objects in the scene.

Object Classification:Major Approaches

Supervised Classifiers Adaboost (Viola & Jones), Naive Bayes

(Schniederman et al.), SVMs (Papageorgiou & Poggio)

Limitations Requirement of large number of training

examples, 1000000 negative examples for face detection (Zhang et al.). More than 10000 examples used by (Viola & Jones)

Fixed parameters after training. After deployment, parameters are not tunable to best performance in a particular scenario.

Object Classification:Major Approaches

Semi-Supervised Classifiers Co-training (Levin et al.) Limitations:

Requirement for collection of large amount of training data, though no need for labels.

Offline training, i.e., Fixed parameters in the testing phase.

Properties of an “Ideal” Object Detection System

Learns both background and object models online with no prior training.

Adapts quickly to changing background and object properties

Overview of the Proposed Approach

In a single boosted framework, Obtain regions of Interest (ROI) from a background

subtraction approach. Obtain motion and appearance features from the

ROI. Use separate views (motion and appearance

features) of the data for online co-training, i.e., If one set of features confidently predicts a label of an

object, then use this label to online update the base classifiers and the boosting parameters.

Use combined view (both features) for classification decisions.

Properties of the Proposed Object Detection Method

Background model is learned online. Object models are learned offline with a small

number of training examples. The object classifier parameters are

continuously updated online using co-training to improve detection rates.

Proposed Object Detection Method

Co-Training Decision (if confident prediction by one set)

ROIs

Background

Appearance FeatureExtraction

Background

Updated weak learners

Background Models Foreground Models

Updated parameters

Classification Output

Color Classifier

Base Classifiers(Appearance)

Motion FeatureExtraction

Edge Classifier

Base Classifiers (Motion)

Boosted Classifier

Updated Boosted Parameters

Background Detection

First level Per-pixel Mixture of Gaussian color

models Second Level

Gradient magnitude and gradient direction models

Gradient boundary check Feedback to first level

Current Image from video

Output of first level Output of second level

Features for Object Classification

Base classifiers learned from global PCA coefficients of appearance and motion templates of Image regions.

Appearance subspace learned by performing PCA separately on a small set of labeled ‘d’ dimensional gradient magnitude images of people and vehicles.

Features for Object Classification

The people and vehicle appearance subspaces are represented by d x m1 and d x m2 projection matrices (S1 and S2) respectively.

m1 and m2 are chosen such that the eigenvectors account for 99% of variance in the respective subspaces.

Features for Object Classification

Appearance features for base learners are obtained by projecting each training example ‘r’ in the two subspaces

1 1 1[ ,..., ] Tmv v r S

1 1 1 2 2[ ,..., ] Tm m mv v r S

Features for Object Classification

Row 1: Top 3 eigenvectors for person appearance subspace. Row 2: Vehicle appearance subspace

Features for Object Classification

To obtain motion features, person and vehicle motion subspaces (matrices S3 and S4)are constructed from m3 and m4 dimensional person and vehicle examples respectively.

Optical flow is obtained using the method by Lucas and Kanade.

Motion features for base learners are obtained by projecting each training motion example ‘o’ in the two subspaces

1 2 1 1 2 3 3[ ,..., ] Tm m m m mv v o S

1 2 3 1 1 2 3 4 4[ ,..., ] Tm m m m m m mv v o S

Base Classifiers We use the Bayes Classifier as the base

classifier. Let c1, c2 and c3 represent the person,

vehicle and background classes. Each feature vector component vq ,where

q ranges from 1,.., m1+m2+m3+m4 , is used to learn the pdf for each class.

The pdf is represented by a smoothed 1D histogram.

Base Classifiers The classification decision by the qth

base classifier is taken as ci,

( | ) ( | )i q j qP c v P c v i j If

Adaboost Boosting is a method for combining

many base classifiers to come up with a more accurate ‘strong’ classifier.

We use the Adaboost.M1 (Freund and Schapire) to learn the strong classifier, from the initial training data and the base classifiers.

The online co-training Framework

In general co-training requires at least two classifiers trained on independent features for labeling of data. Examples confidently labeled by one classifier are used to train the other.

In our case, individual base classifiers either represent motion or appearance features.

To determine confidence thresholds for each base classifier, we use a validation data set.

The online co-training Framework

For class ci and jth base classifier the confidence threshold, is set to be the highest probability achieved by a negative example, i.e.,

All examples in the validation set with probability higher than the threshold are correctly classified.

, i

basej cT

During the test phase, If more than 20% of the appearance based or motion based classifiers predict the label of an example with the probability higher than the validation threshold, then the example is selected for online update.

Online update is only necessary if the boosted classifier decision has a small or negative margin.

Margin thresholds are also computed from the validation set.

The online co-training Framework

i

adacT

The online co-training Framework

Once an example has been labeled by the co-training mechanism, an online boosting algorithm is used to update the base classifiers

and the boosting coefficients.

Online Co-training Algorithm

Experiments

Initial Training 50 examples of each class All examples scaled to 30x30

vector Validation Set

20 images per class Testing on three sequences

Experiments Results on Sequence1.

Experiments Results on Sequence1.

Performance over time Performance over

number of co-trained examples

Experiments Results on Sequence 2.

Experiments Results on Sequence 2.

Performance over time Performance over

number of co-trained examples

Experiments Results on Sequence 3.

Experiments Results on Sequence 3.

Performance over time Performance over

number of co-trained examples