91

CHAPTER 5

FACE RECOGNITION IN COMPRESSED DOMAIN USING

CANONICAL CORRELATION ANALYSIS BASED FEATURE

VECTOR OPTIMIZATION WITH MODE BASED CLASSIFICATION

5.1 Introduction

Until recently, Canonical Correlation Analysis (CCA) was a relatively

unknown statistical technique. As with almost all of the multivariate techniques, the

availability of computer programs has facilitated its increased application to research

problems. It is particularly useful in situations in which multiple output measures.

The Canonical correlation places the fewest restrictions on the types of data on

which it operates. Because the other techniques impose more rigid restrictions, it is

generally believed that the information obtained from them is of higher quality and

may be presented in a more interpretable manner. For this reason, many researchers

view canonical correlation as a final effort, to be used when all other higher-level

techniques have been exhausted. But in situations with multiple dependent and

independent variables, canonical correlation is the most appropriate and powerful

multivariate technique. It has gained acceptance in many fields and representing as a

useful tool for multivariate analysis, particularly as interest has spread to considering

multiple dependent variables.

In the recent years, the Canonical Correlation Analysis (CCA) arouse the

growing interest of experts in biometrical technologies of people recognition, as a

method which helps to relate sets of observations describing different aspects of

appearance (Jelsovka 2011). The CCA represents a high-dimensional relationship

between two sets of variables with a few pairs of canonical variables. It was intended

to describe relations between two sets of one dimensional data sequences. The CCA

method has been widely used in several fields such as signal processing (Hotelling

92

1936), medical studies, pattern recognition (W. Zhao. 2000) (M. Borga. 2001) etc.

The CCA as a novel method is applied to image processing and biometrics too. It is

compared to other projection approaches like Principal Component Analysis (PCA)

and Independent Component Analysis (ICA), CCA can concurrently deal with two

sets of data.

A face image can be considered as a vector in a high dimensional space.

This high dimensional vector has large variations between two images of the same

person, and therefore it is not directly suited for the face recognition. Principal

Component Analysis (PCA) and Independent Component Analysis (ICA) are

commonly used dimensionality reduction techniques, which give good recognition

rates. Although PCA is a good low dimensional representation for face images, it is

not able to discriminate between variations due to illumination and expression

changes. The ICA solves the illumination change problem to some extent by finding

the transformation such that it maximizes the inter-class separation and minimizes

the intra-class variations.

In (O. Friman. 2001), CCA was used for the segmentation of functional

Magnetic Resonance Images and this motivated to apply CCA for the purpose of

face recognition in this research. The Correlation analysis is useful to find a linear

relationship between two sets of variables, and CCA creates new variables for each

set such that the correlation between these variables is maximized and independent

of affine transformation (Fu-Chang Liu. 2008).

The possibility of using JPEG2000 compression scheme and Canonical

Correlation Analysis for performing face recognition in fully compressed domain is

explored in this chapter. A novel approach for efficient face recognition in

compressed domain has been proposed in this chapter using 2-dimensional Canonical

Correlation Analysis. The Matching of image data has been done by Mode based

Matching method. The experimental results proved that the proposed method

considerably improves the recognition rates and also reduces the computational time

and storage requirements.

93

5.2 Theoretical Foundations for Canonical Correlation Analysis

The Canonical Correlation Analysis (CCA) is a multivariate analysis

method used to identify and measure the association between two sets of variables

(Hotelling 1936). It first searches for a pair of linear combinations which has the

largest variation. Then CCA establishes a pair of linear combinations having the

largest correlation among all pairs uncorrelated with the initially settled pair and so

on. In fact CCA represents a high dimensional relationship between two sets of

variables with a few pairs of canonical variables (Sun Ho Lee 2007). The CCA has

various applications in pose estimation (Z. Lei 2008) and face matching (M. Slamka

2007). For two sets of variables, CCA is to create the CCA subspace to mutually

maximize the correlation between these two sets of variables.

For better understanding of the concepts and terminology used in the chapter, a

review of the important terms is given below:

Canonical correlation: It measures the strength of the overall relationships

between the linear composites (canonical variates) for the independent and

dependent variables. In effect, it represents the bivariate correlation between

the two canonical variates.

Canonical function: It is the relationship (correlational) between two linear

composites (canonical variates). Each canonical function has two canonical

variates, one for the set of dependent variables and one for the set of

independent variables. The strength of the relationship is given by the

canonical correlation.

Canonical loadings: This is a measure of the simple linear correlation

between the independent variables and their respective canonical variates.

These can be interpreted like factor loadings, and are also known as canonical

structure correlations.

Canonical roots / Eigenvalues: The Squared canonical correlations, which

provide an estimate of the amount of shared variance between the respective

94

optimally, weighted canonical variates of dependent and independent

variables.

Canonical variates/Linear composites: The Linear combinations that

represent the weighted sum of two or more variables and can be defined for

either dependent or independent variables.

5. 3 Feature Vector optimization using Canonical Correlation Analysis

The Canonical correlation analysis is a suitable and dominant technique which can

be used for exploring the relationships among multiple dependent and independent

variables. Therefore a powerful feature projection approach for facial images is

proposed based on CCA. The Correlation analysis is useful to find a linear

relationship between two sets of variables, and CCA creates new variables for each

set such that the correlation between these variables is maximized and independent

of transformation. The CCA is useful to find two sets of basic vectors, one for x and

another for y, such that the correlation between the projections of the variables onto

these basic vectors is maximized. The CCA is useful to find wx and wy which are two

pairs of vectors such that the correlation between the projections of the variables is

maximized onto the basic vector: xwx

T

xand

ywyT

y. The Projections x and y are

known as canonical variables. The projections x and y are also referred to as

canonical variates in the perspective of CCA (W. Yang 2008) (X. Wang 2004).

Officially CCA maximizes the function:

(5.1)

Whereby, E represents empirical expectation. For two random variables X and Y, the

total covariance matrix is a block matrix where Cxx∈Rp×p and Cyy∈Rq×q are the

within-sets covariance matrices of x and y respectively and Cxy∈Rp×q is the

between- sets covariance matrix (G. Kukharev 2010). Consider two random variables

yyyyT

xxxxT

xxyxT

wCwwCw

wCEwp

95

X and Y. The total covariance matrix is a block matrix where Cxx and Cyy are the

within-sets covariance matrices of x and y respectively and Cxy=Cyx T is the

between-sets covariance matrix.

(5.2)

The canonical correlations between x and y can be found by solving the

eigenvalue equations:

xyxyyxyxxx wCCCCwp112

(5.3)

yxyxxyxyyy wCCCCwp112

(5.4)

Where, eigenvalues p2 are the squared canonical correlations and eigenvectors wx

and wy are the normalized canonical correlation vectors (R. Hardon 2004). Instead

of the two eigenvalue equations (5.3) and (5.4), the problem can be formulated in

one single eigenvalue equation:

(5.5)

Where

(5.6)

Input data and the structure of CCA are schematically presented in Figure 5.1.

96

Figure 5.1 CCA calculation

5.4 Experimental Setup

The proposed methodology to perform face recognition in compressed

domain uses wavelet transform based compression scheme, Canonical correlation

analysis based feature projection technique and Mode based Matching method for

matching of images have been described here. The clear approach to carry out face

recognition in compressed domain is to utilize the coefficients extracted before

inverse transformation as input to face recognition systems. Both the stages of

inverse quantization and inverse transformations are avoided. The original image

has been transformed using the DWT and quantization and entropy coding was

done on the coefficients. The implementation has been done using MATLAB

2008B (Toolbox used are: Image Processing, JPEG 2000 toolbox and Wavelet

Toolbox), a computer system with Intel i3 Processor 2.20 GHz, memory of 3GB

and hard disk memory of 500GB. All the images were preprocessed as given in

section 5.4.2 before conducting experiments.

5.4.1 Dataset

The AR Face database of face images collected at the Computer Vision

Center in Barcelona, Spain in 1998 (Martinez. A. M. 1998) has been used for the

experiments. The AR Face Database consists of over 4000 images corresponding

97

to 126 subjects (76 males, 50 females). All images are of frontal view with

different expression, occlusions and illumination variations. Image datasets

include image feature frontal view faces with varying facial expressions (fb),

varying illumination conditions (fc) and partial occlusions (fd). The system was

trained with 4 images per subject. There are total of 272 training and 504 testing

images. The configuration of the system used for image capturing is: Pentium

133MHz, 64 MB RAM and 2GB HD, colour camera: SONY 3CCDs, a 12mm

optics and frame grabber: Matrox Meteor RGB.

5.4.2 Image pre-processing

Preprocessing of images prior to wavelet transformation is essential in this

implementation to maintain the size of the training and test images as same. The

RGB image was converted into grayscale image and then cropped to size of 128

x128 pixels. Elliptical masking was utilized to mostly remove the background.

Also images were originally transformed to obtain the eyes at the fixed points and

histogram equalization was done to have better background intensity.

5.4.3 Pixel domain experiments

In these experiments, Canonical Correlation Analysis based face

recognition was performed on the original uncompressed 128x128 images and the

results for all the three image Datasets of AR Face database were noted which is

represented by original images column in Table 5.1. The images were divided into

training and testing image Dataset where training data are formed the whole AR

database images and they are stored as vectors. Test image and train images were

transformed as vectors using the equations given in section 5.3. Face Recognition

was done by measuring canonical correlation coefficient between the test face and

train faces. The correlation between two face images has maximum value when

the correlation coefficient is 1 or -1. Then the faces are identical.

5.4.4 Compressed domain experiments

All the preprocessed images used in the experiments were compressed

according to JPEG2000 compression scheme with various compression ratios of

98

1 bpp, 0.5 bpp, 0.3 bpp and 0.2 bpp. The proposed approach eliminates major part

of the decompression phase as the decoding was interrupted after entropy

decoding and all entropy points obtained were used as input to the feature

projection method based on CCA. The coefficients after entropy decoding phase is

enough to carry out the face recognition process and hence the recognition system

was positioned after the entropy decoding phase in the proposed work.

The training and test images were transformed as vectors using the

equations in section 5.3. Face recognition was performed by measuring the

canonical correlation coefficient between the test face and all training images. The

algorithm for the proposed CCA based face recognition is given in figure 5.2. The

Mode based matching has been used for image classification which is explained in

section 5.4.5. The results for face recognition of the various experiments

conducted on the image Datasets at different compression ratios are shown in the

figures given in section 5.5.

5.4.5 Mode Based matching of images

The Euclidean distance method for matching of images possess only 50%

of the result and another matching algorithms like Kd-tree matching is not suitable

for this feature projection technique based on CCA. Hence mapping of the image

data was done by means of finding Mode method. The differences between two

square roots of canonical correlation were taken as „M‟. If there are 20 image data

in a database, then 20 different „M‟ values will be obtained. The square of each

„M‟ value will be calculated and mode of this squared value is obtained for

mapping purpose.

99

Figure 5.2 Algorithm of the proposed CCA approach

5.5 Experimental results and analysis

Numerous experiments have been conducted using the proposed CCA

based method in compressed domain. The results obtained confirms that the

proposed method based on CCA is superior in terms of total Recognition Rate

(RR) and Normalized Recognition Rate (NRR) with a computational time when

compared to the previous approaches explained in chapter 4.

CCA based Feature projection:

1. Get the image data from the compressed domain after entropy

decoding.

2. Separating single coloured plane from RGB planes of the true

colour image in order to reduce the processing time.

3. Generating identity matrix in the same size of image data from

previous step.

4. Apply Canonical Correlation Analysis for the image data and

identity matrix.

5. This CCA analysis should be taken for both query image and all

the images in Database.

6. Differences between two square root of canonical correlation are

taken as (M).

7. If there are 20 image data in a database then we will obtain 20

different „M‟.

8. Square each „M‟.

9. Mode of this squared value is obtained for mapping purpose.

100

The Recognition rates for various image Datasets obtained by using

entropy points as input to the proposed new classification method based on CCA

are given in tables given below. The table 5.1 gives the recognition rates for

images with varying expressions at different compression ratios. The table 5.2 and

5.3 give the recognition rates for images with varying illumination conditions and

minor occlusions respectively. A comparison of the recognition accuracy of the

CCA based method with existing approaches is given in Table 5.4. Comparatively,

better results were observed when evaluated against approaches based on standard

face recognition systems with Euclidean matching. Also the proposed method

shows significantly better results for the probe sets fc and fd when compared to

approaches based on PCA, ICA and KPCA with Kd-tree matching. For image

Datasets with varying expression (fb), a marginal drop in recognition rate is noted

when compared to the previous approaches based on PCA, KPCA and ICA with

Kd-tree matching.

Images with varying expression (fb probe set)

The fb probe set has shown 0.5% increase at 1 bpp, 0.35 % to 10.28%

decrease at higher compression levels when compared to PCA with Euclidean

matching & 4.85% to 13.18% decrease when compared to ICA with Euclidean

matching.

Images with varying Illumination (fc probe set)

The image Dataset with varying illumination conditions has shown 16.51%

(at .2 bpp) to 33% (at 1 bpp) increase in Recognition Rate when compared to

PCA with Euclidean matching & 4.8%(at .3 bpp) to 13.94% (at .5 bpp) increase

in Recognition Rate when evaluated against ICA with Euclidean matching.

Images with partial occlusion (fd probe set)

Partially occluded image set. al.so has shown 12.14% (at .2 bpp) to

18.04% (at 1 bpp) increase in Recognition Rate (RR) when compared to PCA

with Euclidean matching & 6.17% (at .3 bpp) to 9.26% (at 0.5 bpp) increase in

Recognition Rate when matched to ICA with Euclidean matching.

101

Table 5.1 Recognition Rate (%) of fb probe set for proposed methodology

fb probe set (images with varying expression)

Results at Rank 1 Recognition (%)

Feature

Projection

Technique

Classification

method

applied

Compressed Domain

Original

Image

(Pixel

Domain)

Compressed Images

1 bpp 0.5 bpp 0.3 bpp 0.2 bpp

Canonical

Correlation

Analysis

(CCA)

Mode based

matching 77.76 77.90 78.65 71.52 69.92

Table 5.2 Recognition Rate (%) of fc probe set for proposed methodology

fc probe set (images with varying illumination conditions)

Results at Rank 1 Recognition (%)

Feature

Projection

Technique

Classification

method

applied

Compressed Domain

Original

Image

(Pixel

Domain)

Compressed Images

1 bpp 0.5 bpp 0.3 bpp 0.2 bpp

Canonical

Correlation

Analysis

(CCA)

Mode based

matching 79.32 81.20 81.74 72.20 68.36

102

Table 5.3 Recognition Rate (%) of fd probe set for proposed methodology

fd probe set (Partially occluded images)

Results at Rank 1 Recognition (%)

Feature

Projection

Technique

Classification

method

applied

Compressed Domain

Original

Image

(Pixel

Domain)

Compressed Images

1 bpp 0.5 bpp 0.3 bpp 0.2 bpp

Canonical

Correlation

Analysis

(CCA)

Mode based

matching 53.89 55.24 51.76 48.07 45.14

Table 5.4 Comparison of Results at Rank 1 Recognition (%) for the existing

and new methodologies

The NRR evaluation of the proposed method for the fb probe set (images

with varying expression) at various compression levels is detailed in figure 5.3. It

103

is clearly observed from the values that the proposed approach outperforms the

recognition after decompression. The image Datasets with varying illumination

conditions and partial occlusions also justify the same conclusion (Figure 5.4 and

5.5 respectively).

Figure 5.3 NRR evaluation of fb probe set – CCA Mode based matching

Figure 5.4 NRR evaluation of fc probe set - CCA Mode based matching

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1 0.5 0.3 0.2

No

rmal

ize

d R

eco

gnit

ion

Rat

e (

NR

R)

Compression Ratio ( bpp)

NRR evaluation - fb probe set CCA Mode based matching

NRR afterdecompression

NRR compresseddomain

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1 0.5 0.3 0.2

No

rmal

ize

d R

eco

gnit

ion

Rat

e (

NR

R)

Compression Ratio ( bpp)

NRR evaluation - fc probe set CCA Mode based matching

NRR afterdecompression

NRR compresseddomain

104

Figure 5.5 NRR evaluation of fd probe set - CCA Mode based matching

Computational Time and Storage requirements

The computational time of the proposed CCA approach was compared with

the existing recognition system which uses Euclidean distance method (B. Li.

2002) and Kd-tree matching (section 4.4) respectively for matching of images.

The proposed method took only 2.87 seconds for recognition with a significant

improvement in recognition rate.

Figure 5.6 Comparison of computational time

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1 0.5 0.3 0.2

No

rmal

ize

d R

eco

gnit

ion

Rat

e (

NR

R)

Compression Ratio ( bpp)

NRR evaluation - fd probe set CCA Mode based matching

NRR afterdecompression

NRR compresseddomain

105

5.6 Conclusion

A new approach for performing face recognition systems directly in

JPEG2000 compressed domain based on Canonical Correlation Analysis has been

applied and assessed. The effect of compression on recognition accuracy was

observed and also issues like achieving computational time saving was examined

in the new method. The results obtained confirms that the proposed method based

on CCA is better in terms of total Recognition Rate (RR) and Normalized

Recognition Rate (NRR) with a less computational time when compared to the

previous approaches explained in the previous chapter. The suggestion for future

enhancement is to develop a new method for extracting feature vector from

entropy coded image thus completely eliminating the decompression phase in face

recognition and also a technique to enhance the proposed method by applying

artificial neural networks for better recognition of images with varying

expressions. The researcher has tried to address both the suggestions successfully

in the successive chapters 6 and 7.