6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 1

Pattern Recognition and Applications GroupDepartment of Electrical and Electronic EngineeringUniversity of Cagliari, Italy

PhD Program in Electronic and Computer EngineeringPhD School in Information Engineering

Neighborhood-Based Feature

Weighting for Relevance

Feedback in Content-Based

Retrieval

Luca Pirasluca.piras@diee.unica.it

R AP

Pattern Recognition and

Applications Group

Giorgio Giacinto

giacinto@diee.unica.it

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 2

Outline

• Relevance Feedback

• Image representation

• Weighted similarity measures

• State of the art: Estimation of Feature Relevance

• Neighborhood-Based Feature Weighting

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 3

Aim of this work

• Exploiting neighborhood relations to weight

feature sets

• Weight designed to improve Relevance

Feedback based on Distance weighted kth-

Nearest Neighbor

• Dw k-NN estimate the relevance of an image

according to the (non-)relevant one in its

nearest neighborhood

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 4

Distance weighted kth-Nearest

Neighbor

relevanceNN

I( ) =p

NN

rI( )

pNN

rI( ) + p

NN

nrI( )

=I !NN

nrI( )

I !NNr

I( ) + I !NNnr

I( )

where pNN

rI( ) =

1

N

V I !NNr

I( )( )

and V I !NN I( )( )" I !NN I( )

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 5

Images

database

System

Image

Retrieval

Relevance Feedback

User

• Query by examples

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 6

Images

database

System

Image

Retrieval

Relevance Feedback

User

k best ranked images

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 7

Images

database

System

Image

Retrieval

Relevance Feedback

User

image labelling

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 8

Image representation

color textureshape

col. hist. layout color moments co-occurrence texture

I(F)

F = [ f1 … fi … fF ]

f1,1 … f1,i fFi …

f1,1,1 . . .f1,1,j . . .f1,1,32

f1,i,1 . . .f1,i,j . . .f1,i,9

fF,i,1 . . .fF,i,j . . .fF,i,16

fi,1… fi,j

level

image

feature

representation

components

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 9

Image representation

color histogram layout

f1,1 = [g1,1,1, g1,1,2, g1,1,3, g1,1,4 ]

g1,1,1 = [f1,1,1, …, f1,1,8]

g1,1,2 = [f1,1,9, …, f1,1,16]

g1,1,3 = [f1,1,17, …, f1,1,24]

g1,1,4 = [f1,1,25, …, f1,1,32]

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 10

Image similarity

S fi, j( ) = IA fi, j ,k( ) ! IB fi, j ,k( )p

k=1

N

"#$%&'(

1p

S fi( ) = S fi, j( )j

!

S = S fi( )i

! feature (higher)

representation

components (lower)

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 11

Weighted similarity measures

In order to have good performance into images

retrieval systems

• Relevant images should be considered as

neighbors each others.

• Non-relevant images should not be in the

neighborhood of relevant ones.

• Weighted similarity measures.

• Weights related to the capability of featurespaces of representing relevant images as

nearest-neighbors

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 12

Weighted similarity measures

S fi, j( ) = wi, j ,k IA fi, j ,k( ) ! IB fi, j ,k( )p

k=1

N

"#$%&'(

1p

S fi, j( ) = wg idpgIA , IB( )

g=1

G

!

S fi( ) = wi, j iS fi, j( )j

!

S = wi iS fi( )i

! feature (higher)

representation

components (lower)

component subset

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 13

State of the art

• Inverse of standard deviationRui, Huang, Mehrotra. Int. Conf. on Image Processing , 1997

wfj=1

! j

• Probabilistic learning (PFRL)Peng, Bhanu, Qing. Computer Vision and Image Understanding, 1999

wfj=

eT irf j z( )( )

eT irl z( )( )

l=1

F

!

fj is the j-th feature, !j is its standard deviation

rfj(z) is the measure of relevance of the j-th

feature for the query z

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 14

Neighborhood-Based

Feature Weighting

• “Relevance” of different feature space is

estimated in terms of their capability of

representing relevant images as Nearest

Neighbors

• Relevance of an image is estimated according

to the relevant and non-relevant images in its

nearest nieghborhood

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 15

Neighborhood-Based

Feature Weighting

wfx

=p

NN

r fx( )

pNN

r fx( ) + p

NN

nr fx( )

=

dmin

fx I

i,N( )

i!R

"

dmin

fx I

i,R( )

i!R

" + dmin

fx I

i,N( )

i!R

"

where pNN

r fx( ) =

1

VNN

r fx( )

and VNN

r fx( )! 1

card(R)d

min

fx I

i,R( )

i"R

#

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 16

Neighborhood-Based

Feature Weighting

• Evaluation of capability to exploit neighborhood

relations in terms of weighted similarity measures

and in terms of weighted relevance score :

– Components level

– Component subset level

relevance

NNfi, j( )S fi, j( )

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 17

Why better?

• Inverse of standard deviation

– Doesn’t use information about neighborhood of

relevant images

• Probabilistic learning (PFRL)

– It considers only relevant images

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 18

Dataset

• Corel 19511 images

• 43 classes (min: 96 - max: 1544 images)

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 19

Feature sets

• 4 feature sets

– Co-Occurrence Texture (4x4 subsets)

• 4 directions x 4 values

– Color Moments (3x3 subsets)

• first 3 moments x (H, S, V)

– Color Histogram (4x8 subsets)

• 8 ranges of H x 4 ranges of S

– Color Histogram Layout (4x8 subsets)

• 4 sub-images x 8 color

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 20

Experiment setup

• 500 queries

• 9 iterations

• 20 images retrieved each iteration

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 21

Legend

*Dw 2-NN no weight

SVM no weight

Dw 2-NN Probabilistic learning

Dw 2-NN Inverse of standard deviation

Dw 2-NN Neighborhood-Based

Dw 2-NN N-Based component subset

Dw 2-NN N-Based Score component subset

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 22

Experimental Results

Color Histogram

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 23

Experimental Results

Color Histogram

F =1

1

2 ! prec+

1

2 !recall

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 24

Experimental Results

Color Histogram (PFRL)

6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 25

Conclusions

• Reported results show that a weighted measure

improve the performance of the NN technique

• Weighted distance metric based on feature

subset provided the best results

• Neighborhood-Based weights provide similar or

better results with respect to PFRL but without

annoying tuning operations