Color-Attributes-Related Image Retrieval

Student: Kylie GormanMentor: Yang Zhang

Problem and Solution•Content based image retrieval is a common

problem in computer vision•Object-related image retrieval is a popular area

related to this issue•Attributed-related image retrieval is a possible

solution •Enable a person to retrieve an image based on

attributes of an object•Some people have tried to use color as a starting

point, but this is still a very novel concept

Data Sets•Learn colors from real-world images•Train data: Google data set

▫11 colors with 100 images per color•Test data: EBay data set

▫11 colors with 12 images per color▫Corresponding binary image for each

image

Train Data Steps• Calculate feature matrix based on Color

Moments▫Calculate every box rather than every pixel

• Concatenate feature matrices• Calculate PCA (Principal Component Analysis)• Calculate GMM (Gaussian Mixture Model) based

on PCA results• Multiply individual feature matrices by

coefficient matrix• Use GMM results to calculate Fisher Vectors• Train 11 SVM’s

Test Data Steps•Calculate feature matrix of each image,

isolating the object first using binary images

•Use PCA and GMM results from training data to calculate fisher vectors

•Apply Fisher Vector to each individual result to obtain vectors that are the same size

•Classify eBay images using 11 SVM’s from training data

•Calculate Precision

Steps

CIELAB Results

Average Precision:

~42%

HSV Images

Average Precision: ~45%

RGB Images

Average Precision: ~50%

New Data Sets•Birds 200

▫200 species/categories with 11,788 images total

•Flowers 102▫102 categories with 40-258 images per

category▫8189 images total

•Cartoon▫590 images total

Flowers 102 and Birds 200•Part One

▫Get Feature Matrices with Color Moments▫Calculate PCA and GMM of training data:

1,020 images•Part Two

▫Get Feature Matrices with Dense SIFT▫Calculate PCA and GMM of training data:

100 images•Part Three

▫Use new Color Descriptor

Future Goals•Compare our color moment plus Dense

SIFT against new color descriptor and Dense SIFT▫If no improvement, determine why

•Incorporate object detection and image retrieval