Evaluating Color Descriptors for Object and Scene Recognition
description
Transcript of Evaluating Color Descriptors for Object and Scene Recognition
![Page 1: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/1.jpg)
1
Evaluating Color Descriptorsfor Object and Scene Recognition
Koen E.A. van de Sande, Student Member, IEEE, Theo Gevers, Member, IEEE, andCees G.M. Snoek, Member, IEEE
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 9, SEPTEMBER 2010
![Page 2: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/2.jpg)
2
Outline
• Introduction• Reflectance Model• Diagonal Model• COLOR DESCRIPTORS AND INVARIANTPROPERTIES• EXPERIMENTAL SETUP• RESULTS• Discussion
Conclusion
![Page 3: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/3.jpg)
3
Introduction
• To increase illumination invariance and discriminative power, color descriptors have been proposed.
• This paper studies the invariance properties and the distinctiveness of color descriptors.
• The usefulness of invariance is category-specific.• Recommendations are given on which color
descriptors to use under which circumstances and data sets.
![Page 4: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/4.jpg)
4
Reflectance Model
• An image f can be modeled under the assumption of Lambertian reflectance as follows:
• Shafer [23] proposes adding a diffuse term:
![Page 5: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/5.jpg)
5
Reflectance Model
• The spatial derivative of f at location x on scale is given by:
• Hence, derivatives will yield invariance to diffuse light.
![Page 6: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/6.jpg)
6
Diagonal Model
• Changes in the illumination can be modeled by a diagonal mapping or von Kries Model [18] as follows:
• Diagonal-offset model:
![Page 7: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/7.jpg)
7
Diagonal Model
• 1. light intensity changes
• 2. light intensity shifts
![Page 8: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/8.jpg)
8
• 3. light intensity changes and shift
• 4. light color changes– Allowing (a ≠ b ≠ c)
• 5. light color changes and shift– (a ≠ b ≠ c) & (o1≠o2≠o3)
![Page 9: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/9.jpg)
9
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• Three types of descriptors are discussed.– Histogram do not contain local spatial information and are
inherently pixel-based.– color moment and color moment invariant contain local photometrical and spatial information
derived from pixel values.– SIFT contain local spatial information and are derivative-
based.
![Page 10: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/10.jpg)
10
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 11: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/11.jpg)
11
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 12: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/12.jpg)
12
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• Opponent histogram– Opponent color space:
– O1 and O2 are shift-invariant with respect to light intensity.
![Page 13: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/13.jpg)
13
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 14: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/14.jpg)
14
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• Hue histogram– A certainty of the hue is inversely proportional to
the saturation.– To be more robust, weigh each sample of the hue
by its saturation.
The H color model is scale-invariant and shift-invariant with respect to light intensity [14].
![Page 15: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/15.jpg)
15
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 16: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/16.jpg)
16
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• rghistogram– The normalized RGB color model
– r and g (b is redundant ) are scale-invariant with respect to light intensity changes, shadows, and shading.
![Page 17: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/17.jpg)
17
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 18: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/18.jpg)
18
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• Transformed color distribution
– Scale-invariance and shift-invariance are achieved by normalizing the pixel value distributions.
– Each channel is normalized independently, it also have invariance against changes in light color and arbitrary offsets.
![Page 19: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/19.jpg)
19
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 20: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/20.jpg)
20
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• SIFT– proposed by Lowe [9] describes the local shape of
a region using edge orientation histograms.– The gradient of an image is shift-invariant– The descriptor is normalized, so it have scale-
invariance.
![Page 21: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/21.jpg)
21
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 22: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/22.jpg)
22
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• HSV-SIFT– H color model is scale-invariant and shift-variant.– Complete descriptor have no invariance properties
due to the combination of the HSV channels.• HueSIFT– Similar to hue histogram, is scale-invariant and
shift-invariant.
![Page 23: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/23.jpg)
23
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 24: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/24.jpg)
24
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 25: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/25.jpg)
25
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• C-SIFT – C-invariant, which can be intuitively seen as the
normalized opponent color space and . – C-SIFT is scale-invariant.
![Page 26: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/26.jpg)
26
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 27: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/27.jpg)
27
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
![Page 28: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/28.jpg)
28
COLOR DESCRIPTORS AND INVARIANTPROPERTIES
• RGB-SIFT– SIFT descriptors are computed for every RGB
channel independently.– Its descriptor values are equal to the transformed
color space.
![Page 29: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/29.jpg)
29
EXPERIMENTAL SETUP
• Feature Extraction Pipelines
![Page 30: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/30.jpg)
30
Experiment 1: Illumination Changes
• The Amsterdam Library of Object Images (ALOI) data set [20] contains more than 48,000 images of 1,000 objects, under various illumination conditions.– artificially added “light intensity scaling “ and “light
intensity shifts “ to the data set.– The light color is varied by changing the illumination color
temperature– Objects lighted by a different number of white lights at
increasingly oblique angles– object rotation images– images with differentlevels of JPEG compression.
![Page 31: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/31.jpg)
31
Experiment 2: Image Benchmark
• The PASCAL Visual Object Classes Challenge [21] provides a yearly benchmark for comparison of object classification systems.
![Page 32: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/32.jpg)
32
Experiment 3: Video Benchmark
• The Mediamill Challenge by Snoek et al. [22] provides an annotated video data set, based on the training set of the NIST TRECVID 2005 benchmark [7].
![Page 33: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/33.jpg)
33
Evaluation Criteria
• The top-ranked result should be equal to the original image of the object for successful recognition.
• The percentage of objects where the top-ranked result is indeed the correct object is used as the performance on the ALOI data set.
• The average precision is taken as the performance metric for determining the accuracy of ranked category recognition results.
![Page 34: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/34.jpg)
34
RESULTS – Experiment1
• The theoretical invariance properties of color descriptors are validated.
• The SIFT and color SIFT descriptors perform much better than histogram-based descriptors
• Certain color descriptors are sensitive to compression artifacts, reducing their usefulness.
• The descriptors with the best overall performance are C-SIFT, rgSIFT, OpponentSIFT, and RGB-SIFT.
• The increased invariance comes at the price of reduced discriminative power.
![Page 35: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/35.jpg)
35
RESULTS – Experiment1
![Page 36: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/36.jpg)
36
RESULTS – Experiment1
![Page 37: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/37.jpg)
37
RESULTS – Experiment1
![Page 38: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/38.jpg)
38
RESULTS – Experiment1
![Page 39: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/39.jpg)
39
RESULTS – Experiment1
![Page 40: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/40.jpg)
40
RESULTS – Experiment1
![Page 41: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/41.jpg)
41
RESULTS – Experiment2
• The SIFT variants perform significantly better than color moments, moment invariants, and color histograms.
• The additional invariance makes the descriptor less discriminative for these object categories because a reduction in performance is observed.
![Page 42: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/42.jpg)
42
RESULTS – Experiment2
![Page 43: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/43.jpg)
43
RESULTS – Experiment2
![Page 44: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/44.jpg)
44
RESULTS – Experiment3
• SIFT and color SIFT variants perform significantly better than the other descriptors.
• OpponentSIFT perform better than C-SIFT and rgSIFT for these categories that occur under a wide range of light intensities.
![Page 45: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/45.jpg)
45
RESULTS – Experiment3
![Page 46: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/46.jpg)
46
Comparison with State-of-the-Art
![Page 47: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/47.jpg)
47
Discussion
• The SIFT-based descriptors outperform the other on both two category recognition.
• large variations in lighting conditions occur frequently invariance to light intensity shifts is useful.
• From the results, it can be noticed that invariance to light color changes and shifts is domain-specific.
![Page 48: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/48.jpg)
48
Discussion
![Page 49: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/49.jpg)
49
Discussion
![Page 50: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/50.jpg)
50
Conclusion
• These invariance properties were validated using a data set with known photometric changes.
• The addition of color descriptors over SIFT improves category recognition by 8 percent and 7 percent, respectively.
• Choosing a single descriptor and no prior knowledge about the data set and object and scene categories is available.
![Page 51: Evaluating Color Descriptors for Object and Scene Recognition](https://reader036.fdocuments.in/reader036/viewer/2022062323/568163f6550346895dd58964/html5/thumbnails/51.jpg)
51
Conclusion