Intelligent content-based retrieval Chabane Djeraba, Cherif El Asri Mohamed

IRIN, Ecole Polytechnique de l’Universit6 de Nantes 2, Rue de la Houssinih-e, 44322 Nantes Cedex, France

dj eraba @ irin.univ-nantes. fr

Abstract This paper deals with the challenge of extending classical image retrieval by including visual rules. The visual rules are extracted automatically from classes of images. They contribute to make the retrieval process more accurate. The visual-rules extraction are based on symbolic representations of image descriptors. The symbolic representations are the results of color and texture clustering. Key words Advanced retrieval, visual features, texture, color, knowledge, discovery, image.

1. Introduction In this paper, we experiment a new way to deal with the challenge of building automatically the knowledge (rules) of images, based on basic content descriptions? We believe that discovering hidden rules among basic image features contributes to the extraction of semantic descriptions useful to make efficient the content-based image retrieval. In our case, the discovery of rules consists of two important steps: symbolic clustering and discovery of relevant rules. The symbolic clustering concerns the definition of a learning algorithm, based on:

o data reduction, o powerful image descriptors, o and suitable similarity measures.

We organize the paper as follows: In sections 2 and 3, we will present how the content of images is extracted and represented, how descriptors of images may be used to discover rules between descriptors, and how the discovered rules are useful to content-based image retrieval. In section 4, we describe experimental results.

2. Discovery of hidden rules On the basis of image content description, knowledge is discovered. The discovered knowledge characterizes visual properties shared by images of the same semantic classes (Birds, Animals, Aircraft, Cliffs, etc.). The discovery consists of two steps: symbolic clustering, and discovery and validation of rules.

In the first step, numerical descriptions of images are transformed into symbolic form. The similar features are clustered together in the same symbolic features. Clustering simplifies, significantly, the extraction process. For example, in the figure presented below, the image is composed of region1 and region2. Region1 is characterized by light red color, and region2 by watercolor and water texture.

Figure 1: Original representation of the image.

Figure 2: Image symbolic representation.

/* Declaration of composition rules between images and regions. *I

* Region features declaration. A region is usuallj fescribed by texture and color */

features(region1, [[texture, waterfall], [color vhite]], [color, heavy-light]]).

is-composed-of(image193200, [region 11).

1082-3409/00 $10.00 Q 2000 IEEE

White color is not described by a simple string, but by a color histogram. Even if the region colors of

262

different images of the same class are similar (i.e. white), the histograms (numerical representation of color) associated with them are not generally identical. In the second step, the knowledge discovery engine automatically determines common features between the considered images in rule form. These rules are in the form of Premise => Conclusion with a certain accuracy. These rules are called statistical as they accept counter-examples.

3. Symbolic clustering algorithm

We have implemented a technique that clusters numerical representation of color, texture, by using data quantization of colors and textures, we also use the term of feature book creation. The color and texture clustering algorithms are similar; the difference lies in the distance used.

3.1 Principle of the algorithm The algorithm is a classification approach based on the following observation. The scalar quantification of Lloyd developed in 1957 is valid for our vectors (color histogram, Fourier coefficients), for rate distribution and for a wide variety of distortion criteria. It generalizes the algorithm by modifying the feature book iteratively. This generalization is known as k-means [Lin 801. The objective of the algorithm is to create a feature book, based on automatic classifications themselves based on a learning set. The learning set is composed of feature vectors of unknown probability density. Two steps should be distinguished: - A first step of classification that clusters each vector of the learning set around the initial feature book that is the most similar. The objective is to create the most representative partition of the vector space. - A second step of optimization that allows the correct adaptation to a class of the feature book vector. The gravity center of the class created in the previous step is computed. The algorithm is reiterated in the new feature book in order to obtain a new partition. The algorithm converges to stable position by evolving the distortion criteria at each iteration. Each application of the iteration of the algorithm should reduce the mean distortion. The choice of the initial feature book will influence the local minimum distortion that the algorithm will achieve. The global minimum corresponds to the initial feature book. The creation

of the initial feature book is inspired from the splitting technique [Gra 841. The splitting method breaks down a feature book Yk into two different feature books Yk-, and Yk+,, where E

is a random vector of weak energy, and its distortion depends on the distortion of the split vector. The algorithm is then applied to the new feature book in order to optimize the reproduction vectors.

3.2 Distances Quadratic distance makes it possible to obtain satisfactory results [Haf 951 since it appreciates color similarity correctly. However, its major drawback is that it is time-consuming compared to the other distances. The Euclidean distance results from the quadratic distance where A matrix is the identity matrix (no correlation between the histogram bins). In our example, the white color zones in the different images are grouped together in the symbolic form “white”, as they are similar. In the same way and based on appropriate distances, the system clusters respectively similar colors and similar textures together in the symbolic form. The experimental results have shown that the distortion values decrease quickly as the number of splitting rises. After a quick initial decrease, the distortion values decrease very slowly. Conversely, the entropy increases quickly as the number of splitting rises, and then, it increases very slowly. The experimental results have shown that the distortion values decrease quickly as the number of splitting rises. After a quick initial decrease, the distortion values decrease very slowly. Conversely, the entropy increases quickly as the number of splitting rises, and then, it increases very slowly.

3.3 Discovery and validation of rules

On the basis of the feature book, the discovery engine is triggered to discover the shared knowledge in the form of rules, and this constitutes the second step of the general algorithm. Accuracy is very important in order to estimate the quality of the rules induced. The user should indicate the threshold above which rules discovered will be kept (relevant rules). In fact, the weak rules are rules that are not representative of the shared knowledge. In order to estimate the accuracy of rules, we implement two statistical measures : conditional probability and implication intensity. The conditional probability formula of the rule a => b makes it possible to answer

263

the following question: “what are the chances of proposition b being true when proposition a is true ? The definition of this measure is:

I P(b/a) = Card(AnB)/Card(A)

More intuitively, conditional probability allows us to estimate the accuracy of a rule, considering the number of counter-examples. For example, let us consider p1 (a => b) and p2 (b => a) conditional probabilities are respectively 100% and 5.6%. So, the rule b =>a has a lot of counter-examples. In E (universe set), there are lots of objects that belong to B, but not to A. Conversely, the rule a => b has no counter-example. So, objects that respect proposition a, respect also proposition b. Conditional probability allows the system to determine the discriminating characteristics of considered images. Furthermore, we completed it by the intensity of implication [Gra 821. For example, implication intensity requires a certain number of examples or counter-examples. When the doubt area is reached, the intensity value increases or decreases rapidly contrary to the conditional probability that is linear. In fact, implication intensity simulates human behavior better than other statistical measures and particularly conditional probability. Moreover, implication intensity increases with the considered population sample representativity. The considered sample must be large enough in order to draw relevant conclusions. Finally, implication intensity takes into consideration the sizes of sets and consequently their influence. For example, conditional probability of a => b is PI (100%) and implication intensity of a =>b is cpl (23%) values are very different because conditional probability does not take into consideration the fact that proposition b is verified by lots of objects. On the contrary, implication intensity considers that it is not surprising that an object of A verifies proposition b because proposition b is verified by many objects of the considered sample. Let A,B and E sets respectively be the sets of instances that verify proposition a, the set of instances that verify proposition b, and the set of all instances or the universe set. From a theoretical point of view, implication intensity measures the degree of statistical astonishment of size A n (this set contains objects that verify proposition a and that do not verify proposition b) considering the sizes of A, B and E sets, and assuming there is no a priori link between A and B. The cardinals or the sizes of A and B subsets

of E are determined by the objects of the database belonging to A and B. The knowledge discovery engine returns the rules in the form of Premise => Conclusion whose intensity and conditional probability are greater than or equal to a certain threshold. For the moment, this threshold is defined manually. Samples of extracted rules by the prototype are (texture, water) => (color, heavy-light), (texture, waterfall) => (color, white) with respective conditional probability values of 100% and loo%, and implication intensity values of 96.08% and 87.08%.

4. Experimental results We have conducted extensive experiments of varied data sets to measure the performance of the advanced content-based query. The retrieval system can be evaluated by considering its capacity to effectively retrieve information relevant to a user. It is called the retrieval goodness. Retrieval goodness is measured by recall and precision metrics [Rij 791, [Sal 681. For a given query and a given number of images retrieved, recall gives the ratio between the number of relevant images retrieved and the total number of relevant images in the collection considered. Precision gives the ratio between the number of relevant images retrieved and the number of retrieved images. Precision = lrelevant n resultsl / Iresults; Recall = lrelevant n resultsl / lrelevantsl Recall and precision values for a system can be represented in a recall and precision graph [Rag 891, where the precision of the system is plotted as a function of the recall. This representation allows, for instance, to measure the precision at different recall points. Judging on the experiments, it is obvious that the use of knowledge leads to improvements in both precision and recall over majority queries tested. The average improvements of advanced content-based queries over classic content-based queries are 23% for precision and 17 % for recall. Precision and recall are better for advanced-based queries (queries driven by visual features and rules) than for queries that use only visual features such as colors and textures. We observe that for a sub-set of classical content- based queries, the precision is better than the precision of advanced content-based queries. A possible explanation for this is that the qualification of rules used in advanced content-based queries is too restrictive. The general principle of ccthe larger the

264

retrieved set, the higher the recall, and the lower the precision>> is observed.

5. Conclusion We have presented the interest of the rule learning in content-based retrieval in large image databases. The learning function extracts knowledge that characterizes each image class. The extraction process is strongly based on feature book creation and rule discovery. From the features of images belonging to the same class, the system finds the pattern of interest in the form of rules based on two statistical measures (conditional probability, implication intensity). These induced rules are very helpful for the comprehension of the considered class. This functionality is helpful for the automatic classification of new images during their insertion in the image database, for obtaining results with more semantics and for improving the retrieval process. We, strongly, believe that learning applied to multimedia features will play an important role in content-based multimedia indexing and retrieval.

References

[Dje 001

[Gra 821

Djeraba C., Bouet M., Henri B., Khenchaf A, <<Visual and Textual content based indexing and retrieval >>, to appear in International Journal on Digital Libraries, Springer-Verlag 2000.

Gras RCgis, The EISCAT correlator, EISCAT technical note, Kiruna 1982, EISCAT Report 82134,1982.

[Gra 841 Gray R. M. << Vector Quantization >>, IEEE ASSP Mag., pages 4-29, April 1984.

[Gup 971 Amarnath Gupta, Ramesh Jain <Visual Information Retrieval,, A communication of the ACM, May 1997Nol. 40, N"5.

[Haf95] Hafner J., al. <<Efficient Color Histogram Indexing for Quadratic Distance Functions>>. In IEEE Transaction on Pattern analysis and Machine Intelligence, July 1995.

[Jai 981 Ramesh Jain: Content-based Multimedia Information Management. ICDE 1998:

[Lin 801 Linde Y., Buzo A., Gray R. M. <<An algorithm for Vector Quantizer Design n, IEEE Trans. On Comm., Vol. COM-28, No 1, pages 84-95, January, 1980.

[Rag891 V. Raghavan, G.S. Jung, and P. Bollman <<A Critical Investigation of Recall and Precision as Measures of Retrieval System Performance >>, pages 205-229, 1989.

C. J. Keith van Rijsbergen <<Information retrieval,,, Second edition, London: Butterworths, 1979

[Sal 681 Salton Gerard <<Automatic Information Organization and Retrieval,, McGraw Hill Book CO, New York, 1968, Chapter 4.

[Zah72] C. T. Zahn, R. 2. Roskies, <<Fourier descriptors for plane closed curves >>, IEEE Trans. On Computers, 1972.

252-253

[Rij 791

265