58 1094-7167/03/$17.00 © 2003 IEEE IEEE INTELLIGENT SYSTEMSPublished by the IEEE Computer Society

I n f o r m a t i o n R e t r i e v a l

Citation-BasedRetrieval for ScholarlyPublicationsY. He, University of Cambridge

S.C. Hui, Nanyang Technological University

A.C.M. Fong, Massey University

Many scholarly publications are available on the Internet or in digital libraries.1–3

However, the information is not always well organized, which makes search-

ing for relevant publications difficult and time consuming. Commercial search engines

such as Yahoo!, Lycos, and Excite help users locate specific information by matching

queries against a database of stored, indexed docu-ments. However, many of these search engines haveproved ineffective for searching scholarly publica-tions accurately.

Researchers have developed autonomous citationindexing agents, such as CiteSeer,4 to search com-puter-science-related literature online. These agentsextract citation information from the literature andstore it in a database. CiteSeer can convert PostScriptand PDF documents to text using the pstotext pro-gram from the Digital Virtual Paper project (seewww.research.digital.com/SRC/virtualpaper/home.html and www.research.digital.com/SRC/virtualpaper/pstotext.html). The citation indices created are sim-ilar to the Science Citation Index.5 As such, you cangenerate a citation database to store citation infor-mation. When new publications become available,the citation indexing agent can detect them and storethem in the database.

Citation databases contain rich information thatyou can mine to retrieve publications. Our intelligent-retrieval technique for scholarly publications storedin a citation database uses Kohonen’s self-organiz-ing map (KSOM).6 The technique consists of a train-ing process that generates cluster information and aretrieval process that ranks publications’ relevanceon the basis of user queries.

The scholarly publication retrievalsystem

Figure 1 shows our scholarly publication retrievalsystem’s architecture. Without losing generality, wefocus on scholarly publications found on the Internet.

The term publications repository emphasizes the factthat we could apply the same retrieval technique todocuments in a digital library.

In our system, the two major components are thecitation indexing agent and the intelligent retrievalagent. The citation indexing agent finds scholarly pub-lications in two ways. The first is similar to CiteSeer,which uses search engines to locate Web sites con-taining publication keywords. The other lets usersspecify publication Web sites through indexing clients.The indexing clients let users determine how often thecitation indexing agent visits these sites. The index-ing agent then downloads the publications from theWeb sites and converts them from PDF or PostScriptformat to text using the pstotext tool. It identifies theWeb publications’bibliographic section through key-words such as “bibliography” or “references,” thenextracts citation information from the bibliographicsection and stores it in the citation database.

In addition to using indexing clients, the systemincorporates several retrieval clients. A retrievalclient provides the necessary user interface forinputting queries that will be passed to the intelli-gent retrieval agent for further processing. The intel-ligent retrieval agent mines the citation database toidentify hidden relationships and explore usefulknowledge to improve the efficiency and effective-ness of scholarly literature retrieval.

The citation databasePublished papers generally contain some cited ref-

erences for readers to probe further. These citationsprovide valuable information and directives for

Scholarly publications

are available online

and in digital libraries,

but existing search

engines are mostly

ineffective for these

publications. The

proposed publication

retrieval system is

based on Kohonen’s

self-organizing map

and offers fast

retrieval speeds and

high precision in

terms of relevance.

researchers in the exchange of ideas, currenttrends, and future developments in theirrespective fields. A citation index containsthe references a paper cites, linking thesource literature to the cited works. You canuse citation indices to identify existingresearch fields or newly emerging areas, ana-lyze research trends, discover the scholarlyimpact, and avoid duplicating previouslyreported works.

A citation database is a data warehousefor storing citation indices. Some of thestored information includes the author name,title, and journal name. The database con-tains all the cited references (footnotes orbibliographies) published with the articles.These references reveal how the sourcepaper is connected to prior relevant researchbecause the citing and cited references havea strong link through semantics. Therefore,citation indices can help facilitate the searchfor and management of information. Somecommercial citation index databases, suchas those the Institute for Scientific Informa-tion (ISI) provides, are available online (seewww.isinet.com).

As we discussed earlier, the citation index-ing agent can generate a citation database.For our experiment, we set up a test citationdatabase by downloading papers publishedfrom 1987 to 1997 that were stored in theinformation retrieval field of ISI’s Social Sci-ence Citation Index, which includes all thejournals on library and information science.We selected 1,466 IR-related papers from 367journals with 44,836 citations.

Figure 2 shows our citation database’sstructure, which consists of a source tableand a citation table. The source table storesinformation from the source papers, and thecitation table stores all the citations extractedfrom the source papers. Most of these twotables’attributes are identical—for example,paper title, author names, journal name, jour-nal volume, journal issue, pages, and year ofpublication. You can access a linked article’sfull text through the URL stored as one of theattributes. The primary keys in the two tablesare paper_ID in the source table and citation_IDin the citation table. In the source table,no_of_citation indicates the number of refer-ences the source paper contains. In the cita-tion table, source_ID links to paper_ID in thesource table to identify the source paper thatcites the particular publication the citationtable is storing. If two different source paperscite a publication, it is stored in the citationtable with two different citation_IDs.

Document clustering usingKSOM

Various artificial neural network modelshave been applied to document clustering. Inparticular, KSOM is a popular unsupervisedtool for ordering high-dimensional statisti-cal data in a way similar to how input itemsare mapped close to each other. With KSOM,users can display a colorful map of topic con-centrations that they can further explore bydrilling down to browse the specific topic, asWebsom demonstrates.7 AuthorLink, whichprovides a visual display of related authors,demonstrates the benefits of using KSOM togenerate a visualization interface for co-citation mapping (see http://cite.cis.drexel.edu and http:faculty.cis.drexel.edu/~xlin/authorlink.html).

KSOM is essentially a stochastic versionof K-means clustering (see the “DocumentClustering Techniques” sidebar). What dis-tinguishes KSOM from K-means is thatKSOM updates not only the closest modelbut also the neighboring models. Because theKSOM algorithm is a nonlinear projectionof the patterns of arbitrary dimensionalityinto a one- or two-dimensional array of neu-rons, the neighborhood refers to the best-

MARCH/APRIL 2003 computer.org/intelligent 59

Citationdatabase

Publicationsrepository

Indexing client

Intelligent retrieval agentCitation indexing agent

Indexing client Retrieval client Retrieval client

Figure 1. The scholarly publication retrieval system architecture.

Figure 2. The citation database.

matching neuron’s spatial neighbors, whichare also updated to react to the same input.

Adapting KSOM for retrievingscholarly publications

Citation information lets us judge docu-ment relevance because authors cite articlesthat are related. The measure CCIDF (com-mon citation × inverse document frequency)is analogous to the word-oriented TFIDF(term frequency × inverse document fre-quency) word weights.8 CCIDF assigns aweight to each citation that equals the inverseof the citation’s frequency in the entire data-base. Then, for every two documents, theweights of their common citations aresummed. The resulting value indicates howrelated the two documents are—the higherthe summed value, the greater the relation-

ship. However, we find that this method isnot very practical; identifying citations to thesame article is difficult because they can beformatted differently in different sourcepapers. Therefore, we propose a new way ofcalculating documents’ relatedness in thecitation database.

Instead of extracting keywords from thedocument as the feature factors, we can actu-ally extract the keywords from its citations.If two documents share the same citation,they must also share the same keywords. Inthe citation database, the full-text content ofcited articles is not available. The keywordsare extracted solely from the titles of all cita-tions. Each extracted keyword forms an ele-ment of a document vector. If d denotes thedocument vector, then each keyword will bedenoted by di where i is between 1 and N, and

N is the total number of distinct keywords.For each document, we extract the 20 mostfrequently occurring keywords from its cita-tions as the feature factors. (We determinedexperimentally that using 20 keywords givesus the best result.) We can then adopt theTFIDF method (see the sidebar) to representthe document vector. After solving the doc-ument representation problem, we useKSOM to categorize documents in the cita-tion database.

Figure 3 shows the citation-based retrievaltechnique using KSOM. The citation index-ing agent generates the citation database. TheKSOM retrieval technique consists of twoprocesses: training and retrieval. The train-ing process mines the citation database togenerate cluster information, and the retrievalprocess retrieves and ranks the publications

60 computer.org/intelligent IEEE INTELLIGENT SYSTEMS

I n f o r m a t i o n R e t r i e v a l

Clustering approaches using TFIDF (term frequency × inversedocument frequency) representations for text are the most pop-ular.1 Each component of a document vector is calculated as theproduct of TF (term frequency, or the number of times word wi

occurs in a document) and IDF (log[D/DF(wi)], where D is thenumber of documents and DF(wi), or document frequency, is thenumber of documents where word wi occurs at least once). Weuse the cosine measure, which is a popular similarity measure,to compute the angle between any two sparse vectors. Usingthis method, we can classify the documents into differentgroups according to the distance between them.

We broadly divide clustering algorithms into two basic cate-gories: hierarchical and nonhierarchical algorithms.2 As thename implies, hierarchical clustering algorithms involve a tree-like construction process. Agglomerative hierarchical cluster-ing algorithms are among the most commonly used. Thesealgorithms are typically slow when applied to large documentcollections. AHC algorithms are sensitive to halting criteriabecause the stopping point greatly affects the results: a clustercombination created too early or too late often causes poorresults. Nonhierarchical clustering algorithms select the clusterseeds first and assign objects into clusters on the basis of theseeds specified. The seeds might be adjusted accordingly untilall clusters are stabilized. These algorithms are faster than AHCalgorithms. The K-means algorithm3 is a nonhierarchical clus-tering algorithm that can produce overlapping clusters. How-ever, its disadvantage is that the selection of initial seeds mightgreatly impact the final result.

Recently, many other document-clustering algorithms havebeen proposed, including Suffix Tree Clustering,4 DistributionalClustering,5 and supervised clustering.6 Suffix Tree Clustering isa linear-time clustering algorithm based on identifying thephrases that are common to groups of documents, as opposedto other algorithms that treat a document as a set of unorderedwords. Distributional Clustering clusters words into groups onthe basis of the distribution of class labels associated with each

word. This way, the dimensionality of a document’s featurespace is reduced while maintaining the document classificationaccuracy. In contrast to all other clustering algorithms, whichare unsupervised clustering, supervised clustering starts with aset of seeds that represent the classes in the original taxonomy.The subsequent clustering process is independent of any fur-ther supervision. The number of clusters is maintained by eithermerging two clusters—if the similarity of their seeds is higherthan a predefined threshold—or by discarding a cluster, if thenumber of documents in the corresponding cluster is less than apredefined value.

References

1. G. Salton and M.J. McGill, Introduction to Modern InformationRetrieval, McGraw-Hill, 1983.

2. L. Kaufman and P.J. Rousseeuw, Finding Groups on Data: An Intro-duction to Cluster Analysis, John Wiley & Sons, 1990.

3. J.J. Rocchio, Document Retrieval Systems—Optimization and Evalua-tion, doctoral dissertation, Computational Laboratory, Harvard Univ.,1966.

4. O. Zamir and O. Etzioni, “Web Document Clustering: A FeasibilityDemonstration,” Proc. 21st Ann. Int’l ACM SIGIR Conf. Research andDevelopment in Information Retrieval, ACM Press, 1998, pp. 46–54.

5. L. Baker and A. McCallum, “Distributional Clustering of Words forText Classification,” Proc. 21st Ann. Int’l ACM SIGIR Conf. Researchand Development in Information Retrieval, ACM Press, 1998, pp.96–103.

6. C. Aggarwal, S. Gates, and P. Yu, “On the Merits of Building Cate-gorization System by Supervised Clustering,” Proc. 5th ACMSIGKDD Int’l Conf. Knowledge Discovery and Data Mining, ACMPress, 1999, pp. 352–356.

Document Clustering Techniques

according to user queries through theretrieval client.

Training processThe system is first trained using existing

documents from the citation database togroup the potentially relevant documents.During the training process, the system pre-processes the keywords of each documentstored in the citation database and encodesthem into tuples in floating-point represen-tations. It then feeds them into the KSOMneural network to determine the winningclusters (those closest to the input vector).The system then updates the winning clus-ters’ weights and saves them to a file.

Keyword preprocessing uses the WordNet9

thesaurus to remove stop words—for exam-ple, “the,” “a,” and “to”—using a stop list andto stem extracted keywords into their rootforms. For example, “deletion,” “deletes,” and“deleted” are various inflectional forms of the

stem word “delete.” WordNet also identifiesantonym forms of words and converts them tothe form of “not + verb” using a logical-notlist. For example, “not delete” and “cannot

delete” are converted to “not + delete.” Fig-ure 4 shows the keyword preprocessing algo-rithm, which we implemented using algo-rithms from WordNet.

MARCH/APRIL 2003 computer.org/intelligent 61

Encodingprocess

Keywordspreprocessing

Documents

Preprocessedkeywords

Coded input

Weights

Training

Weight-updatingprocess

User's query

Ranked document list

Retrieval client

Citation indexing agent

KSOMnetwork

Publicationsrepository

Citationdatabase

Encodingprocess

Keywordspreprocessing

Extracted citation information Publications

Documents

Preprocessedquery

Coded input

Retrieval

Rankingprocess

Retrieveddocument list

KSOMnetwork

WordNetthesaurus

Clusterinfo

database

Weightsfile

Weights

Cluster informationCluster information

Figure 3. The citation-based retrieval technique using the Kohonen’s self-organizing map (KSOM) network.

1. Sort all records in the Web citation database in ascending order of source paper_ID.2. For each document entry read from the Web citation database, do steps 3 through 6.3. If the source paper_ID is the same—that is, the citations belong to the same source paper—

extract all keywords from citations and store in a temporary database.4. Stem extracted keywords to their root forms, identify logical phrases, and remove

stop words.5. For every keyword, accumulate the number of occurrences.6. If the source paper_ID is different—that is, the keywords from the citations of the previous

source paper are all extracted—sort the keywords for the previous source paper on the basis of their occurrence; only take the first 20 keywords as the feature factors of the previoussource paper.

7. Go to step 3 for the next source paper.

Figure 4. The keyword preprocessing algorithm.

The encoding process converts documentsto vectors before feeding them into the neuralnetwork for training. Traditionally, documentsare represented using the vector space model.This model’s major problem is its largevocabulary, which results in a vast dimen-sionality of the document vectors. Latentsemantic indexing10 attempts to reduce thedocument vectors’ dimensionality by form-ing a matrix in which each column corre-sponds to a document vector. However, thismethod still incurs expensive computationtime. Our system uses a random projectionmethod to reduce the document vectors’dimensionality without losing the power ofdiscrimination between documents.11 Thebasic idea is to multiply the original documentvector by a random matrix R normalize theEuclidean length of each column to unity.

Figure 5 shows our encoding algorithm.We extract the first 20 of the most frequentlyoccurring keywords from each document’scitations. Altogether, 1,614 distinct key-words exist for all the citations in the cita-tion database. Using the vector space model,each document would be represented as avector with 1,614 dimensions. By multi-plying the original document vector with the matrix R, the final document vectorsobtained only have 300 dimensions. Thisincreases the learning speed dramatically,compared to previously reported methods.Figure 6 demonstrates the training process.

The reduced-dimension document vectorsare then fed into the KSOM neural networkto determine the winning cluster. Theweights of the network are initialized withrandom real numbers within the interval [0, 1]. In the testing citation database, thereare 1,466 records in the source table and44,836 records in the citation table. TheKSOM neural network retrieval’s perfor-mance depends on how many clusters it gen-erates and the average number of documentsin a cluster. However, deciding on the clus-ter map’s best size requires some insight intothe training data’s structure. In our imple-mentation, the system generates a numberof clusters equal to the square root of thetotal number of documents to be categorized.This achieves fast retrieval. Therefore, thenumber of clusters the KSOM neural net-work generates is set to 100. The initialneighborhood size is set to half the numberof clusters. The number of iterations and theinitial learning rate are set to 5,000 and 0.5,respectively. Figure 7 summarizes the KSOMnetwork training process.

62 computer.org/intelligent IEEE INTELLIGENT SYSTEMS

I n f o r m a t i o n R e t r i e v a l

(Ni, t) = (t) exp (– ) for i ∈ = Nm(t)(ri – rm 2(t)

where rm and ri are the position vectors of the winning cell and of the winning neighborhood nodes respectively, and α(t) and σ(t) are suitable decreasing functions of learning time t.

3. Update the cluster information database by adding an entry to record the link between the input vector and the winning cluster number.

For each encoded input vector x, do steps 1 through 3, and repeat the same process for the whole training set 5,000 times:

1. Obtain the similarity measure between the input vector and the weight vectors of the output nodes, and compute the winning output node as the one with the shortest Euclidean distance given as

x–wm = min {x–wm },

where wi is the weight vector and wm is the weight of the heaviest node.

2. Update weight vectors as∆wi (t ) = (Ni, t)[x(t) – wi (t )] for i ∈ = Nm(t),

where Nm(t) denotes the current spatial neighborhood, α is a positive-valued learning function,and 0 < α(Ni t) <1. The function of α can be represented as

α ασ

α

Figure 7. The KSOM network training algorithm.

1. For each document, extract the first 20 of the most frequently occurring keywords from the documents' citations.

2. Using the vector space model, represent each document as a vector with 1,614 dimensions (corresponding to the 1,614 distinct keywords for all the citations in the citation database).

3. Each element of the vectors is weighted by TFIDF (term frequency × inverse document frequency), given by wi = f (ti , d) × log(N/n), wheref( ti , d) is the frequency of term ti in document d, N is the total number of documents in the collection, and n is the number of documents containing ti.

4. Multiply document vector by a random matrix R to reduce vector dimension.

Figure 5. The encoding algorithm.

IRsearch interfacedesignask systeminformation needretrieval strategyrelevant system designgeneral system designuser searchingOPAC

Preprocessing Encoding

information retrievalsearch interfacedesigninformationretrievalstrategyrelevant documentgeneral system designuser searchopac

0.21340.11200.22510.25500.34560.54340.65980.34560.55670.9972

Extractedkeywords

Processedkeywords

Paper title: “User Goals on an Online Public Access Catalogue”

Floating-point value

Figure 6. An example of the training process.

The weight-updating process is the laststep of the training process. Whenever thesystem finds a winning cluster, it must adjustthe weights of the winning cluster togetherwith its neighborhood to adapt to the inputpattern. It stores the updated weights in a file.After the training process, the system writesthe cluster information into the cluster infor-mation database to indicate which documentbelongs to which cluster. It also stores thelinks between documents in the cluster infor-mation database and the original papers inthe citation database.

Retrieval processDuring the retrieval process, a user first

submits a query as free-text natural language.The system then preprocesses, parses, andencodes the query in a way similar to the key-word preprocessing and encoding in thetraining process. The encoded user queryinputs are 300-dimensional vectors and arecompatible with the document vectors fromthe training process. We feed these queryvectors into the KSOM network to determinewhich clusters should be activated. Thisprocess resembles using indices in the doc-ument-retrieval process. In this scenario,instead of having an index file, the index isencoded into the KSOM neural network inthe form of weight distribution.

Once the best cluster is found, the rank-ing process uses the query term’s Euclideandistance from all the documents in the clus-ter. This is based on the observation that thedocuments with a lower Euclidean distanceto the query will be semantically and con-ceptually closer to it.8 Given a document vec-tor d and a query vector q, their similarityfunction, sim(d,q), is

,

where wdi and wqi are the weight of the ithelement in the document vector d and queryvector q, respectively.

This equation measures the Euclidean dis-tance between the document vector and thequery vector. The document with the mini-mum value of sim(d,q) in a cluster gets thehighest ranking in that cluster. Other docu-ments in the cluster are sorted on the basis ofthis principle. If two or more documents in

the cluster have the same sim(d,q) value, thesystem ranks these documents arbitrarily.

Figure 8 shows the two-dimensional clus-ter map returned by searching the input string“information retrieval.” In this case, the best-matching cluster is 96. The client interfacelets the user browse through the cluster map.The system has grouped the documents intobroad areas on the basis of the particular

neighborhood relationships. Users can fur-ther examine each cluster to retrieve the doc-uments in it. These documents are ranked onthe basis of their similarity to the user’squery, as described earlier.

Figure 9 shows the documents in cluster96. The system ranks them in descendingorder of relevance to the query item on thebasis of the similarity function. The under-

sim d qd q

d q

,( ) =

×( )

( ) × ( )=

==

∑

∑∑

w w

w w

i ii

n

i ii

n

i

n

1

2 2

11

MARCH/APRIL 2003 computer.org/intelligent 63

Figure 9. Search results for cluster 96.

Figure 8. Cluster map for searching “information retrieval.” The best matching cluster isnumber 96.

lined paper titles are URL links that userscan click to get a paper’s full-text content.The “citing” and “cited” links let the usergo deeper into a particular publication’s cit-ing or cited documents.

Performance evaluationWe evaluated our citation-based retrieval

technique’s performance on the basis of itstraining performance, retrieval speed, andretrieval precision. The experiments were car-ried out on a 450-MHz Pentium II machinewith 128 Mbytes of RAM. During the exper-iments, we used the following test data:

• The number of keywords in the keywordlist was 1,614.

• The number of words to be searched in theWordNet dictionary was 121,962.

• The number of clusters was predefined at100.

• The total number of documents used fortraining (the training set) was 1,000.

We measured training performance on the

basis of the number of iterations the neuralnetwork requires to reach the convergentstate and the total training time. We measuredthe retrieval performance on the basis of theaverage online retrieval speed and theretrieval precision. While speeds of opera-tion are easy to measure, retrieval precisioninvolves considering the relevance betweenthe query and the output. Our research usedboth system-based and user-based relevancemeasures. The former used the similarityfunction discussed earlier, whereas wederived the latter from the users’assessment.Miranda Lee Pao12 defined three categoriesfor relevance assessment: highly relevant,partially relevant, and not relevant. In ourwork, we introduced two more numericalcategories for finer differentiation. Weassigned the five categories the respectivevalues of 1.0, 0.75, 0.5, 0.25, and 0.

We conducted the experiment as follows.We submitted 50 queries to the system, andwe measured the average online retrievalspeed from the time difference between themoment of query submission and the pointwhen the user received the results. For eachquery result, we examined only the first 20publications to check their relevance to thequery. We obtained the user-based relevancemeasure from an average of 10 users. Wethen derived a collective measure of retrievalprecision from the system-based and user-based relevance measures. Figure 10 com-pares system-based and user-based retrievalrelevance. Table 1 gives the overall perfor-mance results. Evidently, the KSOM networkcan perform quite efficiently, with high-retrieval precision. Also, little differenceexists between the system-based and user-based relevance measures.

A lthough we used Web publications asa case study for analysis, our pro-

posed method is generic enough to retrieverelevant publications from other reposito-ries, such as digital libraries. In terms of scal-ability, you can easily adapt our method tocope with large document collections.Although the number of training sampleswill increase with larger document collec-tions, the average length of each documentvector is not affected much. This means youonly need to redefine the number of clustersKSOM generates, while other proceduresremain unchanged.

Apart from the proposed citation-baseddocument retrieval technique, we are investi-gating other techniques, such as co-citationanalysis,13 to support document clusteringand author clustering in the publicationretrieval system. Additionally, we are usingKSOM to explore techniques that can enhancethe visual representation of cluster maps.

References

1. B. Schatz and H. Chen, “Building Large-Scale Digital Libraries,” Computer, vol. 29,no. 5, May 1996, pp. 22–26.

2. D.M. Levy and C.C. Marshall, “Going Digi-tal: A Look at Assumptions Underlying Dig-ital Libraries,” Comm. ACM, vol. 38, no. 4,Apr. 1995, pp. 77–84.

3. E.A. Fox et al., “Digital Libraries,” Comm.ACM, vol. 38, no. 4, Apr. 1995, pp. 22–28.

4. S. Lawrence, C.L. Giles, and K.D. Bollacker,“Digital Libraries and Autonomous CitationIndexing,” Computer, vol. 32, no. 6, June1999, pp. 67–71.

5. E. Garfield, Citation Indexing: Its Theory andApplication in Science, Technology, andHumanities, John Wiley & Sons, 1979.

6. T. Kohonen, Self-Organizing Maps, Springer-Verlag, 1995.

7. K. Lagus et al., “WEBSOM for Textual DataMining,” Artificial Intelligence Rev., vol. 13,nos. 5–6., Dec. 1997, pp. 310–315.

8. G. Salton, “Developments in Automatic TextRetreival,” Science, vol. 253, 30 Aug. 1991,pp. 974–979.

9. G. Miller, WordNet: An Electronic LexicalDatabase, C. Fellbaum, ed., MIT Press, 1998,preface.

64 computer.org/intelligent IEEE INTELLIGENT SYSTEMS

I n f o r m a t i o n R e t r i e v a l

Table 1. Performance results of the KSOM network.

Training

Preprocessing 2 min 34 secNumber of iterations 5,000Training time 52 min 47 secTotal number of clusters 100

Retrieval

Average retrieval speed 1.6 secSystem-based relevance 85.5%User-based relevance 83.0%Average retrieval precision 84.3%

0

0.2

0.4

0.6

0.8

1.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Rank order of documents by the system

Retri

eval

rele

vanc

e va

lue

System-basedUser-based

Figure 10. Comparison of retrieval relevance.

10. S. Deerwester et al., “Indexing by LatentSemantic Analysis,” J. Am. Soc. for Informa-tion Science, vol. 41, no. 6, June 1990, pp.391–407.

11. S. Kaski, “Dimensionality Reduction by Ran-dom Mapping: Fast Similarity Computationfor Clustering,” Proc. Int’l Joint Conf. NeuralNetworks (IJCNN 98), vol. 1, IEEE Press,1998, pp. 413–418.

12. M.L. Pao, “Term and Citation Retrieval: AField Study,” Information Processing & Man-agement, vol. 29, no. 1, Jan./Feb 1993, pp.95–112.

13. H.D. White and K.W. McCain, “Visualizinga Discipline: An Author Co-citation Analysisof Information Science, 1972–1995,” J. Am.Soc. for Information Science, vol. 49, no. 4,Apr. 1998, pp. 327–355.

For more on this or any other computing topic,see our Digital Library at http://computer.org/publications/dlib.

T h e A u t h o r sY. He is a PhD student in Cambridge University’s Speech, Vision, and Robot-ics Group. The work this article describes was carried out while she was asenior tutor at Nanyang Technological University’s School of Computer Engi-neering. Her research interests include data mining and information retrieval.She received her BASc and MEng from Nanyang Technological University.Contact her at the Speech, Vision, and Robotics Group, Cambridge Univ.Eng. Dept., Trumpington St., Cambridge, CB2 1PZ; yh213@cam.ac.uk.

S.C. Hui is an associate professor in Nanyang Technological University’sSchool of Computer Engineering. His research interests include data min-ing, Internet technology, and multimedia systems. He received his BSc inmathematics and DPhil in computer science from the University of Sussex.Contact him at the School of Computer Eng., Nanyang Technological Univ.,Nanyang Ave., Singapore 639798; asschui@ntu.edu.sg

A.C.M. Fong is a lecturer at Massey University. The work this articledescribes was carried out while he was with Nanyang Technological Uni-versity’s School of Computer Engineering. His research interests include var-ious aspects of Internet technology, data mining, information and coding the-ory, and signal processing. He received his BEng in electronic and electricalengineering and computer science and his MSc in EEE from Imperial Col-lege, London, and a PhD in EEE from the University of Auckland. He is amember of the IEEE and IEE and is a chartered engineer. Contact him at theInst. of Information and Mathematical Sciences, Massey Univ., Albany Cam-

pus, Private Bag 102-904, North Shore Mail Ctr., Auckland, New Zealand; a.c.fong@massey.ac.nz.

Submission deadline: 1 June 2003Submission address: lyang@stfx.ca

IEEE Distributed Systems Online

This special issue is intended for researchers to exchange and discuss the latest advancement of data management, I/O techniques, and hierarchical storage systems for large-scale data-intensive applications. Original and unpublished research results on allaspects of research, development, and application of this area are solicited, including, but not limited to,

• I/O and Data Storage Systems• Data Management and Service Systems• Distributed Databases• Data Repositories and Digital Libraries• Data Grid• Performance Evaluations of Data-Intensive Applications and Techniques

For more information, see http://dsonline.computer.org/cfp1.htm.

Submissions should be 4,000 to 6,000 words (tables and figures count as 250 words each). Because this is an online publication, we encour-age submissions with multimedia materials, such as animation, movies, and so on. Authors should email their papers in either PS or PDF for-mat with 5 keywords to Laurence T. Yang at lyang@stfx.ca. For our author guidelines, go to http://dsonline.computer.org/author.htm.

Guest Editors: Laurence T. Yang, lyang@stfx.ca; Marcin Paprzycki, marcin@cs.okstate.edu; Xiaohui Shen, axs095@email.mot.com; Xingfu Wu, wuxf@ece.nwu.edu.

Data Management, I/O Techniques, and Storage Systems for Large-Scale Data-Intensive Applications

Data Management, I/O Techniques, and Storage Systems for Large-Scale Data-Intensive Applications