Creating and Visualizing

Document Classification

J. Gelernter, D. Cao, R. Lu, E. Fink, J. Carbonell

Justification for fuzzy document classification

Fuzzy aims….how can you know exactly what you’re looking for when you don’t know the possibilities? “anomalous state of knowledge”

(Belkin et al 1982)

So fuzzy clusters reflect the cognitive state

Hypothesis: Fuzzy results clustering and visualization should save time by directing searchers to the level of results that they wish to view (rather than breaking off arbitrarily at screen bottom)

…in a prototype digital library for paleontology

Research overview

Talk overview

Background: often classification with algorithms alone, de-emphasizing document

Approach: * Facets and browse categories* Metadata generation

* Classifier algorithms* Visualization: labels and color grid

Findings from paleontologist experiments positive response to our fuzzy classification muted response to our fuzzy visualization

Background: fuzzy clustering

Text classification is well-researched (Sebastiani, 2002 review). It depends on algorithm used

(k-nearest neighbor, naïve bayes, support vector, etc.)

and on document representation (bag of words, or with natural language processing factors)

Our work differs from others’ in its emphasis on document representation which we hoped would provide greater precision.

Background: fuzzy info visualization

“Research in visualisation of fuzzy systems is still at an early stage” (Pham and Brown, 2003)

-- location on the page— with the top being most relevant (see left, ours)

-- 3D -- icons (see left)

-- color gradations with dark most relevant (ours)

Pre-set queries: facets based on user needs

Queries are supported by controlled vocabulary, or ontology

Metadata generation: classification according to

article rhetoric (could be improved)

Knowledge Engineeringrather than machine learning for small document set

Rules for finding matches of document to query

Example: Ma [number] Mya [number] Myr [number] B.P [number]

in document matches to associated time periods

Rules for clustering documents into fuzzy categories (requires metadata generation)

Example: *** Highly relevant if match found in title or abstract

** Relevant if match found in caption…

To solve problem of showing uncertainty clusters in a familiar list

To solve problem of showing more results per

screen as well as showing clusters

Participants: 3 paleontologists (undergraduate, graduate and museum curator)

Method: Compare classifications of people and system for same articles

• Sample: 30 articles, mix of training and non-training set articles, from 3 categories: gingko (3 levels relevancy), allosaurus (3 levels relevancy), neither

RESULTS: 70% agreed at least 1/3 of participant ratings

Participant experiments (algorithm testing)

Pilot testing with paleontologist in our group• Paleontology conferences:

– Spring 2009 NACP (North American Paleontological Convention) – 17 returned

– Fall 2009 SVP (Society of Vertebrate Paleontologists) Ask 3 graduate or undergraduates in paleontology to classify the articles – results not yet returned

• Questionnaires – Spring questionnaire: design focus – Fall: comparative focus (features as well as design)

RESULTS 58.8% liked our labels 35.7% liked our grid

Participant experiments (interface testing)

Future directions

To improve fuzzy classification:

adapt CiteSeer parse algorithm to improve our classification

To improve visualization:

list view with labels and colors for uncertainty levels

Contributions in summary

(1) Fuzzy result groupings represent “fuzzy” concept of search aim as in user’s mind, so uncertainly labels are appreciated

(2) Fuzzy color blocks that represent abstract categories are not liked; stick to minor modifications of the familiar list

References

Belkin, N.J., Oddy, R.N. and Brooks H.M. (1982) ASK for information retrieval,. Part I: Background and theory; Part II: Results of a design study, Journal of Documentation, vol. 3, no. 2&3, pp. 61-71: 145-164, 1982.

Pham, B. & Brown, R. (2003). Analysis of visualization requirement for fuzzy systems. Proceedings of the 1st international conference on computer graphics and interactive techniques in Australasia and South East Asia, Melbourne, Australia, 181 ff.

Sebastiani, (2002) Machine learning in automated text

categorization, ACM Computing Surveys, 34 (1), 1-47.