Document Maps
-
Upload
kristen-haynes -
Category
Documents
-
view
23 -
download
0
description
Transcript of Document Maps
Document MapsDocument Maps Slawomir Wierzchon , Mieczyslaw Klopotek
Michal Draminski Krzysztof Ciesielski Mariusz Kujawiak
Institute of Computer Science, Polish Academy of SciencesWarsaw
Research partially supported by the KBN research project 4 T11C 026 25 "Maps and intelligent navigation in WWW using
Bayesian networks and artificial immune systems"
Agenda
Motivation What is a document map Map creation Clustering Experimental results Future directions
Motivation
The Web as well as intranets become increasingly content-rich: simple ranked lists or even hierarchies of results seem not to be adequate anymore
A good way of presenting massive document sets in an understandable form will be crucial in the near future
Document map
Many attempts have been made to visualize sets of dicuments not just like a list, but rather in two dimensions
A document map is a mapping of a set of documents to 2-D representing their inter-relationships
Linear relationship presentation(Internet Cartographer)
A relationship
A link between hypertext documents Citation in the bibliography Content similarity
A tree of relations with central subject (Inxight – Tree Studio )
Selforganizing map (WebSOM)dissimilarity of grouops of
documents
Document frequency in clusters
A meta search engine map
Our approach – multiple representations (BEATCA)
Map visualizations in 3D (BEATCA)
Future research – hypergeometric representation
(Fish-Eye eEffect)
........
INTERNET
DBREGISTRY
HT-Base
HT-Base
VEC-BaseMAP-Base
DocGR-Base
Search Engine
Indexing +Optimizing
SpiderDownloading
MappingClustering
of docs
........
CellGR-Base
Clusteringof cells
........
........ ........ ........
Processing Flow Diagram - BEATCA
The preparation of documents is done by an indexer, which turns a document into a vector-space model representation
Indexer also identifies frequent phrases in document set for clustering and labelling purposes
Subsequently, dictionary optimization is performed - extreme entropy and extremely frequent terms excluded
The map creator is applied, turning the vector-space representation into a form appropriate for on-the-fly map generation
‘The best’ (wrt some similarity measure) map is used by the query processor in response to the user’s query
How are the maps created A modified WebSOM method is used:
– compact reference vectors representation– broad-topic initialization method– joint winner search method– multi-level (hierarchical) maps– multi-phase document clustering:
• initial grouping to identify major topics
• Initial document grouping
• WEBSOM on document groups
• fuzzy cell clusters extraction and labelling
Document model in search engines
In the so-called vector model a document is considered as a vector in space spanned by the words it contains.
dogfood
walk
My dog likes this food
When walking, I take some food
Document model in search engines
The relevance of a document to a query or to another document is measured as cosine of angle between the query and the document.
dogfood
walk
Query: walk
Reference vector representation
Vectors are sparse by nature During learning process they become even
sparser Represented as a balanced red-black trees Tolerance threshold imposed Terms (dimensions) below threshold are removed Significant complexity reduction without
negative quality impact
Topic-sensitive initialization
Inter-topic similarities important both for map learning and visualization/cluster extraction
Simple approach:– Use LSI to select K main broad topics– Select K map cells (evenly spread over the map) as
the fixpoints for individual topics– Initialize selected fixpoints with broad topics– Initialize remaining cells with „in-between values”
Clustering document vectors
Document space 2D map
mxr
Mocna zmiana położenia (gruba
strzałka)
Important difference to general clustering: not only clusters with similar documents, but also neighboring clusters similar
Joint winner search
Global winner search: accurate but slow Local winner search: faster but can be inaccurate
during rapid changes Start with single phase of global search Document movements become more smooth
during learning process: usually local search is enough
Use global search when occassional sudden moves occur (eg. outliers, neighbourhood width decrease)
Hierarchical maps Bottom-up approach Feasible (with joint
winner search method)
Start with most detailed map
Compute weighted centroids of map areas
Use them as seeds for coarser map
Top-down approach is possible but requires fixpoints
21-28
Clustering document groups Numerous methods exists but none of them directly
applicable:– Extremely fuzzy structure of topical groups in SOM cells– Neccesity of taking into account similiarity measures both in
original document space and in the map space– Outlier-handling problem during cluster formation– No a priori estimation of the number of topical groups
Fuzzy C-MEANS on lattice of map cells applied Graph theoretical approach (density- and distance- based
MST) combined with fuzzy clustering Clustered documents are labeled by weighted centroids of
cell reference vectors scaled with between-group entropy
Experiments with map convergence
We examined the convergence of the maps to a stable state depending on:– type of alpha function (search radius
reduction)– type of winner search method– type of initialization method
Convergence – alpha functions (linear versus reciprocal)
Convergence – winner search (joint versus local)
Experiments with execution time
The impact of the following factors on the speed of map creation was investigated:– Map size (total number of cells)– Optimization methods:
• dictionary optimization • reference vector representation
Map quality assessment:– Compare with ‘ideal’ map (e.g. without optimizations)– Identical initialization and learning parameters– Compute sum of squared distances of location of each
document on both maps
Execution time - map size
Execution time - optimizations
Future research
Maps for joint term-citation model, taking into account between-group link flow direction
Fully distributed map creation Adaptive document retrieval and clustering:
– Bayesian network based relevance measure– Survival models for document update rate estimation– Dead link propagation methods for page freshness estimation
We also intend to integrate Bayesian and immune system methodologies with WebSOM in order to achieve new clustering effects
Future research
Bayesian networks will be applied in particular to: – measure relevance and classify documents– accelerate document clustering processes– construct a thesaurus supporting query
enrichment– keyword extraction– between-topic dependencies estimation
Thank you!
Any
questions?
Any
questions?