Mining di dati web Lezione n° 6 Clustering di Documenti Web Gli Algoritmi Basati sul Contenuto A.A...

of 49 /49
Mining di dati web Lezione n° 6 Clustering di Documenti Web Gli Algoritmi Basati sul Contenuto A.A 2005/2006

Embed Size (px)

Transcript of Mining di dati web Lezione n° 6 Clustering di Documenti Web Gli Algoritmi Basati sul Contenuto A.A...

  • Mining di dati webLezione n 6Clustering di Documenti WebGli Algoritmi Basati sul Contenuto

    A.A 2005/2006

  • Document ClusteringClassical clustering algorithms are not suitable for high dimensional data.Dimensionality Reduction is a viable but expensive solution.Different kind of clustering exists:Partitional (or Top-Down)Hierarchical (or Bottom-Up)

  • Partitional ClusteringDirectly decomposes the data set into a set of disjoint clusters.The most famous is the K-Means algorithm.Usually they are linear in the number of elements to cluster.

  • Hierarchical PartitioningProceeds successively by either merging smaller clusters into larger ones, or by splitting larger clusters.The clustering methods differ in the rule by which it is decided which two small clusters are merged or which large cluster is split.The end result of the algorithm is a tree of clusters called a dendrogram, which shows how the clusters are related.By cutting the dendrogram at a desired level a clustering of the data items into disjoint groups is obtained.

  • Dendrogram Example

  • Clustering in Web Content MiningPossible uses of clustering in Web Content Mining.Automatic Document Classification.Search Engine Results Presentation.Search Engine Optimization:Collection Reorganization.Index Reorganization.Dimensionality Reduction!!!!

  • Advanced Document Clustering TechniquesCo-ClusteringDhillon, I. S., Mallela, S., and Modha, D. S. 2003. Information-theoretic co-clustering. In Proceedings of the Ninth ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (Washington, D.C., August 24 - 27, 2003). KDD '03. ACM Press, New York, NY, 89-98.Syntactic ClusteringBroder, A. Z., Glassman, S. C., Manasse, M. S., and Zweig, G. 1997. Syntactic clustering of the Web. Comput. Netw. ISDN Syst. 29, 8-13 (Sep. 1997), 1157-1166.

  • Co-ClusteringIdea: represent a collection with its term-document matrix and then cluster both rows and columns.It has a strong theoretical foundation.It is based on the assumption that the best clustering is the one that leads to the largest mutual information between the clustered random variables.

  • Information TheoryEntropy of a random variable X with probability distribution p(x): The Kullback-Leibler(KL) Divergence or Relative Entropy between two probability distributions p and q: Mutual Information between random variables X and Y:

  • Contingency TableLet X and Y be discrete random variables that take values in the sets {x1, x2, , xm} and {y1, y2, , yn}.Let p(X,Y) denote the joint probability distribution between X and Y.

  • Problem FormulationCo-clustering is concerned with simulteously clustering X into (at most) k disjoint clusters and Y into (at most) l disjoint clusters.Let the k clusters of X be written as: {x1, x2, , xk}, and let the l clusters of Y be written as: {y1, y2, , yl}.(CX,CY) is defined co-clustering, where:Cx: {x1, x2, , xm} {x1, x2, , xk}CY: {y1, y2, , yn} {y1, y2, , yl}An optimal co-clustering minimizes I(X;Y) - I(X=CX(X);Y=CY(Y)) = I(X;Y) - I(X-Y)

  • Lemma 2.1For a fixed co-clustering (CX, CY), we can write the loss in mutual information as I(X;Y) - I(X;Y) = D(p(X,Y)||q(X,Y)), where D(-||-) denotes the Kullback-Leibler divergence and q(X,Y) is a distribution of the form q(x,y)=p(x,y)p(x|x)p(y|y) where x x, y y.

  • The Approximation Matrix q(X,Y)q(x,y)=p(x,y)p(x|x)p(y|y).p(x)=x x p(x)p(y)=y y p(y)p(x|x)=p(x)/p(x)p(y|y)=p(y)/p(y)

  • Proof of Lemma 2.1

  • Some UsefulEqualities

  • Co-Clustering Algorithm

  • Co-Clustering SoundnessTheorem: The co-clustering algorithm monotonically decreases loss in mutual information (objective function value)Marginals p(x) and p(y) are preserved at every step (q(x)=p(x) and q(y)=p(y) )

  • Co-ClusteringComplexityThe algorithm is computationally efficientEven for sparse dataIf nz is the number of nonzeros in the imput joint distribution p(X,Y), t is the number of iterations: O(nz * t * (k + l))Experimentally t = 20.

  • A Toy Example

  • A Real ExampleBefore

  • A Real ExampleAfter

  • Application:Dimensionality ReductionFeature Selection

    Feature ClusteringDocumentBag-of-words1mVectorOf wordsDocumentBag-of-wordsVectorOf words Do not throw away words Cluster words instead Use clusters as featuresWord#1Word#k Select the best words Throw away rest Frequency based pruning Information criterion based pruning1mCluster#1Cluster#k

  • Syntactic ClusteringFinding syntactically similar documents.Approach based on two different similarity measures:ResemblanceContainmentA sketch of few hundreds bytes is kept for each document.

  • Document ModelWe view each document as a sequence of words.Start by lexically analyzing the doc into a canonical sequence of tokens.This canonical form ignores minor details such as formatting, html commands, and capitalization.We then associate with every document D a set of subsequences of tokens S(D,w).

  • ShinglingA contiguous subsequence contained in D is called a shingle.Given a document D we define its w-shingling S(D,w) as the set of all unique shingles of size w contained in D.For instance the 4-shingling of (a,rose,is,a,rose,is,a,rose) is the set:{(a,rose,is,a);(rose,is,a,rose);(is,a,rose,is)}.

  • ResemblaceFor a given shingle size, the resemblance r of two documents A and B is defined as where |A| is the size of set A.

  • ContainmentFor a given shingle size, the containment c of two documents A and B is defined as where |A| is the size of set A.

  • Properties of r and cThe resemblance is a number between 0 and 1.r(A,A) = 1The containment is a number between 0 and 1.If AB then c(A,B)=1.Experiments show that the definitions capture the informal notions of roughly the same and roughly contained.

  • Resemblance DistanceResemblance is not transitive.Version 100 of a document is probably quite different from version 1.The Resemblance Distance d(A,B)=1-r(A,B) is a not metric but obeys the triangle inequality.

  • Resemblance and Containment EstimatesFix a shingle size w.Let U be the set of all shingles of size w.U is countable thus we can view its elements as numbers.Fix a parameter s.For a set WU define MINs(W) as where smallest refers to numerical order on U, and define

  • Resemblance and Containment EstimatesTheorem. Let :UU a permutation of U chosen u.a.r. Let F(A)=MINs((S(A))) and V(A)=MODm((S(A))). Define F(B) and V(B) analogously. Then is an unbiased estimate of the resemblance of A and B. is an unbiased estimate of the resemblance of A and B. is an unbiased estimate of the containment of A in B.

  • The SketchChoose a random permutation of U.The Sketch of a document D consists of the set F(D) and/or V(D).F(D) has fixed size. Allows only the estimation of resemblance.V(D) has variable size. Grows as D grows.

  • Practical Sketches RepresentationCanonicalize documents by removing HTML formatting and converting all words to lowercase.The shingle size w is 10.Use a 40 bit fingerprint function, based on Rabin Fingerprints, enhanced to behave as a random permutation. Now a shingle is this fingerprint value.m in the modulus is set to 25.

  • Rabin FingerprintsIs based on the use of irreducible polynomials with coefficients in Galois Field 2.Let A=(a1, , am) be a binary string. a1=1.A(t)=a1tm-1+a2tm-2++amLet P(t) be an irriducible polynomial of degree k, over Z2.f(A)=A(t) mod P(t) is the Rabin Fingerprint of A.

  • Shingle ClusteringRetrieve every document on the Web.Calculate the sketch for each document.Compare the sketches for each pair of documents to see if they exceed a threshold of resemblance.Combine the pairs of similar documents to make the clusters of similar documents.

  • Efficiency30,000,000 HTML docsA pairwaise comparison would involve O(1015) comparisons!!!!Just one bit per document in a data structure requires 4 Mbytes. A sketch size of 800 bytes per documents requires 24 Gbytes!!!One millisecond of computation per document translates into 8 hours of computation!!!Any algorithm involving random disk accesses or that causes paging activity is completely infeasible.INEfficiency

  • Divide, Compute, MergeTake the data, divide it into pieces of size m (in order to fit the data entirely in memory)Compute on each piece separatelyMerge the results.The merging process is I/O bound:Each merge pass is linearlog(n/m) passes are required.The overall performance is O(n log(n/m)).

  • The real Clustering Algorithm (I phase)Calculate a sketch for every document. This step is linear in the total lengths of documents.

  • The real Clustering Algorithm (II phase)Produce a list of all the shingles and the documents they appear in, sorted by shingle value. To do this, the sketch for each document is expanded into a list of pairs. Sort the list using the divide, sort merge approach.Remember: shingle value, means rabin fingerprint of the sketch.

  • The real Clustering Algorithm (III phase)Generate a list of all the pairs of documents that share any shingles, along with the number of shingles they have in common. To do this, take the file of sorted couples and expand it into a list of triplets:take each shingle that appears in multiple documents and generate the complete set of triplets.Apply divide, sort, merge procedure (summing up the counts for matching ID-ID pairs) to produce a single file of all triplets sorted by the first document ID. This phase requires the greatest amount of disk space because the initial expansion of the document ID triplets is quadratic in the number of documents sharing a shingle, and initially produces many triplets with a count of 1.

  • The real Clustering Algorithm (IV phase)Produce the complete clustering. Examine each triplet and decide if the document pair exceeds our threshold for resemblance. If it does, we add a link between the two documents in a union-find algorithm. The connected components output by the union-find algorithm form the final clusters. This phase has the greatest memory requirements because we need to hold the entire union-find data structure in memory.

  • Performance IssuesCommon Shingles.Shared by more than 1,000 documents.The number of document ID pairs is quadratic in the number of documents sharing a shingle.Remove shingles that are more frequent than a given threshold.Identical Documents.Identical documents do not need to be handled. Remove identical documents from collection. Remove documents having the same fingerpring.Super shingles.Compute a meta-sketch shingling the shinglesDocuments sharing shingles in the meta-sketch are very likely to have a high resemblance value.Need to carefully choose super-shingle size.

  • Super-shingles based ClusteringCompute the list of super shingles for each documentExpand the list of super shingles into a sorted list of pairs.Any documents that share a super shingle resemble each other and are added into the cluster.

  • Problems withSuper-shinglesSuper shingles are not as flexible or as accurate as computing resemblance with regular sketches.They do not work well for shor documents. Short documents do not contain many shingles, even regular shingles are not accurate in computing resemblance.Super-shingles represent sequence of shingles, and so, shorter documents, with fewer super shingles, have a lower probability of producing a common super shingle.Super-shingles cannot detect containment.

  • A Nice Application:Page Changing CharacterizationWe can use the technique of comparing sketches over time to characterize the behavior of pages on the web.For instance, we can observe a page at different times and see how similar each version is to the preceding version.We can thus answer some basic questions like:How often do pages change?How much do they change per time interval?How often do pages move? Within a server? Between servers?How long do pages live? How many are created? How many die?

  • Experiments30,000,000 HTML Pages. 150Gbytes (5k per document)The file containing just the URLs of the documents took up 1.8Gbytes (an average of 60 bytes per URL).10 word long shingles, 5 byte fingerprint. 1 in 25 of the shingles found were kept.600M shingles and the raw sketch files took up 3 Gbytes.

  • ExperimentsIn the third phase - the creation of triples - the storage required was 20 Gbytes. At the end the file took 6 Gbytes.The final clustering phase is the most memory intensive. The final file took up less than 100MBytes.

  • ExperimentsResemblance threshold set to 50%.3.6 million clusters found containing a total of 12.3 million documents.2.1 million clusters contained only identical documents (5.3 million documents).The remainig 1.5 million clusters contained 7 million documents (a mixture of exact duplicates and similar).

  • Experiments

    PhaseTime(CPU-days)Paralle-lizableSketching4.6YESDuplicate elimination0.3Shingle merging1.7YESID-ID pair formation0.7ID-ID merging2.6YESCluster formation0.5Total 10.5