Text Categorization With Support Vector Machines: Learning With Many Relevant Features
-
Upload
klarika-farkas -
Category
Documents
-
view
26 -
download
1
description
Transcript of Text Categorization With Support Vector Machines: Learning With Many Relevant Features
![Page 1: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/1.jpg)
Text Categorization With Support Vector Machines: Learning With Many Relevant Features
By Thornsten JoachimsPresented By Meghneel Gore
![Page 2: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/2.jpg)
Goal of Text Categorization
Classify documents into a number of pre-defined categories. Documents can be in multiple
categories Documents can be in none of the
categories
![Page 3: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/3.jpg)
Applications of Text Categorization Categorization of news stories for
online retrieval Finding interesting information from
the WWW Guiding a user's search through
hypertext
![Page 4: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/4.jpg)
Representation of Text
Removal of stop words Reduction of word to its stem Preparation of feature vector
![Page 5: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/5.jpg)
Representation of Text
..........................................................................................................................................................
2 Comput1 Process2 Buy3 Memory....
This is a Document Vector
![Page 6: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/6.jpg)
What's Next...
Appropriateness of support vector machines for this application
Support vector machine theory Conventional learning methods Experiments Results Conclusions
![Page 7: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/7.jpg)
Why SVMs?
High dimensional input space Few irrelevant features Sparse document vectors Text categorization problems are
linearly separable
![Page 8: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/8.jpg)
Support Vector Machines
Visualization of a Support Vector Machine
![Page 9: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/9.jpg)
Support Vector Machines Structural risk minimization
ndn
dherrortrainherrorP 4
ln)12
(ln2)(_))((
![Page 10: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/10.jpg)
Support Vector Machines We define a structure of hypothesis
spaces Hi such that their respective VC dimensions di increases
![Page 11: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/11.jpg)
Support Vector Machines Lemma [Vapnik, 1982]
Consider hyperplanes
}{)( bdwsigndh
As hypotheses
![Page 12: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/12.jpg)
Support Vector Machines
Awwithbdw
,1
If all example vectors are contained in A hypersphere of radius R and it is Required that
![Page 13: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/13.jpg)
Support Vector Machines Then this set of hyperplane has a
VC dimension d bounded by
1)],min([ 22 nARd
![Page 14: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/14.jpg)
Minimize
Support Vector Machines
Such that
w
ibdwy ii ,1][
![Page 15: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/15.jpg)
Conventional Learning Methods Naïve Bayes classifier Rocchio algorithm K-nearest Neighbors Decision tree classifier
![Page 16: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/16.jpg)
Naïve Bayes Classifier Consider a document vector with
attributes a1, a2… an with target values v Bayesian approach:
),,,(maxarg 21 njVv
map aaavPvj
![Page 17: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/17.jpg)
Naïve Bayes Classifier We can rewrite that using Bayes
theorem as
)()...,(maxarg
)...,(
)()...,(maxarg
21
21
21
jjnVv
n
jjn
Vvmap
vPvaaaP
aaaP
vPvaaaPv
j
j
![Page 18: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/18.jpg)
Naïve Bayes Classifier Naïve Bayes method assumes that
the attributes are independent
)""(
...)""()""()(maxarg
)()(maxarg
11
21},{
1},{
j
jjjdislikelikev
n
ijij
dislikelikevNB
vsnowaP
vhadaPvMaryaPvP
vaPvPv
j
j
![Page 19: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/19.jpg)
Experiments
Datasets Performance measures Results
![Page 20: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/20.jpg)
Datasets Reuters-21578 dataset
9603 training examples 3299 testing documents
Ohsumed Corpus 10000 training documents 10000 testing examples
![Page 21: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/21.jpg)
Performance Measures
Precision Probability that a document predicted
to be in class ‘x’ truly belongs to that class
Recall Probability that a document belonging
to class ‘x’ is classified into that class Precision/recall breakeven point
![Page 22: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/22.jpg)
Results
Precision/recall break-even point on Ohsumed dataset
![Page 23: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/23.jpg)
Results
Precision/recall break-even point on Reuters dataset
![Page 24: Text Categorization With Support Vector Machines: Learning With Many Relevant Features](https://reader034.fdocuments.in/reader034/viewer/2022042822/56812d4d550346895d924f54/html5/thumbnails/24.jpg)
Conclusions
Introduces SVMs for text categorization
Theoretical and empirical evidence that SVMs are well suited for text categorization
Consistent improvement in accuracy over other methods