Generalized Model Selection For Unsupervised Learning in High Dimension Vaithyanathan and Dom IBM...
-
Upload
hillary-clarke -
Category
Documents
-
view
212 -
download
0
Transcript of Generalized Model Selection For Unsupervised Learning in High Dimension Vaithyanathan and Dom IBM...
Generalized Model SelectionFor Unsupervised Learning
in High Dimension
Vaithyanathan and DomIBM Almaden Research Center
NIPS’99
Abstract• Bayesian approach to model selection in
unsupervised learning– propose a unified objective function whose
arguments include both the feature space and number of clusters.
• determining feature set (dividing feature set into noise features and useful features
• determining the number of clusters
– marginal likelihood with Bayesian scheme vs. cross-validation(cross-validated likelihood).
• DC (Distributional clustering of terms) for initial feature selection.
Model Selection in Clustering
• Bayesian approaches1), cross-validation2) techniques, MDL approaches3).
• Need for unified objective function– the optimal number of clusters is
dependent on the feature space in which the clustering is performed.
– c.f. feature selection in clustering
Model Selection in Clustering (Cont’d)
• Generalized model for clustering– data D = {d1,…,d}, feature space T with
dimension M– likelihood P(DT|) maximization, where
(with parameter ) is the structure of the model (# of clusters, the partitioning of the feature set into U(useful set), N(noise set) and the assignment of patterns to clusters).
• Bayesian approach to model selection– regularization using marginal likelihood
Bayesian Approach to Model Selection for Clustering
• Data– data D = {d1,…,dn}, feature space T with
dimension M
• Clustering D– finding and such that
– where is the structure of the model and is the set of all parameter vectors
– the model structure consists of the # of clusters + the partitioning of the feature set and the assignment of patterns to clusters.
(1) ),|(maxarg)ˆ,ˆ( , TDP
Assumptions1. The feature sets T represented by U and N are
conditionally independent and the data is independent.
2. Data = {d1,…,dn} is i.i.d
),|(),|(),|( UNT DPDPDP
)|()|(
11
Uk
Uj
Dj
K
k
NNi
idpdp
k
(2) )},|(),|({maxarg)ˆ,ˆ( )1( , UN DPDP
n
i
Uik
Ui
NNi
UN dpdpDPDP1
)( )|()|(),|(),|(
(4) })|()|({maxarg
(3) })|()|({maxarg)ˆ,ˆ( )2(
1 1,
1)(,
n
i
K
k Dj
Uk
Uj
NNi
n
i
Uik
Ui
NNi
k
dpdp
dpdp
lack ofregularizationmarginal
or integrated likelihood
3. All parameter vectors are independent.
– marginal likelihood
– Approximations to Marginal Likelihood/Stochastic Complexity
)()()(1
Uk
K
k
N
(5) )( )|(
)( )|()|(
1
1
Uk
Uk
K
k Dj
Uk
Ui
NNn
i
NNi
T
ddp
ddpDP
Uk
N
computationallyvery expensivepruning of search space by reducing
the number of feature partitions
model complexity
Document Clustering• Marginal likelihood
(11)
(10) )()(}|{
)()(}|{
)( )(
)|(
,
,
1 ,
1 ,
NN
Nn
tn
j nj
Nj
Uk
Uk
Uu
tUk
K
k Di Ui
Ui
SStS
N
UN
dNnt
t
dUut
t
dt
ttDP
nj
ui
k
U
adapting multinomial modelsusing term counts as the
features
assuming that priors (..)
is conjugate to the Dirichlet distribution
NLML (Negative Log Marginal Likelihood)
• Cross-Validated likelihood
Document Clustering (cont’)
Distributional clustering for feature subset selection
• heuristic method to obtain a subset of tokens that are topical and can be used as features in the bag-of-words model to cluster documents
• reduce feature size M to C• by clustering words based on their
distributions over the documents.• A histogram for each token
– the first bin: # of documents with zero occurrences of the token
– the second bin: # of documents consisting of a single occurrence of the token
– the third bin: # of documents that contain two or more occurrence of the term
DC for feature subset selection(Cont’d)
• measure of similarity of the histograms– relative entropy or the K-L distance
(.||.)• e.g. for two terms with prob. p1(.), p2(.)
• k-means DC
t tp
tptptptp
)(
)(log)())(||)((
2
1121
Experimental Setup
• AP Reuters Newswire articles from the TREC-6– 8235 documents from the routing
track, 25 classes, disregard multiple classes
– 32450 unique terms (discarding terms that appeared in less than 3 documents)
• Evaluation measure of clustering– MI
)|()()()(
),(log),();( KGHGH
KpGp
KGpKGpKGI
i jji
jiji
Results of Distributional Clustering
• cluster 32450 tokens into 3,4,5 clusters.
• eliminating function words
function words
Figure 1. centroid of atypical high-frequencyfunction-words cluster
Finding the Optimum Features and Document Clusters for a Fixed Number of
Clusters• Now, apply the objective function (11)
to the feature subsets selected by DC– EM/CEM (Classification EM: hard-
thresholded version of the EM)1)
• initialization: k-means algorithm
• Comparison of feature-selection heuristics• FBTop20: Removal of the top 20% of the most frequent
terms• FBTop40: Removal of the top 40% of the most frequent
terms• FBTop40Bot10: Removal of top 40% of the most frequent
terms and removal of all tokens that do not appear in at least 10 documents
• NF: No feature selection• CSW: Common stop words removed