A generic approach to topic models and its application to ... · A generic approach to topic models...

A generic approach to topic modelsand its application to virtual communities

Gregor Heinrich

PhD presentation (English translation incl. backup slides, 45min)Faculty of Mathematics and Computer Science

University of Leipzig

28 November 2012Version 2.9 EN BU

Gregor Heinrich A generic approach to topic models 1 / 35

Overview

Introduction

Generic topic models

Inference methods

Application to virtual communities

Conclusions and outlook


Motivation: Virtual communities

Retrieval performance.The most prominentretrieval measures areprecision and recall.Recall is defined asthe ratio between thenumber of retrievedrelevant items tothe total number ofexisting relevant items.Precision is defined asthe ratio between the



cooperation

authorship

citationauthorship

recommendation

authorshipcitation

annotationsimilarity

annotation

Virtual communities = groups of persons who exchange informationand knowledge electronically

Examples: organisations, digital libraries, “Web 2.0” applicationsincl. social networks

Data are multimodal: text content; authorship, citation, annotationsand recommendations; cooperation and other social relations

Typical case: discrete data with high dynamics and large volumes


Motivation: Unsupervised mining of discrete data

Identification of relationships in large data volumesOnly data (and possibly model) required (information retrieval,network analysis, clustering, NLP methods)Density problem: Features too sparse for analysis inhigh-dimensional feature spaceVocabulary problem: Semantic similarity , lexical similarity(polysemy, synonymy, etc.)

4/24/12 1:58 AM

Page 1 of 2file:///data/workspace/knowceans-freshmind-lucene3/bar.svg

judicial assembly

location

long objectpressure unit

furniture

verb

long object

verb

verb

people

people

location

bank

computer

furniture

bar

court

restaurant

stickatm

counter

prevent

staff

adhere

glue

personnel

employees

yard

teller

network

table

barbar


Topic models as approach

...

...

words documents

Probabilistic representations of grouped discrete dataIllustrative for text: Words grouped in documents

Latent Topics = Probability distributions over vocabulary. Dominantterms of a topic are semantically similar.Language = Mixture of topics (latent semantic structure)

→ Reduce vocabulary problem: Find semantic relations→ Reduce density problem: Dimensionality reduction



...

...

wordstopics

documents






...

...

wordstopics

documents

bar

rhythm

drum






...

...

wordstopics

documents

bar

restaurant

wine

rhythm

drum






...

...

wordstopics

documents

“Rhythm & Spice

“Playing Drums

“Leipzig’s Barsand Restaurants”

for Beginners”

Jamaican Grill”bar

restaurant

wine

rhythm

drum






...

...

wordstopics

documents





Language models: Unigram model

wm,n

zz

w2,2 w2,3w2,1w1,2 w1,3w1,1

︸︷︷︸document 1


p(w | z)

document m

word n

1 2 3

One distribution for all data


Language models: Unigram mixture model

wm,n

zmz2z1

w2,2 w2,3w2,1w1,2 w1,3w1,1



p(w | z1) p(w | z2)

document m

word n

1 2 3 1 2 3

One distribution per document


Language models: Unigram admixture model

wm,n

zm,n

document m

word n

z2,2z1,2

w2,2 w2,3w2,1w1,2 w1,3w1,1



z1,1 z1,3 z2,1 z2,3

p(w | z1,1) . . . . . . p(w | z2,3)

One distribution per word→ basic topic model


Language models: Unigram admixture model

wm,n

zm,nz2,2z1,2

w2,2 w2,3w2,1w1,2 w1,3w1,1



z1,1 z1,3 z2,1 z2,3

p(w | z1,1) . . . . . . p(w | z2,3)

document m

word n

One distribution per word→ basic topic model


Bayesian topic models: The Dirichlet distribution

p3

p2

p1

p(w | z) ∼ Dir(~α)

~α = (4, 4, 2)

1 2 3 1 2 3 1 2 3

ϑ1 = 1 ϑ2 = 1

ϑ3 = 1

ϑ1 = 1 ϑ2 = 1

ϑ3 = 1

ϑ1 = 1 ϑ2 = 1

ϑ3 = 1

Bayesian methodology:

Distributions generatedfrom prior distributionsSpeech + other discretedata: Dirichlet distributionimportant prior:

Defined on simplex:Surface containing alldiscrete distributionsParameter ~α controlsbehaviour

→ Bayesian topic model:Latent Dirichlet Allocation(LDA) (Blei et al. 2003)


Latent Dirichlet Allocation

wm,n

zm,n

β ~ϕk

~ϑm

document m

word ntopic k

α

Dir(β)

Dir(α)

Latent Dirichlet Allocation (Blei et al. 2003)



wm,n

zm,n

β ~ϕk

~ϑm

document m

word ntopic k

α

Dir(β)

Dir(α)

Concert tonight at Rhythm and Spice Restaurant . . .

Latent Dirichlet Allocation (Blei et al. 2003)



wm,n

zm,n

β ~ϕk

~ϑm

document m

word ntopic k

α

Dir(β)

restau

rant

barfood

grill

concertmusic

rhythm bar

. . .

. . .

topic 2

topic 1


~ϕ2

~ϕ1

Dir(α)

Generating word distributions for all topics



wm,n

zm,n

β ~ϕk

~ϑm

document m

word ntopic k

α

Dir(β)

restau

rant

barfood

grill

concertmusic

rhythm bar

. . .

. . .

topic 2

topic 1


~ϕ2

~ϕ1

Dir(α)

topic 1

~ϑ1

topic 2 . . .

document 1

Generating topic distribution for document



wm,n

zm,n

β ~ϕk

~ϑm

document m

word ntopic k

α

Dir(β)

restau

rant

barfood

grill

concertmusic

rhythm bar

. . .

. . .

topic 2

topic 1


~ϕ2

~ϕ1

Dir(α)

topic 1

~ϑ1

topic 2 . . .

document 1

Sampling the topic index for first word, z = 2



wm,n

zm,n

β ~ϕk

~ϑm

document m

word ntopic k

α

Dir(β)

restau

rant

barfood

grill

concertmusic

rhythm bar

. . .

. . .

topic 2

topic 1


~ϕ2

~ϕ1

Dir(α)

topic 1

~ϑ1

topic 2 . . .

document 1

Sampling a word from term distribution for topic 2, “concert”


State of the art

Large number of published models that extend LDA:Authors (Rosen-Zvi et al. 2004),Citations (Dietz et al. 2007),Hierarchy (Li and McCallum 2006; Li et al. 2007),Image features and captions (Barnard et al. 2003) etc.Results for “topic model” (title + abstract) only since 2012: ACM>400, Google Scholar >1300.

→ Expanding research area with practical relevanceBut: No existing analysis as generic model classPartly tedious derivation, especially for inference algorithms

Conjecture:Important properties generic across modelsSimplifications for derivation of concrete model properties, inferencealgorithms and design methods


State of the art




wm,n

zm,n

cm, j

ym, j

~am

βγ

~ψk ~ϕk

~ϑx

α

xm, j xm,n

m ∈ [1,M]

n ∈ [1,Nm]j ∈ [1, Jm] k ∈ [1,K]k ∈ [1,K]

x ∈ [1, A]

Expert–tag–topic model

(Heinrich 2011)


State of the art





Research questions

“How can topic models be described in a generic way in order to usetheir properties across particular applications?”

“Can generic topic models be implemented generically and, if so,can repeated structures be exploited for optimisations?”

“How can generic models be applied to data in virtual communities?”


Overview

Introduction


Inference methods




“How can topic models be described in a generic way in order to usetheir properties across particular applications?”


Topic models: Example structures

m∈[1,M]

k∈[1,K] n∈[1,Nm]

zm,n

β ~ϕkwm,n

~ϑmα

(a) Latent Dirichlet allocation (LDA)

m∈[1,M ]

k∈[1,K] n∈[1,Nm]

zm,n

cm,n

~ϑc

c∈[1,C]

α

β ~ϕk wm,n

~ζm

(b) Author–topic model (ATM)

m∈[1,M]

n∈[1,Nm]

~β ~ϕk

ℓ∈[1,L]

wm,n

y∈[1,K]

~αx

~αr

z1m,n

z2m,n

~ϑrm

~ϑm,x z3m,n

(c) Pachinko allocation model (PAM4)

m∈[1,M]

n∈[1,Nm]

wm,n

T∈[1,|T |]

~α0 zTm,n

~ϑm,0

~ϑm,T ztm,n~αT

~β ~ϕk

k∈[1,|t|+|T |+1]

ℓm,n γ~ζT,t

T,t∈[1,|T ||t|]

(d) Hierarchical PAM (hPAM)(Blei et al. 2003; Rosen-Zvi et al. 2004; Li and McCallum 2006; Li et al. 2007)


Generic topic models – “NoMMs”

α

xout

Kxin

k= f (xin)

~θk

Dir(~θk |α)

~θ3

~θ2

~θ1

Generic characteristics of topic models:Levels with discrete components ~θk, generated from DirichletdistributionsCoupling via values of discrete variables x

→ “Network of mixed membership” (NoMM):Compact representation for topic modelsDirected acyclical graphNode: sample from mixture component, selection via incomingedges; terminal node: observationEdge: propagation of discrete values to child nodes.



α

xout

Kxin

k= f (xin)

α

~θk

xout

Kxin1

xin2

k= f (xin1 , xin2)

α

xout1

Kxin

xout2

k= f (xin)

~θk ~θk





α

Kk= f (xin)

α

~θk

Kxin1

xin2

k= f (xin1 , xin2)

α

xout1

K

xout2

k= f (xin)

~θk ~θk

x1 x2





~θk |αxin1 x1xin2 xout2

xout1

~θk |α~θk |αx2





~θk |αxin1 x1xin2 xout2

xout1

~θk |α~θk |αx2

Dir(~θk |α)Dir(~θk |α)Dir(~θk |α)





~ϑm |αm

~ϕk | βzm,n

Dir(~ϕk |α)Dir(~ϑk |α)

wm,n

LDA





~ϑm |αm=1

~ϕk | βzm,n


wm,n

LDA~ϑ1





~ϑm |αm=1

~ϕk | βz1,1=3


wm,n

LDA~ϑ1





~ϑm |αm=1

~ϕk | βz1,1=3


wm,n

LDA~ϕ3





~ϑm |αm=1

~ϕk | βz1,1=3


w1,1=2

LDA~ϕ3




Topic models as NoMMs

~ϑm | αzm,n=k

[V][K]

m

[M]

wm,n=t~ϕk | β

(a) Latent Dirichlet allocation, LDA

m

[M]

xm,n=x

[A]~ϑx|α

zm,n=k

[V][K]

wm,n=t~ϕk | β~am

(b) Author–topic model, ATM

~ϑrm | ~αr

m

[M]

z2m,n=x

[s1]~ϑm,x|~αx

z3m,n=y

[V][s2]

wm,n=t~ϑy | ~β

(c) Pachinko allocation model, PAMm

[M]zT

m,n=T

~ϑm,T |~αT

ztm,n=t ~ζT,t | γ

~ϑm,0|~α0

m

[M]

[V]

wm,n~ϕ`,T,t|β

`m,n[|T |]

[|t|]

[3][|T |+|t|+1]

zTm,n=T

(d) Hierarchical pachinko allocation model, hPAM(Blei et al. 2003; Rosen-Zvi et al. 2004; Li and McCallum 2006; Li et al. 2007)

m∈[1,M]

k∈[1,K] n∈[1,Nm]

zm,n

β ~ϕkwm,n

~ϑmα

m∈[1,M ]

k∈[1,K] n∈[1,Nm]

zm,n

cm,n

~ϑc

c∈[1,C]

α

β ~ϕk wm,n

~ζm

m∈[1,M]

n∈[1,Nm]

~β ~ϕk

ℓ∈[1,L]

wm,n

y∈[1,K]

~αx

~αr

z1m,n

z2m,n

~ϑrm

~ϑm,x z3m,n

m∈[1,M]

n∈[1,Nm]

wm,n

T∈[1,|T |]

~α0 zTm,n

~ϑm,0

~ϑm,T ztm,n~αT

~β ~ϕk

k∈[1,|t|+|T |+1]

ℓm,n γ~ζT,t

T,t∈[1,|T ||t|]


Overview

Introduction


Inference methods




“Can generic topic models be implemented generically. . . ?”


Bayesian inference problem and Gibbs sampler

Bayesian inference: “inversion of generative process”:Find distributions over parameters Θ and latent variables/topics H,given observations V and Dirichlet parameters A

= Determine posterior distribution p(H, Θ |V ,A)Intractability→ approximative approachesGibbs sampling: Variant of Markov-Chain Monte Carlo (MCMC)

In topic models: Marginalise parameters Θ (“Collapsed” GS)Sample topics Hi for each data point i in turn: Hi ∼ p(Hi |H¬i,V ,A)

xm y w

VΘ1 H1 Θ2 H2 Θ3

~ϑrm |αr ~ϑm,x|α ~ϕy | β






~ϑrm |αr

m~ϑm,x|α

w~ϕy | β

Θ1 H1 Θ2 H2 Θ3 V

? ? ?

? ?

p( , , , , | , A)






~ϑrm |αr

m~ϑm,x|α

w~ϕy | β? ?

H1i H2

i Vp( , | , A)H¬i ,H1i ,H

2i ∼


Sampling distribution for NoMMs

xm y w~ϑr

m |αr ~ϑm,x|α ~ϕy | β

Gibbs sampler can be generically derived (Heinrich 2009)

Typical case: Quotients of factors over levels `:

p(Hi|H¬i,V ,A) ∝∏

`

n¬i

k,t + α∑t n¬i

k,t + α

[`]

nk,t = count of co-occurrences between input and output values of alevel (components and samples)

More complex variants covered by q(k, t) ,beta({nk,t}Tt=1 + α)

beta({n¬ik,t}Tt=1 + α)


Sampling distribution for NoMMs

xm y wq(m, x) q((m,x), y) q(y,w)

Gibbs sampler can be generically derived (Heinrich 2009)

Typical case: Quotients of factors over levels `:

p(Hi|H¬i,V ,A) ∝∏

`

n¬i

k,t + α∑t n¬i

k,t + α

[`]

=∏

`

[q(k, t)

][`]

nk,t = count of co-occurrences between input and output values of alevel (components and samples)

More complex variants covered by q(k, t) ,beta({nk,t}Tt=1 + α)

beta({n¬ik,t}Tt=1 + α)


Typology of NoMM substructures

~ϑa |α

~ϑb |α

a

b~ϑk |α

z

a~ϑz |α

b

~ϑa |α~ϑz |α

c

c

~ϑa |α

~ϑb |α

a

b~ϑz |α

c

x

z1

yza

~ϑz |αb

~ϑa |α

N1. Dirichlet–multinomial

v

za~ϑz |α~ϑc

a

b

x

a~ϑx |α

b

~ϑa |α y

~ϑy |αc

q(a, z) q(z, b) q(a, x ⊕ y) q(x, b) q(y, c) q(a, x) q(b, y) q(k, c), k = f (x, y)

ϑca,z q(z, b) q(a, z) q(z, b) q(z, c) ≈ q(a, z1) q(b, z2) q(z1, c ⊕ c) q(z2, c ⊕ c)

E2. Autonomous edges C2. Combined indices

N2. Observed parameters E3. Coupled edges C3. Interleaved indices

z2

NoMM substructures: Nodes, edges/branches, componentindices/merging of edges:

Representation via q-functions and likelihoodMultiple samples per data point: q(a, x⊕ y) for respective level

“Library” incl. additional structures: alternative distributions,regression, aggregation etc. ↔ q-functions + other factors


Implementation: Gibbs “meta-sampler”

NoMM codegenerator

C/Java codeprototype

Java VM native platform

optimise

codetemplates

generate

Java modelinstance

C/Java codemodule

validate deploydata =⇒

compile

data =⇒

specificationtopic model

︸︷︷︸︸︷︷︸

Code generator for topic models in Java and C

Separation of knowledge domains: topic model applications vs.machine learning vs. computing architecture


Implementation: Gibbs “meta-sampler”

codegenerator

C/Java codeprototype

Java VM native platform

optimise

codetemplates

generate

Java modelinstance

C/Java codemodule

validate deploydata =⇒

compile

data =⇒

specificationtopic model

︸︷︷︸︸︷︷︸

NoMM

Code generator for topic models in Java and C

Separation of knowledge domains: topic model applications vs.machine learning vs. computing architecture


Example NoMM script and generated kernel: hPAM2

~ϕk | βwmn

xmn=x

[V]

~ϑm |αm[M]

[Y]

[X]ymn=y

~ϑxm,x|αx

xm

[M] x = 0 : k = 0x , 0, y = 0 : k = 1 + x

x, y , 0 : k = 1 + X + y

model = HPAM2

description:

Hierarchical PAM model 2 (HPAM2)

sequences:

# variables sampled for each (m,n)

w, x, y : m, n

network:

# each line one NoMM node

m >> theta | alpha >> x

m,x >> thetax | alphax[x] >> y

x,y >> phi[k] >> w

# java code to assign k

k : {

if (x==0) { k = 0; }

else if (y==0) k = 1 + x;

else k = 1 + X + y;

}.

−→

// hidden edge

for (hx = 0; hx < X; hx++) {

// hidden edge

for (hy = 0; hy < Y; hy++) {

mxsel = X * m + hx;

mxjsel = hx;

if (hx == 0)

ksel = 0;

else if (hy == 0)

ksel = 1 + hx;

else

ksel = 1 + X + hy;

pp[hx][hy] = (nmx[m][hx] + alpha[hx])

* (nmxy[mxsel][hy] + alphax[mxjsel][hy])

/ (nmxysum[mxsel] + alphaxsum[mxjsel])

* (nkw[ksel][w[m][n]] + beta)

/ (nkwsum[ksel] + betasum);

psum += pp[hx][hy];

} // for h

} // for h

sub

root

sup

sub sub

supwords

words

words

words words words


Example NoMM script and generated kernel: hPAM2

~ϕk | βwmn

xmn=x

[V]

~ϑm |αm[M]

[Y]

[X]ymn=y

~ϑxm,x|αx

xm

[M] x = 0 : k = 0x , 0, y = 0 : k = 1 + x

x, y , 0 : k = 1 + X + y

model = HPAM2

description:

Hierarchical PAM model 2 (HPAM2)

sequences:

# variables sampled for each (m,n)

w, x, y : m, n

network:

# each line one NoMM node

m >> theta | alpha >> x

m,x >> thetax | alphax[x] >> y

x,y >> phi[k] >> w

# java code to assign k

k : {

if (x==0) { k = 0; }

else if (y==0) k = 1 + x;

else k = 1 + X + y;

}.

−→

// hidden edge

for (hx = 0; hx < X; hx++) {

// hidden edge

for (hy = 0; hy < Y; hy++) {

mxsel = X * m + hx;

mxjsel = hx;

if (hx == 0)

ksel = 0;

else if (hy == 0)

ksel = 1 + hx;

else

ksel = 1 + X + hy;

pp[hx][hy] = (nmx[m][hx] + alpha[hx])

* (nmxy[mxsel][hy] + alphax[mxjsel][hy])

/ (nmxysum[mxsel] + alphaxsum[mxjsel])

* (nkw[ksel][w[m][n]] + beta)

/ (nkwsum[ksel] + betasum);

psum += pp[hx][hy];

} // for h

} // for h


Document–Topic distribution in Gibbs sampler

Iteration 1

Document–topic matrix ϑ (200 documents, 50 topics)



Iteration 5




Iteration 10




Iteration 15




Iteration 20




Iteration 30




Iteration 40




Iteration 50




Iteration 60




Iteration 80




Iteration 100




Iteration 120




Iteration 150




Iteration 200




Iteration 300




Iteration 500, converged↔ stationary state



Fast sampling: Hybrid scaling methods

1 10 100 1000iterations

depend.indep.

5000

perp

lexi

ty

PAM4

model dim. speedup

LDA 500 30.2

PAM4 40 × 40 7.4

PAM4 24.1

PAM4 49.8

par. indep.ser.

✓✓

✓✓✓✓✓ ✓

40 × 40

40 × 40

—

Serial and parallel scaling methods:Generalised results for LDA to generic NoMMs, specifically (Porteouset al. 2008; Newman et al. 2009) + novel approach

Problem: Sampling space for stat. dependent variables: K × L × . . .→ Independence assumption: Separate samplers with dimensions

K + L + . . . � K × L × . . .Empirical result: Iterations ↑, but topic quality comparable

→ Hybrid approaches with independent samplers highly effectiveImplementation: complexity covered by meta-sampler


Overview

Introduction


Inference methods




“How can generic models be applied to data in virtual communities?”


NoMM design process~ϑa |α

~ϑb |α

a

b~ϑk |α

z

a~ϑz |α

b

~ϑa |α~ϑz |α

c

c

~ϑa |α

~ϑb |α

a

b~ϑz |α

c

x

z1

yza

~ϑz |αb

~ϑa |α

N1. Dirichlet–multinomial

v

za~ϑz |α~ϑc

a

b

x

a~ϑx |α

b

~ϑa |α y

~ϑy |αc

q(a, z) q(z, b) q(a, x ⊕ y) q(x, b) q(y, c) q(a, x) q(b, y) q(k, c), k = f (x, y)

ϑca,z q(z, b) q(a, z) q(z, b) q(z, c) ≈ q(a, z1) q(b, z2) q(z1, c ⊕ c) q(z2, c ⊕ c)

E2. Autonomous edges C2. Combined indices

N2. Observed parameters E3. Coupled edges C3. Interleaved indices

z2

Typology→ “Library” of NoMM substructures→ Idea: Construct models from simple substructures that connect

terminal nodes:Terminal nodes↔ multimodal data (virtual communities...)Substructures↔ relationships in data; latent semantics

→ Process:Assumptions on dependencies in dataIterative association to structures in model (usage of typology)

Gibbs distribution known! ↔ model behaviour: q(x, y) = “rich get richer”Implementation and test with Gibbs meta-sampler; possibly iteration


NoMM design process

1 2 3 4 51 2 3 4 5Define Definemodelling task Create

and metricsevidence model terminals

Formulatemodel

assumptions

Composemodel and

predict properties

6 7 8 9 106 7 8 9 10Write Generateand adapt Implement

NoMM script target metric

Evaluatebased on

test corpus

Optimiseand integrate fortarget platformGibbs sampler

Typology→ “Library” of NoMM substructures→ Idea: Construct models from simple substructures that connect

terminal nodes:Terminal nodes↔ multimodal data (virtual communities...)Substructures↔ relationships in data; latent semantics

→ Process:Assumptions on dependencies in dataIterative association to structures in model (usage of typology)

Gibbs distribution known! ↔ model behaviour: q(x, y) = “rich get richer”Implementation and test with Gibbs meta-sampler; possibly iteration


Application: Expert finding with tag annotations

authorship

annotationRetrieval performance.The most prominentretrieval measures areprecision and recall.Recall is defined asthe ratio between thenumber of retrievedrelevant items tothe total number ofexisting relevant items.Precision is defined asthe ratio between the

authorship

Scenario: Expert finding via documents with tag annotationsAuthors of relevant documents→ expertsFrequently documents with additional annotations, here: tags

→ Goal: Enable tag queries, improve quality of text queriesProblem: Tags often incomplete, partly wrong

→ Connection of tags and experts via topics

(1) Data: For each document m: text ~wm, authors ~am, tags ~cm

(2) Goal: Tag query ~c ′: p(~c ′| a) = max, word query ~w ′: p(~w ′| a) = max(3) Terminal nodes: Authors in input, words and tags in output


Application: Expert finding with tag annotations


14~wm 13

5 14 33AB

12

TH

AB

TH

5

33

~am ~cm

document m

authors tags

Scenario: Expert finding via documents with tag annotationsAuthors of relevant documents→ expertsFrequently documents with additional annotations, here: tags

→ Goal: Enable tag queries, improve quality of text queriesProblem: Tags often incomplete, partly wrong

→ Connection of tags and experts via topics

(1) Data: For each document m: text ~wm, authors ~am, tags ~cm

(2) Goal: Tag query ~c ′: p(~c ′| a) = max, word query ~w ′: p(~w ′| a) = max(3) Terminal nodes: Authors in input, words and tags in output


Model assumptions


14~wm 13

5 14 33AB

12

TH

AB

TH

5

33

~am ~cm

document m

authors tags

(4) Model assumptions:(a) Expertise of an author is weighted with the portion of authorship(b) Semantics of expertise expressed by topics z. Each author has a

single field of expertise (topic distribution).(c) Semantics of tags expressed by topics y


Model assumptions


14~wm 13

5 14 33

AB

TH

5

33~cm

AB

12

TH

~am

document m

authors tags

(4) Model assumptions:(a) Expertise of an author is weighted with the portion of authorship(b) Semantics of expertise expressed by topics z. Each author has a

single field of expertise (topic distribution).(c) Semantics of tags expressed by topics y


Model construction

~cm

~amwm,n

[M,Nm]

p(. . . |~a, ~w, ~c) ∝ . . .

authors word

tags

(5) Model construction: (a) Start with terminal nodes (from step 3)


Model construction

~cm

. . .xm,n

~am

m wm,n[M,Nm]

p(x, . . . |·) ∝ am,x q(x, . . . ) . . .

document author word

tags

(b) Authorship ~am given as observed distribution→ node sampels author x of a word


Model construction

~cm

q(x, z)xm,n

~am

m zm,nwm,n

[M,Nm]

p(x, z, . . . |·) ∝ am,x q(x, z) . . .

document author word topic word

tags

(c) Each author has only a single field of expertise (topic distribution)→ q(x, z) associates (word-)topics with sampled authors x (cf. ATM)


Model construction

~cm

q(x, z)xm,n

~am

mq(z,w)

zm,nwm,n

[M,Nm]

p(x, z, . . . |·) ∝ am,x q(x, z) q(z,w) . . .

document author word topic word

tags

(d) Topic distribution over terms→ connect z and w via q(z,w)


Model construction

cm, jq(x,z⊕y)

xm,n∪j~am

mq(z,w)zm,n

wm,n[M,Nm]

p(x, z, y |·) ∝ am,x q(x, z ⊕ y) q(z,w) q(y, c)

q(y, c) [M, Jm]

ym, jdocument author

word topic

tag topic

word

tag

(e) Introduce tag topics ym,j for cm,j as distributions over tagsq(x, z ⊕ y) overlays values for z and y


Model construction

cm, jq(x,z⊕y)

xm,n~am

mq(z,w)zm,n

wm,n[M,Nm]


q(y, c) [M, Jm]


word topic

tag topic

word

tag



Model construction

cm, jq(x,z⊕y)

xm, j~am

mq(z,w)zm,n

wm,n[M,Nm]


q(y, c) [M, Jm]


word topic

tag topic

word

tag



Model construction – ordinary approach

wm,n

zm,n

cm, j

ym, j

~am

βγ

~ψk ~ϕk

~ϑx

α

xm, j xm,n

m ∈ [1,M]

n ∈ [1,Nm]j ∈ [1, Jm] k ∈ [1,K]k ∈ [1,K]

x ∈ [1, A]

Expert–tag–topic model

(Heinrich 2011)


Expert–tag–topic model: Evaluation

ATM ETT

AP@

10

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ETT

word queries tag queries

ATM ETT-500

-450

-400

-350

-300

-250

-200

-150

cohe

renc

esc

ore

NIPS Corpus: 2.3 million words, 2037 authors, 165 tagsRetrieval: Average Precision @10:

Term queries: ETT > ATMTag queries: Similarly good AP values

Topic coherence (Mimno et al. 2011): ETT > ATMSemi-supervised learning: Tag queries retrieve items without tags


Overview

Introduction


Inference methods




Conclusions: Resesarch contributions

Networks of Mixed Membership: Generic model anddomain-specific compact representation of topic modelsInference algorithms: Generic Gibbs sampler

Fast sampling methods (serial, parallel, independent)Implementation in Gibbs meta-sampler

Design process based on typology of NoMM substructures

Application to virtual communities: Expert–tag–topic model forexpert finding with annotated documents

∑Contribution to facilitated “model-based” construction of topicmodels, specifically for virtual communities and other multimodalscenarios





+ Variational Inference for NoMMs (Heinrich and Goesele 2009)

Design process based on typology of NoMM substructures








Design process based on typology of NoMM substructures+ AMQ model: Meta-model for virtual communities as formal basis for

scenario modelling (Heinrich 2010)








Design process based on typology of NoMM substructures+ AMQ model: Meta-model for virtual communities as formal basis for

scenario modelling (Heinrich 2010)


+ Models ETT2 and ETT3 incl. novel NoMM structure; retrievalapproaches (Heinrich 2011b)






+ Variational Inference for NoMMs (Heinrich and Goesele 2009)Design process based on typology of NoMM substructures

+ AMQ model: Meta-model for virtual communities as formal basis forscenario modelling (Heinrich 2010)



+ Graph-based expert search using ETT models: Integration ofexplorative search/browsing and distributions from topic models∑

Contribution to facilitated “model-based” construction of topicmodels, specifically for virtual communities and other multimodalscenarios


Conclusions: Resesarch contributions4/15/12 1:47 AM

Page 1 of 2file:///data/workspace/knowceans-freshmind-lucene3/fmica.svg

1.

2.

4.

8.

3.

6.

9.

7.

10.

5.

authors

authors

authorsauthors

cites

authorsauthors

cites

cites

authors

authors

authors

authors

cites

cites

authors

cites

authors

cites

authors

cites

cites

authorscites

cites

authors

authors

authors

authorscites

authorsauthors

cites cites

authors

cites

authors

citescitesauthors

authors

authors

authors

authors

cites

cites

authors

[blind source separation] [independent component analysis]










Yang_H1


Cichocki_A

Bell_A2

Rokni_U

Lee_T1

Parra_L1

Shouval_H

Oja_E1

Lin_J

Hyvarinen_A

A New Learning Algorithm for Blind Signal Separation

9

Independent Component Analysis of Eiectroencephalographic Data

4

Unmixing Hyperspectral Data,

4

Factorizing Multivariate Function Classes,

1

A Non-linear Information Maximisation Algorithm that Performs Blind Separation

4

Maximum Likelihood Blind Source Separation A Context-Sensitive Generalization of ICA,

4

Semiparametric Approach to Multichannel Blind Deconvolution of Nonminimum Phase Systems,

2

One-unit Learning Rules for Independent Component Analysis,

Extended ICA Removes Artifacts from Electroencephalographic Recordings,

8

Multiplicative Updating Rule for Blind Separation Derived from the Method of Scoring,

1

Algorithms for Independent Components Analysis and Higher Order Statistics,

2

Edges are the "Independent Components" of Natural Scenes,

2

Blind Separation of Delayed and Convolved Sources,

2

The Efficiency and the Robustness of Natural Gradient Descent Learning Rule,

2

Receptive Field Formation in Natural Scene Environments Comparison of Single Cell Learning Rules,

4

New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit,

Search for Information Bearing Components in Speech,

1

Source Separation and Density Estimation by Faithful Equivariant SOM,

2

Data Visualization and Feature Selection New Algorithms for Nongaussian Data,

1

Independent Component Analysis for Identification of Artifacts in Magnetoencephalographic Recordings,

5

Unsupervised Classification with Non-Gaussian Mixture Models Using ICA,

3

Sparse Code Shrinkage.' Denoising by Nonlinear Maximum Likelihood Estimation,

2

Symplectic Nonlinear Component Analysis

Blind Separation of Filtered Sources Using State-Space Approach,

1




Figure: ETT1: Expert search in community browser





+ Variational Inference for NoMMs (Heinrich and Goesele 2009)Design process based on typology of NoMM substructures

+ AMQ model: Meta-model for virtual communities as formal basis forscenario modelling (Heinrich 2010)



+ Graph-based expert search using ETT models: Integration ofexplorative search/browsing and distributions from topic models∑

Contribution to facilitated “model-based” construction of topicmodels, specifically for virtual communities and other multimodalscenarios


Outlook

New applications and NoMM structures, e.g., time as variableAlternative inference methods:

Generic Collapsed Variational Bayes (Teh et al. 2007): Structuresimilar to Collapsed Gibbs-SamplerNon-parametric methods: Learning model dimensions using Dirichletor Pitman–Yor process priors (Teh et al. 2004; Buntine and Hutter2010), NoMM polymorphism (Heinrich 2011a)

Improved support in design process:Data-driven design: Search over model structures to obtain bestmodel for data setArchitecture-specific Gibbs meta-sampler, e.g., massively-parallel orFPGA, cf. (Heinrich et al. 2011)

Integration with interactive user interfaces: Models can be createdon the fly, e.g., for visual analytics


Thank you!

Q + A


References I

References

Barnard, K., P. Duygulu, D. Forsyth, N. de Freitas, D. Blei, and M. Jordan (2003, August).Matching words and pictures.JMLR – Special Issue on Machine Learning Methods for Text and Images 3(6), 1107–1136.

Bellegarda, J. (2000, August).Exploiting latent semantic information in statistical language modeling.Proc. IEEE 88(8), 1279–1296.

Blei, D., A. Ng, and M. Jordan (2003, January).Latent Dirichlet allocation.Journal of Machine Learning Research 3, 993–1022.

Buntine, W. and M. Hutter (2010).A Bayesian review of the Poisson-Dirichlet process.arXiv:1007.0296v1 [math.ST].

Chang, J., J. Boyd-Graber, S. Gerrish, C. Wang, and D. Blei (2009).Reading tea leaves: How humans interpret topic models.In Proc. Neural Information Processing Systems (NIPS).


References II

Dietz, L., S. Bickel, and T. Scheffer (2007, June).Unsupervised prediction of citation influences.In Proceedings of the 24th International Conference on Machine Learning, Corvallis, Oregon,

USA.

Heinrich, G. (2009).A generic approach to topic models.In Proc. European Conf. on Mach. Learn. / Principles and Pract. of Know. Discov. in Databases

(ECML/PKDD), Part 1, pp. 517–532.

Heinrich, G. (2010).Actors—media—qualities: a generic model for information retrieval in virtual communities.In Proc. 7th International Workshop on Innovative Internet Community Systems (I2CS 2007), part

of I2CS Jubilee proceedings, Lecture Notes in Informatics, GI.

Heinrich, G. (2011a, March).“Infinite LDA” – Implementing the HDP with minimum code complexity.Technical note TN2011/1, arbylon.net.

Heinrich, G. (2011b).Typology of mixed-membership models: Towards a design method.In Proc. European Conf. on Mach. Learn. / Principles and Pract. of Know. Discov. in Databases

(ECML/PKDD).


References III

Heinrich, G. and M. Goesele (2009).Variational Bayes for generic topic models.In Proc. 32nd Annual German Conference on Artificial Intelligence (KI2009).

Heinrich, G., J. Kindermann, C. Lauth, G. Paaß, and J. Sanchez-Monzon (2005).Investigating word correlation at different scopes – a latent concept approach.In Workshop Lexical Ontology Learning at Int. Conf. Mach. Learning.

Heinrich, G., F. Logemann, V. Hahn, C. Jung, G. Figueiredo, and W. Luk (2011).HW/SW co-design for heterogeneous multi-core platforms: The hArtes toolchain, Chapter Audio

array processing for telepresence, pp. 173–207.Springer.

Li, W., D. Blei, and A. McCallum (2007).Mixtures of hierarchical topics with pachinko allocation.In International Conference on Machine Learning.

Li, W. and A. McCallum (2006).Pachinko allocation: DAG-structured mixture models of topic correlations.In ICML ’06: Proceedings of the 23rd international conference on Machine learning, New York,

NY, USA, pp. 577–584. ACM.


References IV

Mimno, D., H. M. Wallach, E. Talley, M. Leenders, and A. McCallum (2011, July).Optimizing semantic coherence in topic models.In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,

Edinburgh, UK, pp. 262272.

Newman, D., A. Asuncion, P. Smyth, and M. Welling (2009, August).Distributed algorithms for topic models.JMLR 10, 1801–1828.

Porteous, I., D. Newman, A. Ihler, A. Asuncion, P. Smyth, and M. Welling (2008).Fast collapsed Gibbs sampling for latent Dirichlet allocation.In KDD ’08: Proceeding of the 14th ACM SIGKDD international conference on Knowledge

discovery and data mining, New York, NY, USA, pp. 569–577. ACM.

Rosen-Zvi, M., T. Griffiths, M. Steyvers, and P. Smyth (2004).The author-topic model for authors and documents.In Proc. 20th Conference on Uncertainty in Artificial Intelligence (UAI).

Teh, Y., M. Jordan, M. Beal, and D. Blei (2004).Hierarchical Dirichlet processes.Technical Report 653, Department of Statistics, University of California at Berkeley.

Teh, Y. W., D. Newman, and M. Welling (2007).A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation.In Advances in Neural Information Processing Systems, Volume 19.


Appendix


Example: Text mining for semantic clusters

Topic label Dominant terms according to ϕk,t = p(term |topic)

Bundesliga FC SC Munchen Borussia SV VfL Kickers SpVgg Uhr Koln Bochum Freiburg VfBEintracht Bayern Hamburger Bayern+Munchen

Polizei / Unfall Polizei verletzt schwer Auto Unfall Fahrer Angaben schwer+verletzt Menschen Wa-gen Verletzungen Lawine Mann vier Meter Straße

Tschetschenien Rebellen russischen Grosny russische Tschetschenien Truppen Kaukasus MoskauAngaben Interfax tschetschenischen Agentur

Politik / Hessen FDP Koch Hessen CDU Koalition Gerhardt Wagner Liberalen hessischen Wester-welle Wolfgang Roland+Koch Wolfgang+Gerhardt

Wetter Grad Temperaturen Regen Schnee Suden Norden Sonne Wetter Wolken Deutsch-land zwischen Nacht Wetterdienst Wind

Politik / Kroatien Parlament Partei Stimmen Mehrheit Wahlen Wahl Opposition Kroatien PrasidentParlamentswahlen Mesic Abstimmung HDZ

Die Grunen Grunen Parteitag Atomausstieg Trittin Grune Partei Trennung Mandat Ausstieg AmtRoestel Jahren Muller Radcke Koalition

Russische Politik Russland Putin Moskau russischen russische Jelzin Wladimir Tschetschenien Rus-slands Wladimir+Putin Kreml Boris Prasidenten

Polizei / Schulen Polizei Schulen Schuler Tater Polizisten Schule Tat Lehrer erschossen BeamtenMann Polizist Beamte verletzt Waffe

Bigram-LDA: Topics from 18400 dpa news messages, Jan. 2000 (Heinrich et al. 2005)


Notation: Bayesian network vs. NoMM levels

α

~ϑk

xi

K

ki

~ϑk |α [T ]

xiki

[K]

parameters ϑ + hyperparameters α⇔ nodes (ϑ |α)

variables ki, xi ⇔ edges ki, xi

plates (i.i.d. repetitions) i, k ⇔ indexes i + dimensions k


NoMM representation: Variable dependencies

α

~ϑk ~ϑk ~ϑk

α

~ϑk

vi

α

~ϑk

α

~ϑk

α

`=1 `=2 `=3 `=4 `=5 `=6

α

~ϑk Θ

A

X

vi

`=8`=7

α

~ϑk

vihi

hihi

hi hi

~ϑk |α

`=1

`=2 `=3

`=4

`=5 `=6

~ϑk |α ~ϑk |α

~ϑk |α~ϑk |α ~ϑk |α

~ϑk

`=8

h1i

h2i

h4i

h3i v5

i h6i

v8i

`=7

~ϑk |αv7

i

k1i

k2i


Collapsed Gibbs sampler

~x (0)

x2

x1

~x ∗

p(~x |V)

Collapsed Gibbs sampler: Stochastic EM / MCMC:NoMMs: parameters Θ correlate with H → marginaliseFor each data point, i: draw latent variables, Hi = (yi, zi, . . . ), given allother data, latent, H¬i, and observed, V:

Hi ∼ p(Hi |H¬i,V ,A) . (1)

Stationary state: full conditional distribution (1) simulates posterior

Faster absolute convergence for NoMMs than, e.g., variationalinference (Heinrich and Goesele 2009)



~x (0)

x2

x1

~x ∗~x continuous, i ∈ [1, 2]

H discrete, i ∈ [1,W]l


Hi ∼ p(Hi |H¬i,V ,A) . (1)

Stationary state: full conditional distribution (1) simulates posterior

Faster absolute convergence for NoMMs than, e.g., variationalinference (Heinrich and Goesele 2009)



x2

x1i = 1

~xp(x1 | x2)


Hi ∼ p(Hi |H¬i,V ,A) . (1)

Stationary state: full conditional distribution (1) simulates posteriorFaster absolute convergence for NoMMs than, e.g., variationalinference (Heinrich and Goesele 2009)
