Topic Cube: Topic Modeling for OLAP on Multidimensional Text ...
Modeling Knowledge Incorporation into Topic Models and ...
Transcript of Modeling Knowledge Incorporation into Topic Models and ...
![Page 1: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/1.jpg)
Modeling Knowledge Incorporation into Topic Models
and their EvaluationSilvia [email protected]@TerragniSilvia
1
EURECOM (from Milan), 17/06/2021
![Page 2: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/2.jpg)
Outline● Introduction and state of the art of topic models
● Incorporating knowledge into topic models○ relationships between documents and words
○ pre-trained contextualized representations
● Evaluation of topic models○ framework for comparing topic models
○ hyperparameter optimization
2
![Page 3: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/3.jpg)
Topic Modeling
3
![Page 4: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/4.jpg)
What is Topic Modeling
4
TOPIC MODEL
Corpus of documents
TOPIC 1
Supervised
learning
classify
prediction
class
TOPIC 2
Learning
reinforce
reward
agent
q-learning
TOPIC 3
clustering
learning
model
similarity
centroid
Topic indicators
Distribution of topics in each document
![Page 5: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/5.jpg)
Topic Models as probabilistic models
5
The human genome is the complete set of nucleic acid sequences for humans, encoded as DNA within the 23 chromosome pairs in cell nuclei and in a small DNA molecule found within individual mitochondria…
We can express a document as a multinomial distribution over the topics: a document talks about different topics in different proportions
EvolutionDisease
Human
a document that talks about diseases only
a document that talks about human evolution
a document that talks about evolution, diseases and a little about humans
![Page 6: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/6.jpg)
Topic Models as probabilistic models
6
Human
Genome
Dna
Genetic
Genes
Sequence
Gene
Molecular
Map
This is not just a unordered list of words. We can expressed it as a multinomial distribution over the vocabulary
It’s a probability distribution! It sums to 1
![Page 7: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/7.jpg)
Latent Dirichlet Allocation
7
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
![Page 8: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/8.jpg)
Latent Dirichlet Allocation
Topics are expressed by a multinomial distribution over the vocabulary
8
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
vocabulary
![Page 9: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/9.jpg)
Latent Dirichlet Allocation
Dirichlet hyperparameter that controls how the sparsity of the words characterizing a topic
9
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
![Page 10: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/10.jpg)
Latent Dirichlet Allocation
10
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
A document is expressed as a multinomial distribution
Document-topic distribution
![Page 11: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/11.jpg)
Latent Dirichlet Allocation
The hyperparameter that controls the sparsity of the topics in a document
11
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
![Page 12: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/12.jpg)
Latent Dirichlet Allocation
A topic is assigned to each word
12
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
![Page 13: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/13.jpg)
Latent Dirichlet Allocation
Words are sampled from the word distribution given the topic assignment.
13
● Most known topic model: LDA [Blei+ 03]
● Fully unsupervised (the only observations are the words in documents)
![Page 14: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/14.jpg)
State-of-the-art Topic models
14
■ Usually based on Latent Dirichlet Allocation (LDA) [Blei et al., 2003]
■ Increase the capacity of the model by extending LDA:– relaxing some assumptions of the model
[Wallach et al., 2006]– incorporating external knowledge [Nguyen et
al., 2015]– changing the representation of words [Das et
al., 2015]
![Page 15: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/15.jpg)
● Neural Topic Models:○ usually based on Variational
Autoencoders (VAEs) [Miao et al., 2016]
○ the encoder discovers the latent topic document representation
○ the top-words of a topic are obtained by the weight matrix that reconstructs the BOW
15
State-of-the-art Topic models Document as bag of words
Reconstructed document
Document-topic representation
![Page 16: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/16.jpg)
16
RQ1: How can we incorporate knowledge into topic models?
RQ2: How can we ensure fairer comparisons between the models?
Research Questions
![Page 17: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/17.jpg)
Incorporating Knowledge in Topic Models:Relationships
17
![Page 18: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/18.jpg)
Relational Topic ModelsMost topic models assume that documents and its constituents (i.e. words) are
independent from each other
citationmachine learning
semantic relationship
18
Word-level [Yang et al, 2015;
Nguyen et al, 2015]
Document-level [Chang et al., 2009;
Yang et al., 2016]
RQ1: How can we incorporate knowledge into topic models?
![Page 19: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/19.jpg)
(Document) Relational Topic Models
19
citation
RQ1: How can we incorporate knowledge into topic models?
random variable encodes relationship
Document d Document d’
Chang, J.& Blei, D.M.: Relational Topic Models for Document Networks. AISTATS 2009: 81-88 (2009)
![Page 20: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/20.jpg)
Document Constrained Relational Topic Models
20
conference
Document labels in the form of relationships: Two documents that share the same label are more likely to share the same topics
citation
constraint on the topic assignment
RQ1: How can we incorporate knowledge into topic models?
S. Terragni, E. Fersini, E. Messina. Constrained Relational Topic Models. Information Sciences 512: 581-594 (2020) https://github.com/MIND-Lab/Constrained-RTM
Document d Document d’
![Page 21: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/21.jpg)
the higher the better
Document Constrained Relational Topic Models
21
conference
citation
RQ1: How can we incorporate knowledge into topic models?
Document labels in the form of relationships: Two documents that share the same label are more likely to share the same topics
S. Terragni, E. Fersini, E. Messina. Constrained Relational Topic Models. Information Sciences 512: 581-594 (2020) https://github.com/MIND-Lab/Constrained-RTM
![Page 22: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/22.jpg)
citation
Entity Constrained Relational Topic Models
related concepts
related concepts
22
■ Relationships between documents (RTM)
■ Relationships between words and entities: two named-entities or words that are related are more likely to share the same topics
RQ1: How can we incorporate knowledge into topic models?
S. Terragni, D. Nozza, E. Fersini, E. Messina. Which Matters Most? Comparing the Impact of Concept and Document Relationships in Topic Models. Insights @ EMNLP 2020 [https://github.com/MIND-Lab/EC-RTM]
![Page 23: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/23.jpg)
Entity Constrained Relational Topic Models■ Relationships between documents (RTM)
■ Relationships between words and entities: two named-entities or words that are related are more likely to share the same topics
the higher the better
23
citation
related concepts
related concepts
RQ1: How can we incorporate knowledge into topic models?
S. Terragni, D. Nozza, E. Fersini, E. Messina. Which Matters Most? Comparing the Impact of Concept and Document Relationships in Topic Models. Insights @ EMNLP 2020 [https://github.com/MIND-Lab/EC-RTM]
![Page 24: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/24.jpg)
Entity Constrained Relational Topic Models■ Relationships between documents (RTM)
■ Relationships between words and entities: two named-entities or words that are related are more likely to share the same topics
the higher the better
24
citation
related concepts
related concepts
RQ1: How can we incorporate knowledge into topic models?
S. Terragni, D. Nozza, E. Fersini, E. Messina. Which Matters Most? Comparing the Impact of Concept and Document Relationships in Topic Models. Insights @ EMNLP 2020 [https://github.com/MIND-Lab/EC-RTM]
![Page 25: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/25.jpg)
Incorporating Knowledge in Topic Models:Pre-trained Representations
25
![Page 26: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/26.jpg)
Why using pre-trained representations
26
Sentence
BERT
0 1 6 9 1 5
RQ1: How can we incorporate knowledge into topic models?
- capture syntactic and semantic information of the sentence
- can be multilingual- handle out-of-vocabulary (OOV)
words
![Page 27: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/27.jpg)
27
Contextualized Topic Models: Combined CTM
sampling
document BOW representation
topic document representation
hidden layers
reconstructed BOW representation
contextualized representation
27
concatenation
Bianchi, F., Terragni, S., & Hovy, D. (2020). Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. ACL 2021
RQ1: How can we incorporate knowledge into topic models?
Combined CTM
Open-source python library: https://github.com/MilaNLProc/contextualized-topic-modelsWe reached over 32k downloads and 440 github stars :)
- concatenation of BOW and Sentence BERT
- improve the coherence of the topics
- effective on short texts- RoBERTa outperforms BERT
![Page 28: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/28.jpg)
Contextualized Topic Models: Zero-shot CTM
28
What if we replace the BOW representation with pre-trained multilingual representations?
We can zero-shot predict the topics of a document in an unseen language
also multilingual
sampling
topic document representation
hidden layers
reconstructed BOW representation
contextualized representation
Bianchi, F., Terragni, S., Hovy, D., Nozza, D., & Fersini, E. (2020). Cross-lingual Contextualized Topic Models with Zero-shot Learning. EACL 2021
RQ1: How can we incorporate knowledge into topic models?
Open-source python library: https://github.com/MilaNLProc/contextualized-topic-modelsWe reached over 32k downloads and 440 github stars :)
![Page 29: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/29.jpg)
29
Contextualized Topic Models: Zero-shot CTM
29
RQ1: How can we incorporate knowledge into topic models?
Text Lang Topic Prediction
Blackmore’s Night is a British/American traditional folk....
EN
rock, band, bass, formed, ...
I Blackmore’s Night sono la band fondatrice del renaissance rock...
IT
rock, band, bass, formed, ….
On nomme fourmi de Langton un automate cellulaire... FR
mathematics, theory, space, numbers, ...
Die Ameise ist eine Turingmaschine mit einem zweidimensionalen...
DE
mathematics, theory, space, numbers, …
also multilingual
sampling
topic document representation
hidden layers
reconstructed BOW representation
contextualized representation
Bianchi, F., Terragni, S., Hovy, D., Nozza, D., & Fersini, E. (2020). Cross-lingual Contextualized Topic Models with Zero-shot Learning. EACL 2021
![Page 30: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/30.jpg)
Evaluating Topic Models
30
![Page 31: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/31.jpg)
Evaluating a Topic Model
31
● Evaluating an unsupervised model is not trivial● Recall that a topic model has two main outputs:
TOPIC 1
Supervised
learning
classify
prediction
class
TOPIC 2
Learning
reinforce
reward
agent
q-learning
TOPIC 3
clustering
learning
model
similarity
centroid
Topic indicators Topic distribution in each document
RQ2: How can we ensure fairer comparisons between the models?
![Page 32: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/32.jpg)
Main aspects of the top words of the topics:
1) how coherent are the topics?
2) how diverse are the topics?
Evaluation of the top words
Evolution
Evolutionary
Human
Organisms
Life
Dna
Human
Genome
Dna
Genetic
Genes
Sequence
Disease
Pizza
Music
Diseases
Sport
Bacterial
32
RQ2: How can we ensure fairer comparisons between the models?
![Page 33: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/33.jpg)
Evaluation of the top words
Evolution
Evolutionary
Human
Organisms
Life
Dna
Human
Genome
Dna
Genetic
Genes
Sequence
Main aspects of the top words of the topics:
1) how coherent are the topics?
2) how diverse are the topics?
Disease
Pizza
Music
Diseases
Sport
Bacterial
JUNK TOPICGOOD TOPICS
Some words are not related to others!
33
RQ2: How can we ensure fairer comparisons between the models?
![Page 34: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/34.jpg)
Main aspects of the top words of the topics:
1) how coherent are the topics?
2) how diverse are the topics?
Evaluation of the top words
Evolution
Evolutionary
Human
Organisms
Life
Dna
Human
Genome
Dna
Genetic
Genes
Sequence
Disease
Pizza
Music
Diseases
Sport
Bacterial
NOT SIMILARSIMILAR TOPICS
We’d like that topics express separate ideas or semantic areas
34
RQ2: How can we ensure fairer comparisons between the models?
![Page 35: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/35.jpg)
Evaluation of the document-topic distribution
- intrinsic evaluation:
- perplexity: what is the likelihood that the words of the test document x have been generated by the trained topic model?
- extrinsic evaluation:
- evaluate the classification performance- any other external task
35
document-topic distribution
other features
classifier
prediction
RQ2: How can we ensure fairer comparisons between the models?
![Page 36: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/36.jpg)
36
● No benchmark datasets and non-standard pre-processing
● Stochasticity of the results
● Which topic model? Few releases in different programming languages, need to adapt data to each different implementation
A first solution: ToModAPI
Why evaluating topic models is hardRQ2: How can we ensure fairer comparisons between the models?
Lisena, P., Harrando, I., Kandakji, O. & Troncy, R (2020): TOMODAPI: A Topic Modeling API to Train, Use and Compare Topic Models, 2nd Workshop for NLP Open Source Software (NLP-OSS)
![Page 37: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/37.jpg)
Why evaluating topic models is hard● Hyperparameters setting:
○ Comparing the models by fixing their hyperparameters is not fair
○ Finding the best hyperparameter configuration is time-consuming
37
RQ2: How can we ensure fairer comparisons between the models?
![Page 38: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/38.jpg)
38
Optimizing and Comparing Topic Models is Simple!Pre-processing:
○ Most common pre-processing tools
○ Ready-to-use pre-processed datasets
Topic models:○ 4 classical topic models○ 4 neural topic models
Evaluation metrics○ Topic coherence○ Topic diversity○ Topic significance○ Document classification
Hyperparameter search○ Bayesian optimization for
optimizing the hyperparameters
Open-source python library & local web dashboard: https://github.com/mind-lab/octis We reached over 8k downloads and 170 github stars :)
Terragni, S., Fersini, E., Galuzzi, B. G., Tropeano, P., & Candelieri, A. (2021). OCTIS: Comparing and Optimizing Topic models is Simple!. EACL 2021 (System Demonstrations)
RQ2: How can we ensure fairer comparisons between the models?
![Page 39: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/39.jpg)
Bayesian OptimizationRQ2: How can we ensure fairer comparisons between the models?
● probabilistic surrogate model: approximates the objective function
● acquisition function: select the next configuration using the mean and the confidence of the surrogate model
![Page 40: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/40.jpg)
40
TOPIC MODEL
Hyperparameter configuration
Evaluation metric
Bayesian Optimization
algorithm
dog
cat
lion
animal
fruit
banananature
waterwood
lakeapple
kiwi
Bayesian Optimization for Topic Modeling
Repeat for N iterations
Score
It’s a black box!
RQ2: How can we ensure fairer comparisons between the models?
Repeat for M runs & average
![Page 41: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/41.jpg)
41
● We optimize the performance of relational topic models with respect to the classification metric F1-score
● We also evaluate other qualitative metrics to investigate different aspects of the RTMs
Optimizing the HyperparametersRQ2: How can we ensure fairer comparisons between the models?
![Page 42: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/42.jpg)
42
● We optimize the performance of relational topic models with respect to the classification metric F1-score
● We also evaluate other qualitative metrics to investigate different aspects of the RTMs
Optimizing the Hyperparameters
The configuration identified by BO leads to a better performance with respect to its initial configuration
RQ2: How can we ensure fairer comparisons between the models?
![Page 43: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/43.jpg)
43
● We optimize the performance of relational topic models with respect to the classification metric F1-score
● We also evaluate other qualitative metrics to investigate different aspects of the RTMs
Optimizing the Hyperparameters
Optimizing for classification purposes can be detrimental to different qualitative metrics
The configuration identified by BO leads to a better performance with respect to its initial configuration
RQ2: How can we ensure fairer comparisons between the models?
![Page 44: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/44.jpg)
44
What’s next?● If we optimize for a metric, what happens to the others?
● BO can be expensive:○ Which hyperparameters are important to optimize? ○ Can we reduce the space of the hyperparameters?○ Hyperparameter transfer
RQ2: How can we ensure fairer comparisons between the models?
![Page 45: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/45.jpg)
Thank you :)
45
![Page 46: Modeling Knowledge Incorporation into Topic Models and ...](https://reader034.fdocuments.in/reader034/viewer/2022042221/625a508277b39c6c5e0bf46f/html5/thumbnails/46.jpg)
References● Jonathan Chang, David M. Blei: Relational Topic Models for Document Networks. AISTATS 2009: 81-88● Rajarshi Das, Manzil Zaheer, Chris Dyer: Gaussian LDA for Topic Models with Word Embeddings. ACL (1) 2015:
795-804● Yishu Miao, Lei Yu, Phil Blunsom: Neural Variational Inference for Text Processing. ICML 2016: 1727-1736● David M. Mimno, Wei Li, Andrew McCallum: Mixtures of hierarchical topics with Pachinko allocation. ICML 2007:
633-640● Dat Quoc Nguyen, Richard Billingsley, Lan Du, Mark Johnson: Improving Topic Models with Latent Feature Word
Representations. TACL 3: 299-313 (2015)● Hanna Wallach: Topic Modeling: Beyond Bag-of-Words. ICML 2006● Yi Yang, Doug Downey, Jordan L. Boyd-Graber: Efficient Methods for Incorporating Knowledge into Topic Models.
EMNLP 2015: 308-317● Weiwei Yang, Jordan L. Boyd-Graber, Philip Resnik: A Discriminative Topic Model using Document Network Structure.
ACL (1) 2016● Mingyuan Zhou, Yulai Cong, Bo Chen: Augmentable Gamma Belief Networks. Journal of Machine Learning Research
17: 163:1-163:44 (2016)
46