Studying the History of Ideas Using Topic Models
D. Hall, D. Jurafsky, & C. D. ManningStandord University
EMNLP 2008
Agenda
• Introduction• Methodology• Historical trends in computation linguistics• Is computational linguistics becoming more
applied?• Differences and similarities among COLING,
ACL, and EMNLP• Conclusion
Goal
• Identify and study the exploration of ideas in a scientific field over time.– Periods of gradual development.– Major ruptures.– Waxing and waning of both topic areas and
connections with applied topics and nearby fields?
Citation graphs
Change of ideas
• Rather than deal with papers or authors, this paper is focused on the change of ideas in a field over time.
• Apply Kuhn’s insight that vocabulary and vocabulary shift is a crucial indicator of ideas and shifts in ideas.
• Operationalize on the unsupervised topic model Latent Dirichlet Allocation, LDA (Blei et al. 2003)
Analyzing the trends in CL
• 12,500 documents of the ACL Anthology have been analyzed.
• The CL field gotten more theoretical or more applied?
• What topics have declined over the years, and which ones have remained constant?
• How have fields like Dialogue or MT changed over the years?
• Are there differences among the conferences?
ACL Anthology
• A public repository of all papers in the major journals, conferences, and workshops.– Computational Linguistics.– ACL, COLING, EMNLP, and so on.
• Comprises over 14,000 documents.• From 1965 to 2008.• Indexed by conference and year.• Used as the basis of citation analysis work.
(Joseph & Radev, 2007)
Data in the ACL Anthology
Latent Dirichlet Allocation (LDA)
• A generative latent variable model that treats documents as bags of words generated by one or more topics.– Each document is represented as a multinomial
distribution over topics.– Each topic is in turn characterized by a
multinomial distribution over words.• Parameter estimation using collapsed Gibbs
sampling (Griffiths & Steyvers, 2004)
Topic Modeling
• The empirical probability that an arbitrary paper d written in year y was about topic z:
• I is the indicator function, td is the year document d was written, and p(d|y) = 1/C.
Topic selection
• Apply LDA to induces 100 topics, and took 36 that are relevant.
• Hand selected seed words for 10 more topics to improve coverage of the field.
• These 46 topics were used as priors to a new 100-topic run.
• Finally, 43 topics are selected.
Topics becoming more important
Trend of probabilistic models
• The probabilistic model topic increases around 1988, which seems to have been an important year for this topic.
• What do the papers from 1988 tell us about how probabilistic models entered the field?
Analysys
• 9 of 10 the papers appeared in conference proceedings rather than journal.– New ideas appear in conferences.
• 5 of conference papers appeared in COLING compared to only 1 in ACL.– COLING is more receptive than ACL to new ideas.
• 6 of 10 papers either focus on speech or were written by authors who had published on speech recognition topics.– Speech recognition is an EE field which made early use of
probabilistic and statistical methodologies.
Topics that have declined
Including lexical semantics, conceptual semantics/story understanding, computational semantics, WordNet, WSD, semantic role labeling, RTE and paraphrase, MUC information extraction, and events/temporal.
Paradigm shift in machine translation
Paradigm shift in dialogue
Peaked topics
Is CL becoming more applied?
Including machine translation, spelling correction, dialogue systems, information retrieval, call routing, speech recognition, and biomedical applications.
Six applied topics over time
The years 1989-1994 correspond exactly to the DARPA Speech and Natural Language Workshop, held at different location.
Differences and similarities among COLING, ACL, and EMNLP
• Whether the topics of these conferences are converging or not.
• Are the probabilistic and machine learning trends that are dominant in ACL becoming dominant in COLING as well?
• Is EMNLP adopting some of the topics that are popular at COLING?
Entropy of the 3 conferences over time
Divergence between the 3 conferences
The Jensen-Shannon (JS) divergence between each pair of conference are plotted.
Conclusion
• Proposed method discovers a number of trends in the computational linguistics.
• Show a convergence over time in topic coverage of ACL, COLING, and EMNLP as well an expansion of topic diversity.
• The growth and convergence of the 3 conferences, perhaps influenced by the need to increase recall seems to be leading toward a tripartite realization of a single new “latent” conference.
Top Related