Online Learning ofSemantic Relations

Nir Grinberg and William M. Pottenger, Ph.D.Rutgers University

03/30/2012 1

Introduction

What are semantic relations?“Barack H. Obama is the 44th President of the

United States”“Barack Obama takes the oath of office as President

of the United States”“Barack Obama, in full Barack Hussein Obama II

(born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009– ) and the first African…”

“X was born in Y” or “X is from Y”, etc.

03/30/2012 2

IntroductionWhy are we interested in Semantic

Relations?

Information Extraction, Information Retrieval and Question Answering

Building blocks for IDEAs

Interpretability and Generalization of Topic Models

03/30/2012 3

Related Work

Early works: DIPRE (Brin ’98), Snowball (Agichtein et al. 2000)

ACE and MUC-7 Datasets appearing => Supervised methods appear.Using features like extracted entities, POS,

parse tree… ?Kernel functions

Unsupervised: Dirt (Lin et al. ‘01) and USP (Poon et al ‘09)03/30/2012 4

Related WorkTopic Modeling:

Nubbi (Chang et al. 2009)Rel-LDA and Type-LDA

(Yao et al. 2011)

03/30/2012 5Rel-LDA Type-LDA

What is missing?

Interpretability?

Parallelizable but not O(N)

Interaction with other features?

Higher-Order learning?

03/30/2012 6

One more Related Work

Pachinko Allocation Model: (PAM) by Li et al. 2007

Capture arbitrary:Topic-Topic

correlationsTopic-Word

correlations

Better than LDA and CTM

03/30/2012 7PAM

Our ApproachSemRel: based on

Type-LDA and PAM.

Adds a layer of abstractionImprove interpretabilityAllow feature

interactions

Variational Inference:Stochastic natural

gradient

03/30/2012 8SemRel

PreprocessingTokenization, Lemmatization, POS tagging,

NERUsing StanfordNLP toolbox

Dependency Path ParsingUsing MaltParser

Filtering out long paths and syntactically irrelevant

Filtering out infrequent features and entities 03/30/2012 9

Example

“Gamma Knife, made by the Swedish medical technology firm Elekta, focuses low dosage gamma radiation ...”

03/30/2012 10

The Algorithm

03/30/2012 11

We derived similar online learning algorithms for RelLDA, Type-LDA and PAM

Results

03/30/2012 12

Results

SemRel outperforms Type-LDA:two tailed paired t-test across # topics:

t(4)= -6.01, p<0.002two tailed paired t-test across folds:

p<0.001

Preprocessing is more of bottleneck than the learning algorithm!

03/30/2012 13

Future Work

We’re currently investigating convergence

Complementary qualitative evaluation

Other datasets

Extensions with more features Word, Entities, Higher-Order features, etc.

04/02/2011 14

Conclusions

Yet another topic model, but:

Moved from Bag-Of-Words assumption without breaking the framework

Devised an online learning algorithm

Hopefully, improved on interpretability

04/02/2011 15

Q&A

Thank you!

04/02/2011 16