1

Entity/Event-LevelSentiment Detection and

Inference

Lingjia DengIntelligent Systems Program

University of Pittsburgh

Dr. Janyce Wiebe, Intelligent Systems Program, University of PittsburghDr. Rebecca Hwa, Intelligent Systems Program, University of Pittsburgh

Dr. Yuru Lin, Intelligent Systems Program, University of PittsburghDr. William Cohen, Machine Learning Department, Carnegie Mellon University

3

A World Of Opinions

REVIE

W EDITORIALS

BLOGSTWITTER

NEWS

6

Motivation• ... people protest the country’s same-sex

marriage ban ....

people

positive

negative

protest

same-sex marriage ban

WHAT ABOUT SAME-SEX MARRIAGE?

7

Explicit Opinions• The explicit opinions are revealed by opinion

expressions.

people

positive

negative

protestexplicit

same-sex marriage ban

8

Implicit Opinions• The implicit opinions are not revealed by

expressions, but are indicated in the text.• The system needs to infer implicit opinions.

people

positive

negative

protest

implicit

same-sex marriage ban

9

Goal:Explicit and Implicit Sentiments

people

positive

negative

protest

implicit

explicit

• explicit: negative sentiment• implicit: positive sentiment

same-sex marriage ban

10

Goal:Entity/Event-Level Sentiments• PositivePair(people, same-sex marriage)• NegativePair(people, same-sex marriage ban)

people

positive

negative

protest

implicit

explicit

same-sex marriage ban

11

Three Questions to Solve

• Is there any corpus annotated with both explicit and implicit sentiments?

• No. This proposal develops.

• Is there any inference rules defining how to infer implicit sentiments?

• Yes. (Wiebe and Deng, arXiv, 2014.)

• How do we incorporate the inference rules into computational models?

• This proposal investigates.

12

Completed and Proposed Work

• Expert Annotations on 70 documents (Deng et al., NAACL 2015)

• Non-expert Annotations on hundreds of documents

Corpus:MPQA 3.0

• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)

• A model validating rules (Deng and Wiebe, EACL 2014)

• A model inferring sentiments (Deng et al., COLING 2014)

Sentiment Inference on

+/-Effect Events & Entities

• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)

• Extracting Nested Source and Entity/Event Target• Blocking the rules

Sentiment Inference on

General Events & Entities

13

Background:Sentiment Corpora

Genre Source Target ImplicitOpinions

Review SentimentCorpus (Hu and Liu, 2004)

productreviews

writer the product, feature of the product

✗

Sentiment Tree Bank (Socher et al., 2013)

movie reviews

writer the movie✗

MPQA 2.0 (Wiebe at al., 2005; Wilson, 2008)

news, editorials, blogs, etc

writer, and any entity

an arbitrary span

✗

MPQA 3.0 news, editorials, blogs, etc

writer, and any entity

any entity/eventeTarget

✔

(head of noun phrase/verb phrase)

14

Background:MPQA Corpus• Direct subjective

o nested sourceo attitude

• attitude type• target

• Expressive subjective element (ESE)o nested sourceo polarity

• Objective speech evento nested sourceo target

15

MPQA 2.0: An Example

negative attitude

target

nested source:writer, Imam

When the Imam issued the fatwa against

Salman Rushdie for insulting the Prophet…

16

• Explicit sentimentso Extracting explicit opinion expressions, sources and

targets (Wiebe et al., 2005, Johansson and Moschitti, 2013a, Yang and Cardie, 2013, Moilanen and Pulman, 2007, Choi and Cardie, 2008, Moilanen et al., 2010).

• Implicit sentimentso Investigating features directly indicating implicit

sentiment (Zhang and Liu, 2011; Feng et al., 2013). No inference.

o A rules-based system requiring all oracle information. (Wiebe and Deng, arXiv 2014)

Background:Explicit and Implicit Sentiment

18

Completed and Proposed Work

• Expert Annotations on 70 documents (Deng et al., NAACL 2015)

• Non-expert Annotations on hundreds of documents

Corpus:MPQA 3.0

• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)

• A model validating rules (Deng and Wiebe, EACL 2014)

• A model inferring sentiments (Deng et al., COLING 2014)

Sentiment Inference on

+/-Effect Events & Entities

• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)

• Extracting Nested Source and Entity/Event Target• Blocking the rules

Sentiment Inference on

General Events & Entities

19

Completed and Proposed Work

• Expert Annotations on 70 documents (Deng et al., NAACL 2015)

• Non-expert Annotations on hundreds of documents

Corpus:MPQA 3.0

• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)

• A model validating rules (Deng and Wiebe, EACL 2014)

• A model inferring sentiments (Deng et al., COLING 2014)

Sentiment Inference on

+/-Effect Events & Entities

• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)

• Extracting Nested Source and Entity/Event Target• Blocking the rules

Sentiment Inference on

General Events & Entities

20

From MPQA 2.0 To MPQA 3.0

o “Imam” is negative toward “Rushdie’’.o “Imam” is negative toward “insulting’’.

o “Imam” is NOT negative toward “Prophet”.

negative attitude targ

et

nested source:writer, Imam

When the Imam issued the fatwa against

Salman Rushdie for insulting the Prophet…eTarge

t

21

Expert Annotations• Expert annotators include Dr. Janyce Wiebe and I.

• The expert annotators are asked to select which noun or verb is the eTarget of an attitude or an ESE.

• The expert annotators annotated 70 documents.

• The agreement score is 0.82 on average over four documents.

22

Non-Expert Annotations

• Previous work have tried to ask non expert annotators to annotate subjectivity and opinions (Akkaya et al., 2010, Socher et al., 2013).

• Reliable Annotationso Non-expert annotators with high credits.o Majority vote.o Weighted vote and reliable annotators (Welinder and Perona,

2010).

• Validating Annotation Schemeo 70 documents: Compare non-expert annotations with expert

annotations.o Then, collect non-expert annotations for the remaining corpus.

27

Part 1 Summary

• An entity/event-level sentiment corpus, MPQA 3.0

• Complete expert annotationso 70 documents (Deng and Wiebe, NAACL 2015).

• Propose non-expert annotationso Remaining hundreds of documents.o Crowdsourcing tasks.o Automatically acquiring reliable labels.

28

Completed and Proposed Work

• Expert Annotations on 70 documents (Deng et al., NAACL 2015)

• Non-expert Annotations on hundreds of documents

Corpus:MPQA 3.0

• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)

• A model validating rules (Deng and Wiebe, EACL 2014)

• A model inferring sentiments (Deng et al., COLING 2014)

Sentiment Inference on

+/-Effect Events & Entities

• Joint Models• A pilot study (Deng et al., EMNLP 2015)

• Extracting Nested Source and Entity/Event Target• Blocking the rules

Sentiment Inference on

General Events & Entities

29

+/-Effect Event Definition

• A +effect event has benefiting effect on the theme.o help, increase, etc

• A –effect event has harmful effect on the theme.o harm, decrease, etc

• A triple• <agent, event, theme>

He rejects the paper.

-effect event: rejectAgent: Hetheme: paper

<He, reject, paper>

31

• +Effect(x)o x is a +effect event

• -Effect(x)o x is a –effect event

• Agent(x,a)o a is the agent of +/-effect event x

• Theme(x, h)o h is the theme of +/-effect event x

+/-Effect Event Representation

32

+/-Effect Event Corpus• +/-Effect event information is annotated.

o The +/-effect events.o The agents.o The themes.

• The writer’s sentiments toward the agents and themes are annotated.o positive, negative, neutral

• 134 political editorials.

33

Sentiment Inference Rules

• people protest the country’s same-sex marriage ban.

• explicit sentimento NegativePair(people, ban)

• +/-effect event informationo -Effect(ban)o Theme(ban, same-sex marriage)

NegativePair(people, ban) ^ -Effect(ban) ^ Theme(ban, same-sex marriage)

PositivePair(people, same-sex marriage)

34

• +Effect Rule:• If two entities participate in a +effect event,• the writer’s sentiments toward the entities are

the same.

• -Effect Rule:• If two entities participate in a –effect event,• the writer’s sentiments toward the entities are

the opposite.

Sentiment Inference Rules

Can rules infer sentiments

correctly?(Deng and Wiebe, EACL

2014)

35

Building the graph from

annotations

A

E

D C

B

agent/

theme

• node score: two sentiment scores

(Deng and Wiebe, EACL 2014)

36

Building the graph from

annotations

• edge score: four sentiment constraints scores

• • the score that the sentiment toward D is positive• AND the sentiment toward E is positive

+/-effect

A

E

D C

B

37

Building the graph from

annotations

• edge score: inference rules• if +effect:• if –effect:

+/-effect

A

E

D C

B

38

Loopy Belief Propagation

• Input: the gold standard sentiment of one node• Model: Loopy Belief Propagation• Output: the propagated sentiments of other nodes

+/-effect

A

E

D C

B

agent/

theme

39

Propagating sentiments

• For node E, • can it be propagated with correct sentiment

labels?

+/-effect

A

E

D C

B

agent/

theme

40

Propagating sentiments A E

• Node A is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or

not.

+/-effect

A

E

D C

B

agent/

theme

41

Propagating sentiments B E

• Node B is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or

not.

+/-effect

A

E

D C

B

agent/

theme

42

Propagating sentiments C E

• Node C is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or

not.

+/-effect

A

E

D C

B

agent/

theme

43

Propagating sentiments D E

• Node D is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or

not.

+/-effect

A

E

D C

B

agent/

theme

44

Evaluating E being propagated

correctly

• Node E is propagated with sentiment 4 times.• correctness =

(# node E being propagated correctly)/ 4

• average correctness = 88.74%

+/-effect

A

E

D C

B

agent/

theme

45

Conclusion

• Defining the graph-based model with sentiment inference rules.

• Propagating sentiments correctly in 88.74% cases.

• To validate the inference rules only,• The graph-based propagation model is built from

manual annotations.Can we automatically infer

sentiments?(Deng et al., COLING 2014)

46

Local Detectors

Agent1 Agent2 reversed +effect -effect Theme1 Theme2

pos: 0.7neg: 0.5

pos: 0.5neg: 0.6

pos: 0.5neg: 0.5

pos: 0.7neg: 0.5

reverser: 0.9

+effect: 0.8 -effect: 0.2

(Q1) is it +effect or -effect?

(Q2) is the effect reversed?

(Q3) which spans are agents and themes?

(Q4) what are the writer’s sentiments?

• Given a +/-effect event span in a document,• Run state-of-the-art systems assigning local scores.

(Deng et al., COLING 2014)

47

Local Detectors

Agent1 Agent2 reversed +effect -effect Theme1 Theme2

pos: 0.7neg: 0.5

pos: 0.5neg: 0.6

pos: 0.5neg: 0.5

pos: 0.7neg: 0.5

reverser: 0.9

+effect: 0.8 -effect: 0.2

(Q1) word sense disambiguation

(Q2) negation detected

(Q3) semantic role labeling (Q4) sentiment

analysis

(Deng et al., COLING 2014)

Ambiguity

48

Global Optimization

• The global model selects an optimal set of candidates:o one candidate from the four agent sentiment candidates,

• Agent1-pos, Agent1-neg, Agent2-pos, Agent2-nego one/no candidate from the reversed candidate,o one candidate from the +/-effect candidates,o one candidate from the four theme sentiment candidates.

pos: 0.7neg: 0.5

pos: 0.5neg: 0.6

pos: 0.5neg: 0.5

pos: 0.7neg: 0.5

reverser: 0.9

+effect: 0.8 -effect: 0.2

Agent1 Agent2 reversed +effect -effect Theme1 Theme2

49

Objective Function

• The framework assigns values (0 or 1) to uo maximizing the scores given by the local

detectors,

• and assigns values (0 or 1) to ξ, δo minimizing the cases where +/-effect event

sentiment rules are violated.

• Integer Linear Programming (ILP) is used.

p: candidate local score

u: binary indicator of choosing candidate

ξ, δ: slack variables of triple <i,k,j>representing this triple is an exception to +effect –effect rule (exception: 1)

50

• In a +effect event, sentiments are the same

+Effect Rule Constraints

1 001 10 0 +effect:

1-effect: 0

exception: 1not exception: 0

AND

51

-Effect Rule Constraints

• In a –effect event, sentiments are opposite.

52

Performances

Accur

acy

of Q

1

Accur

acy

of Q

2

Accur

acy

of Q

3

F-mea

sure

of Q

40

0.4

0.8

Light Color: LocalDark Color: ILP

(Q1) is it +effect or -effect?

(Q2) is the effect reversed?

(Q3) which spans are agents and themes?

(Q4) what are the writer’s sentiments?

Precision Q4

Recall of Q4

56

Part 2 Summary• Inferring sentiments toward entities participating

in the +/-effect events.

• Developed an annotated corpus (Deng et al., ACL 2013).

• Developed a graph-based propagation model showing the inference ability of rules (Deng and Wiebe, EACL 2014).

• Developed an Integer Linear Programming model jointly resolving various ambiguities w.r.t. +/-effect events and sentiments (Deng at al., COLING 2014).

57

Completed and Proposed Work

• Expert Annotations on 70 documents (Deng et al., NAACL 2015)

• Non-expert Annotations on hundreds of documents

Corpus:MPQA 3.0

• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)

• A model validating rules (Deng and Wiebe, EACL 2014)

• A model inferring sentiments (Deng et al., COLING 2014)

Sentiment Inference on

+/-Effect Events & Entities

• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)

• Extracting Nested Source and Entity/Event Target• Blocking the rules

Sentiment Inference on

General Events & Entities

59

Joint Models• In (Deng et al., COLING 2014), we use Integer

Linear Programming framework.

• Local systems are run.• Joint models take local scores as input, and

sentiment inference rules as constraints.

• In ILP, the rules are written in equations and in equations.

60

Joint Models: General Inference Rules• Great! Dr. Thompson likes the project. …

• Explicit sentiment: o Positive(Great)o Source(Great, speaker)o ETarget(Great, likes)o PositivePair(speaker, likes)

• Explicit sentiment:o Positive(likes)o Source(likes, Dr. Thompson)o ETarget(likes, project)o PositivePair(Dr. Thompson, project)

61

Joint Models: General Inference Rules• Great! Dr. Thompson likes the project. …

• Explicit sentiment: o Positive(Great)o Source(Great, speaker)o ETarget(Great, likes)o PositivePair(speaker, likes)

• Explicit sentiment:o Positive(likes)o Source(likes, Dr. Thompson)o ETarget(likes, project)o PositivePair(Dr. Thompson, project)

PositivePair(speaker, likes) ^ Positive(likes) ^ ETarget(likes, project) PositivePair(spkear, project)

sentimenttoward

sentiment

62

Joint Models• More complex rules, in first order logics.

• Markov Logic Network (Richardson and Domingos, 2006).o a set of atoms to be groundedo a set of weighted if-then rules

o rule: friend(a,b) ^ voteFor(a,c) voteFor(b,c)o atom: friend(a,b), voteFor(a,c)o ground atom: friend (Mary, Tom)

• MLN selects a set of ground atoms that maximize the number of satisfied rules.

63

• PositivePair(s,t)• NegativePair(s,t)

• Positive(y) Negative(y)• Source(y,s) Etarget(y,t)

• +Effect(x) -Effect(y)• Agent(x,a) Theme(x,a)

Joint Model:Pilot Study Atoms

Predicted by joint models

Assigned scores bylocal systems

66

• 3 joint models:

• Joint-1 (without any inference)• Joint-2 (added general sentiment inference rules)• Joint-3 (added +/-effect event information and the

rules)

Joint Model:Pilot Study Experiments

69

• Task: extracting PostivePair(s,t) and NegativePair(s,t).

• Baselines: for an opinion extracted by state-of-the-art systemso source s: the head of extracted source span

o eTarget t:• ALL NP/VP:

o all the nouns and verbs are eTargets• Opinion/Target Span Heads (state-of-the-art):

o head of extracted target span; o head of opinion span

o PositivePair or NegativePair: the extracted polarity

Joint Model:Pilot Study Performances

70

• Task: extracting PostivePairs and NegativePairs.

Joint Model:Pilot Study Performances

PosP

air A

ccur

acy

Neg

Pair

Accur

acy

PosP

air F

-mea

sure

Neg

Pair

F-mea

sure

0

0.1

0.2

0.3

0.4

0.5

ALL NP/VPOpinion/Target Span HeadsPSL1PSL2PSL3

True Negatives

71

• We cannot directly use the state-of-the-art sentiment analysis system outputs (spans) for entity/event-level sentiment analysis task.

• The inference rules can find more entity/event-level sentiments.

• The most basic joint models in our pilot study can improve in accuracies.

Joint Model:Pilot Study Conclusions

72

• Various variations of Markov Logic Network.• Integer Linear Programming.

• Each local component being improved.o Nested Sourceso ETargeto Blocking the rules

Joint Model:Proposed Extensions

74

Nested Sources nested source:writer, Imam,

Rushdie

negative attitude

When the Imam issued the fatwa against Salman Rushdie for insulting the Prophet…

• How do we know Rushdie is negative toward Prophet?• Because Imam claims so, by issuing the fatwa against

him.

• How do we know Imam has issued the fatwa?• Because the writer tells us so.

75

Nested Sourcesnested source:writer, Imam,

Rushdie

negative attitude

When the Imam issued the fatwa against Salman Rushdie for insulting the Prophet…

• Nested source reveals the embedded private states in MPQA.

• Attributing quotations (Pareti et al., 2013, de La Clergerie et al., 2011, Almeida et al., 2014).

• The overlapped opinions and opinion targets.

76

• Extracting named entities and events as potential eTargets (Pan et al., 2015, Finkel et al., 2005, Nadeau and Sekine, 2007, Li et al., 2013, Chen et al., 2009, Chen and Ji, 2009).

• Entity co-reference resolution (Haghighi and Klein, 2009;

Haghighi and Klein, 2010; Song et al., 2012).• Event co-reference resolution (Li et al., 2013, Chen et

al., 2009, Chen and Ji, 2009).

• Integrating external world knowledgeo Entity Linking to Wikipedia (Ji and Grishman, 2011; Milne and

Witten, 2008; Dai et al., 2011; Rao et al., 2013)

ETarget (Entity/Event Target)

79

• That man killed the lovely squirrel on purpose.o Positive toward squirrelo killing is a –effect evento Negative toward that man

• That man accidentally hurt the lovely squirrel.o Positive toward the squirrelo hurting is a –effect evento Negative toward that man

Blocking Inference Rules

✗

✓

80

• That man killed the lovely squirrel on purpose.o Positive toward squirrelo killing is a –effect evento Negative toward that man

• That man accidentally hurt the lovely squirrel.o Positive toward the squirrelo hurting is a –effect evento Negative toward that man

Blocking Inference Rules

✗

✓

81

Collect Blocking

Cases

Compare and Find differenc

es

Learn to Recogniz

e

Blocking Inference Rules

84

Part 3 Summary• Joint models:

o A pilot study (Deng and Wiebe, EMNLP 2015)o Improved joint models integrating improved

components.

• Explicit Opinions:o Opinion expressions and polarities (state-of-the-art)o Opinion nested sourceso Opinion eTargets (entity/event-level targets)

• Implicit Opinions:o General inference rules (Wiebe and Deng, arxIV 2014)o When the rules are blocked

85

Completed and Proposed Work

• Expert Annotations on 70 documents (Deng et al., NAACL 2015)

• Non-expert Annotations on hundreds of documents

Corpus:MPQA 3.0

• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)

• A model validating rules (Deng and Wiebe, EACL 2014)

• A model inferring sentiments (Deng et al., COLING 2014)

Sentiment Inference on

+/-Effect Events & Entities

• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)

• Extracting Nested Source and Entity/Event Target• Blocking the rules

Sentiment Inference on

General Events & Entities

86

Research Statement• Defining a new sentiment analysis task

(entity/event-level sentiment analysis task), • this work develops annotated corpora as

resources of the task • and investigates joint prediction models • integrating explicit sentiments, entity or event

information and inference rules together • to automatically recognize both explicit and

implicit sentiments expressed among entities and events in the text.

sentiment analysis

information extraction

87

Timeline Date Content Deliverable results

Sep - Nov Collecting Non-Expert Annotation Completed MPQA 3.0 corpus

Nov - Jan Extracting Nested Source and ETarget

NAACL 2016: A System Extracting Nested Source & Entity/Event-Level ETarget

Jan - Mar Analyzing Blocked RulesACL/COLING/EMNLP 2016:Improved Graph-Based Model Performances

Mar - May

An Improved Joint Model Integrating improved Components

Journals submitted

May - Aug Thesis Writing Thesis Ready for Defense

Aug - Dec Thesis Revising Completed Thesis

88