Other Topics You May Also Agree or Disagree: Modeling Inter-Topic Preferences using ... · 2018. 7....
Transcript of Other Topics You May Also Agree or Disagree: Modeling Inter-Topic Preferences using ... · 2018. 7....
Other Topics You May Also Agree or Disagree:
Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
Akira Sasaki, Kazuaki Hanawa, Naoaki Okazaki, Kentaro InuiTohoku University
1SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
Stance classification• Goal• Classify stances of texts in regard to a specific topic
• Applications• Public opinion survey from SNS data• Predicting voting actions
2SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
Input Output
Text: I fully agree with TPP Topic: TPP Stance: Agree
Difficulty of stance classification
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 3
People often talk about topicswithout explicitly mentioning the topic.
How can we classify stance from such a text?
Input Output
Text: It is better to promote domestic consumption Topic: TPP Stance: Disagree
Input Output
Text: It is better to promote free trade Topic: TPP Stance: Agree
Input Output
Text: It is better to promote domestic consumption Topic: TPP Stance: Disagree
Input Output
Text: It is better to promote free trade Topic: TPP Stance: Agree
who agree with TPP
free trade
revision of copyright law domestic consumption
distribution of pharmaceuticals
also agree with disagree with
knowledge
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 4
Use of inter-topic preferences forstance classification
inter-topic preference
A relatively simple example
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 5
Topic words and their surrounding wordsprovide strong clues.(Somasundaran&Wiebe,2010),(Mohammad+,2013)
Input Output
Text: I fully agree with TPP Topic: TPP Stance: Agree
※ Although datasets used in this work are in Japanese, we provide examples in English for readability.
Proposal: modeling inter-topic preferences via matrix factorization
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 6
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
1.0 0.3 -1.0 0.2
-1.0 0.2 0.7 -0.2
-0.4 -0.3 1.0 -1.0
0.2 0.5 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Topic
1To
pic 2
Topic
3To
pic 4
User 1
User 2
User 3
User 4
! !"
≈ × =
!" !Users’ stances for
each topics (user-topic matrix)
Compute users’ densefeature vector and topics’ dense
feature vector via matrix factorization
Complete missing values by
feature vectors
The aim of matrix factorization:
1. capture inter-topic preferences by dense feature vectors2. reveal users’ hidden stances by completion
The whole architecture
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 7
Corpus (tweets)
Tweets posted by userswho have used pro/con hashtags
A good news. [URL] #TPP反対
…
TPP ruins the future of our country
A is completely wrong
We should introduce A
to A
Pattern candidates in whichthe users describe topics
Linguistic pro/conpatterns
PatternExtraction
Sort candidatesand select
useful patterns
I support A / A is necessary /Welcome A / We should introduce A
…
I disagree A / A is completely wrong /A ruins the future of our country
…
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
1.0 0.3 -1.0 0.2
-1.0 0.2 0.7 -0.2
-0.4 -0.3 1.0 -1.0
0.2 0.5 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Topic
1To
pic 2
Topic
3To
pic 4
User 1
User 2
User 3
User 4
! !"
≈ × =
!" !
Mine topicpreferences
① Mining Linguistic Patterns of Agreement and Disagreement
② Extracting Instances ofStances
③ Matrix Factorization
The whole architecture
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 8
Corpus (tweets)
Tweets posted by userswho have used pro/con hashtags
A good news. [URL] #TPP反対
…
TPP ruins the future of our country
A is completely wrong
We should introduce A
to A
Pattern candidates in whichthe users describe topics
Linguistic pro/conpatterns
PatternExtraction
Sort candidatesand select
useful patterns
I support A / A is necessary /Welcome A / We should introduce A
…
I disagree A / A is completely wrong /A ruins the future of our country
…
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
1.0 0.3 -1.0 0.2
-1.0 0.2 0.7 -0.2
-0.4 -0.3 1.0 -1.0
0.2 0.5 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Topic
1To
pic 2
Topic
3To
pic 4
User 1
User 2
User 3
User 4
! !"
≈ × =
!" !
Mine topicpreferences
① Mining Linguistic Patterns of Agreement and Disagreement
② Extracting Instances ofStances
③ Matrix Factorization
Mining linguistic patternsof agreement/disagreement• Focus on pro/con hashtags such as “#X賛成” or
“#X反対” used by users who have strong stances to topics
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 9
Then extractcon linguistic patterns
from other tweets by this user
#X反対 means“disagree with X”
Corpus (Tweet) Tweets posted by userswho have used pro/con hashtags
A good news. [URL] #TPP反対…
TPP is completely wrongA is completely wrong
We should introduce A
to A
Candidates of linguistic patterns
PatternExtraction
user X
user Y
…
…
…
The whole architecture
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 10
Corpus (tweets)
Tweets posted by userswho have used pro/con hashtags
A good news. [URL] #TPP反対
…
TPP ruins the future of our country
A is completely wrong
We should introduce A
to A
Pattern candidates in whichthe users describe topics
Linguistic pro/conpatterns
PatternExtraction
Sort candidatesand select
useful patterns
I support A / A is necessary /Welcome A / We should introduce A
…
I disagree A / A is completely wrong /A ruins the future of our country
…
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
1.0 0.3 -1.0 0.2
-1.0 0.2 0.7 -0.2
-0.4 -0.3 1.0 -1.0
0.2 0.5 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Topic
1To
pic 2
Topic
3To
pic 4
User 1
User 2
User 3
User 4
! !"
≈ × =
!" !
Mine topicpreferences
① Mining Linguistic Patterns of Agreement and Disagreement
② Extracting Instances ofStances
③ Matrix Factorization
• Sort aforementioned pattern candidates by their frequency, and filter manually
Extracting instances ofstances
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 11
A is completely wrong
We should introduce A
to APattern candidates
Linguistic patterns
I support AA is necessaryWelcome A…
I disagree AA is completely wrongA is silly…
PRO
CON
Manual examination
Corpus (Tweet)
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
User 1
User 2
User 3
User 4
TPP…do
mestic
cons
umpti
on
…I support domestic consumption
…
TPP is sillyuser 1
• By using linguistic patterns, we create user-topic matrix
Extracting instances ofstances
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 12
!",$ =#((,),+1) − #((, ), −1)#((,),+1) + #((, ), −1)
Number of timesthe user u agree with the topic v
Number of timesthe user u disagree
with the topic v
Each element of the matrix is:
A is completely wrong
We should introduce A
to APattern candidates
Linguistic patterns
I support AA is necessaryWelcome A…
I disagree AA is completely wrongA is silly…
PRO
CON
Manual examination
The whole architecture
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 13
Corpus (tweets)
Tweets posted by userswho have used pro/con hashtags
A good news. [URL] #TPP反対
…
TPP ruins the future of our country
A is completely wrong
We should introduce A
to A
Pattern candidates in whichthe users describe topics
Linguistic pro/conpatterns
PatternExtraction
Sort candidatesand select
useful patterns
I support A / A is necessary /Welcome A / We should introduce A
…
I disagree A / A is completely wrong /A ruins the future of our country
…
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
1.0 0.3 -1.0 0.2
-1.0 0.2 0.7 -0.2
-0.4 -0.3 1.0 -1.0
0.2 0.5 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Topic
1To
pic 2
Topic
3To
pic 4
User 1
User 2
User 3
User 4
! !"
≈ × =
!" !
Mine topicpreferences
① Mining Linguistic Patterns of Agreement and Disagreement
② Extracting Instances ofStances
③ Matrix Factorization
Matrix factorization• By minimizing following objective function
• We can complete missing values as follows:
• Based on preliminary experiments, we set parameters as 𝑘 = 100,λ(= 0.1,λ*= 0.1 (refer to the paper for more info)
• We use libmf to solve the optimization problemhttps://github.com/cjlin1/libmf
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 14
r̂u,v ' pu|qv
minP,Q
X
(u,v)2R
(ru,v � pu|qv)
2 + �P ||pu||2 + �Q||qv||2
(u, v) 2 R:declared preference: u column vectors of P (user vector)
�P � 0,�Q � 0 : regularization coefficients: v column vectors of Q (topic vector)
pu 2 Rk
qv 2 Rk
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
1.0 0.3 -1.0 0.2
-1.0 0.2 0.7 -0.2
-0.4 -0.3 1.0 -1.0
0.2 0.5 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Topic
1To
pic 2
Topic
3To
pic 4
User 1
User 2
User 3
User 4
! !"
≈ × =
!" !
Evaluation• Ex1: Determining the dimension parameter 𝒌→ RMSE decreased as the number of dimensions ( ) increased
• Ex2: Predicting missing stances→ 80-94% accuracy on predicting missing stances
• Ex3: Correlation between human judgements→ Moderate correlation
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 15
𝑘
Dataset• Tweet corpus
• about 35 Billion tweets crawled from Feb. 2013 to Sep. 2016• about 7 Million users• retweets are removed
• Collected data• 100 pro patterns and 100 con patterns (manually filtered)• about 25 Million tuples (agreement/disagreement declaration)
corresponding to about 3 Million users and about 5,000 topics
• User-topic matrix• removed users and topics that appeared less than five times• about 10 Million tuples corresponding to about 270,000 users and about
2,300 topics• sparsity = 98.43%
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 16
• How accurately can user and topic vectors predict missing stances?
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 17
Ex2: Predicting missing stances
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
1.0 -1.0
-1.0
1.0 -1.0
User 1
User 2
User 3
User 4To
pic 1
Topic
2To
pic 3
Topic
4
hide 5% of elements ≈ ×
!" !
=matrix
factorization
1.0 0.3 -1.0 0.2
-1.0 0.2 0.9 -0.2
-0.2 -0.3 1.0 -1.0
0.2 0.8 0.1 -0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
calculate accuracyin regard to hidden elements
Ex2: Predicting missing stances• How accurately can user and topic vectors
predict missing stances?• majority baseline: predict missing values as
majority one of agree/disagree in regard to the topic
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 18
Matrix factorization
Majority baseline
Ourapproachpredictsmissingtopicpreferencesby80– 94%accuracy
Sincepreferencesofvocalusersdeviatedfromthoseoftheaverage
users,majoritybaselinedecreased
MatrixFactorization
MajorityBaseline
• Are predicted agreements/disagreements by matrix factorization are reasonable?
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 19
Ex2: Predicting missing stances
Ourapproachreasonablypredictsmissingvalues
user sample A
agree with
disagree with
regime changecapital relocation
Abe CabinetOkinawa US military basenuclear weaponsTPP
vote of non-confidence to Cabinetsame-sex partnership ordinancenational people’s government
may also agree with
steamrollering war billworsening dispatch lawsendai nuclear power plantwar bill
predict(matrix factorization) may also disagree with
Conclusion• Modeled inter-topic preferences by matrix factorization
• Our approach accurately predicts missing stancesby 80-94% accuracy
• Future work• Use methods of targeted sentiment analysis
instead of using linguistic patterns
• Extend our approach to other domains• product, company, music, etc
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 20
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 21
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 22
Appendix
SENTIMENT 1: Other Topics You May Also Agree or Disagree:
Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
23
Ex1: Determining the dimension parameter 𝒌• We observed that the reconstruction error decreased
as the iterative method of libmf progressed
• Based on this result,we concluded that
is sufficient forreconstructing theoriginal matrix
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 24
𝑘 = 100
𝑅
Ex2: Predicting missing stances
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 25
• majority baseline: predict missing values as majority one of agree/disagree in regard to the topic
1.0 -1.0
-1.0
1.0 -1.0
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
1.0
1.0
-1.0
agree
agreedisagr
ee
disagr
ee
Ex2: Predicting missing stances
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 26
• Since preferences of vocal users deviated from those of the average users, majority baseline decreased
Ex3: Correlation betweenhuman judgements• Created a dataset of pairwise inter-topic
preferences by using a crowdsourcing service
• Obtained 6-10 human judgements for everytopic pair, then computed the mean of the points
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 27
Q. People who agree with topic A also agree with topic B?
A1. those who agree/disagree with topic A may also agree/disagree with topic B
A2. those who agree/disagree with topic A may conversely disagree/agree with topic B
A3. otherwise(no associaction between topic A and topic B
+1
-1
±0
topic A topic B
Spearman’s rank correlation coefficient = 0.2210
cosine similarity
1.0 0.6
topic C topic D0.8 0.7
topic Y topic Z-1.0 -0.3
…
human judgement
Ex3: Correlation betweenhuman judgements• Compared human judgements and
similarity between vectors of pairs
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 28
450 topic pairs
a moderate correlationeven though
inter-topic preferences are highly subjective
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 29
Sub1: Example of predicted missing topic preference (qualitative)
Sub2: Similarity between topic vectors
• Do the topic vectors obtained by matrix factorization capture inter-topic preferences?
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
Topic:LiberalDemocraticParty(LDP)Top 7 of similar topics cosisne
similarityAbs’s LDP 0.3937resuming nuclear power plant operations
0.3765
bus rapid transit (BRT) 0.3410hate speech countermeasure law 0.3373Henoko relocation 0.3353C-130 0.3338Abe administration 0.3248
Synonymous topics successfully have similar
vectors
• Do the topic vectors obtained by matrix factorization capture inter-topic preferences?
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
Topic:LiberalDemocraticParty(LDP)Top 7 of similar topics cosisne
similarityAbe’s LDP 0.3937resuming nuclear power plant operations
0.3765
bus rapid transit (BRT) 0.3410hate speech countermeasure law 0.3373Henoko relocation 0.3353C-130 0.3338Abe administration 0.3248
Topics promoted by LDP also have similar vectors
Sub2: Similarity between topic vectors
Sub2: Similarity between topic vectors
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 32
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 33
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 34
Unused slides
SENTIMENT 1: Other Topics You May Also Agree or Disagree:
Modeling Inter-Topic Preferences using Tweets and Matrix Factorization
35
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 36
How can we use intrinsic knowledge in stance classification?
Input Output
Text: It is better to promote domestic consumption Topic: TPP Stance: Disagree
Input Output
Text: It is better to promote free trade Topic: TPP Stance: Agree
TPP
domestic consumption
free trade
revision of copyright law
distribution of pharmaceuticals
PROMOTE
PROMOTE SUPPRESS
SUPPRESS
Background knowledge
assume we know that“better to promote X”
means agreement to X
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 37
Previously, we manually annotated these PROMOTE/SUPPRESS knowledge and utilized in stance
classification (Sasaki+, WI2016)
How can we use intrinsic knowledge in stance classification?
Challenge formodeling inter-topic preference• Intuitively, we can see a topic as a vector
consisting of users’ declared stances
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 38
Those who agree with topic A also agree
with topic B
Those who agree with topic A
disagreewith topic B
Topic
A
User 1
cosine similarity = 1
1 : the user agrees with the topic : the user disagrees with the topic-1
1-11-1
1-11-1
User 2
User 3
User 4
Topic
B
Topic
A
User 1
cosine similarity = -1
1-11-1
1-1
1-1
User 2
User 3
User 4
Topic
B
Challenge formodeling inter-topic preference• However, a lot of people declare
agreement/disagreement to only a few topics
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 39
Empty cell meansundeclared stance
1.0 -1.0
-1.0 0.7
-0.4 1.0 -1.0
0.5
User 1
User 2
User 3
User 4
Topic
1To
pic 2
Topic
3To
pic 4
Other usage of inter-topic preference• Public opinion survey• analyze people’s political ideology at low cost
(cf. public opinion poll, census)• finer-grained than liberal/conservative
• Electoral campaigns• we can assume
“those who agree with topic A also vote for party B”
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 40
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 41
SENTIMENT 1: Other Topics You May Also Agree or Disagree:Modeling Inter-Topic Preferences using Tweets and Matrix Factorization 42