A text-based model of foreign-affairs sentiment

Sean Gerrish and David BleiPrinceton University Computer Science

17 December 2011

These news articles tell a story.

A spatial model of foreign relations sentiment

This work develops a model of the sentimentbetween countries over time.

• It models dynamic relationships in aninterpretable way

• It infers sentiment from printed media

• Sentiment is defined by Mechanical Turkers

A spatial model of foreign relations sentiment

To do this, our plan is to:

• Collect a bunch of newspaper articles

• Define a latent variable model to captureinteresting structure in these articles

• Perform posterior inference to estimate thevalue of these random variables

Countries take latent positions x̄ct over time

Time

x

0

-2-1

1 2

x̄c ,t|x̄c ,t−1 ∼ N(x̄c ,t−1, σ2K)

The relationship between countries is observed in the news.

Time

x

0

-2-1

1 2

xc1,d ∼ N(x̄c1,t, σ2D)

xc2,d ∼ N(x̄c2,t, σ2D)

Sentiment sd := xc1,dTxc2,d

The relationship between countries is observed in the news.

Time

x

0

-2-1

1 2

xc1,d ∼ N(x̄c1,t, σ2D)

xc2,d ∼ N(x̄c2,t, σ2D)

Sentiment sd := xc1,dTxc2,d

The relationship between countries over time

xc,t-1 xc,t xc,t+1

xc,t-1 xc,t xc,t+1

C

Dt-1 t t+1

w

sss

ww

zerozero zero

2 2 2

DD

β

Latent position

Position duringinteraction

Sentiment

Observed text

Regularization

Labeling sentiment

1. We found all pairs of paragraphs from the New York Timeswhich discussed exactly two countries

2. A random sample of 3607 paragraphs from New York Timesarticles from 1988 to 2008 were labeled by AmazonMechanical Turk workers

3. Raters rated news articles on the scale −5,−3,−1, 1, 3, 5

Labeling sentiment: typical task

Sentiment and news articles: text regression

sd = wTd β + ε

• wd ∈ RV is the text of a news paragraph

• sd ∈ R is the sentiment between two countries

• β ∈ RV is the “weight” of each word

Sentiment and news articles: text parameter β

towns

looking

eligible

electricity

increasing

specialist

hero

reporterreported

elaborate

reports

military

criticism

divide

explained

replace

cocaine

unit

passport

successful

hurt

midst

hold

locked

pursue

plunged

revenues

owesorganized

vehemently

currency

absolute

complaining

travel

machine

effective

headquarters

rescued

easier

represents

burial

financial

series

laboratory

foundation

threatened

estimate

enormous

ministries

speedy

channels

service

engagement

euro

bitter

ranging

collapse

legally

peril

nations

projectwillingness

dozen

affairs

person

nonproliferation

mouth

letter

organization

drought

professor

fugitives

bomb

reactor

lessons

altogether

societies

release

respond

blew

fair

result

preserve

extend

extentdebt

accident

country

conclusions

demanded

planned

argue

union

life

bans

eastern

commerce

appeals

flooding

played

trusted

rebellion democracies

split

fairly

echoed

academic

ease

easy

prison

possibly

steps

people

captive

brigades

losing

shaken

dollars

citizens

facility

son

raises

reducing

occupyingauthorized

launchingoffer

delegation

inside

devices

adopt

proved

proven

accounting

dealer

negotiationsflood

republic

ambitious

intend

time

chain

veterans

cheap

choice

minute

tear

unify

leave

settle

team

sign

falling

drafted

funeralunderstanding

address

passengers

reclaim

positive

angry

scope

afford

apparent

refrain

believed

gunfire

allowed

monitoring

winter

divided

spot

natural

pulled

insurgency

course

decades

civilian

maintaining

shouted

didn

establishingquarter

square

peacekeepers

breakthrough

entering investigation

formula

superiors

siege

million

possibility

complicated

training

spoken

vote

city

meddling

favors

future

argument

noteroadside

handing

printed

drivelaws

walking

aggressive

dispute

guarded

freezing

prime

worried

bombs

surgery

ostensibly

representatives

sponsorship

jobs

jets

article

considers

stretch

airlines

thousand

tribal

policies

newspaper

situation

engagedtechnology

missiles

summer

overthrow

sums

traffic

vacuum

worldexit

restoring

contingent

power

leadership

stone

package

industryneighbor

mean

burning

image

technically

punished

completesurvived

politically

citizenship

wartime

graduate

tight

external

causedcondition

cable

joined

penalties

past

section

scientists

contrast

hours

civilians

concluding

experience

social

initiated

company

tested

foundations

resourceslearn

male

accept

states

sense

dress

imminent

dismissed

plant

plans

plane

director

fundamental

trade

signs

roots

rapidly

followers

entire

found

investigators

week

reduce

operation

research

plotting

risen

retained

anonymity

heads

jet

threatening

checkpoint

tough

self

internal

play

plan

singled

seize

sometimes

coverresearchers

golf

defended

occupation

writer

failed

banned

interviews

preparing

enemy

surrendering

kill

approaching

set administrator

migration

muted

available

targeted

incident

nonexistent legislature

prospects

allegiancemonthly

acceptance

convicted

champion

fire

representative

consulate

expanded

technical

pressed

owed

binding

renounced

read

virtually

levels

recent

arrests

eager

victimsdemands

couple

projects

imposedconsensus

communications

continue

tribes

names

shipments

spring

sight

transactions

religion

agreement

majorspending

crackdown

considerably

elsewhere

link

line

considerablecamp

nationalist

influence

nationalism

elements

energetic

prosecution

ago

land

fighter

age

summit

confederation

resultsexisting

longstanding

send

citing

reopened

continued

eve

fewer

try

severed

imply

assembly

expressed

led

gathered

poverty

receive

involved

defeat

opinion

makes

involves

standing

confidence

dealings

carrying

baby

retreated

weaker

process

purposes

weaken

delay

halted

ties

allow

produces

move

chosen

hijackersoutlined

accounts

doing

society

agriculture

fundamentally

evacuation

shut

mainlandassumed

enforcement

elite

colleagues

listeners

false

tonight

viewed

studying

visa

atmosphere

bringeconomistdecade

principal

bombings

hope

means

familiar exchanges

following

taxes

temporarily subcommittee

nuclear

onetime

restrictions

invited

drugconclude

picking

willing

waste

rioting

delaying site

terrorism

conflicting

ball

command

credibility

drawing

arrest

increased

government

five

replacement

indications

separate

demonstrators

accompanied

loans

stage

holdings

concession

revise

allianceletters

roads

airline

frustration

building

delivery

official

smooth

convince

distribution

recognize

slowing

details

chance

rule

write

rural

doubts

votes

invaders

worth

announced

established

told

protection

obtained

stressed

uniforms

secure

servicemen

total

word

worn

deprived

indicated

cited

campaigning

totaled

carriersdisappear

interviewordinary

lay

law

green

south

office

bombing

production

scene

break

rocky

reasonable

lifted

crimes

networkvessels

barrier

veto

disputes

created

renew

oppose

system

electronic

rejects

target

powers

forced

forces

involving

sun

respondedanniversary

yen

exchanged

secretary

observer

discussions

cooperation

accordterms

received

ill

receives

requests

tons

speak

mobilization

condemned

dozens

families

separatist

confirmed

incorrectly

independent

raidrail

hand

documents

blamed

kept

stringent

adding

belongs

night

security

born

boredenies

denied

confusion

participation

post

months

prisoners

deception

fight

lowest

peninsula

maximum

promises

evidence

promised

incursion

apartment

stake

reality

holding

test

brothers

welcome

reception

uprisings

diplomacy

global

battle

devoted

killingszone

compensate

trouble blast

fragile

gun

regret

cost

appeal

inauguration

appears

change

regardlessmarket

prove

territories

towers

organizations

carincidents

heart

rebuffed

recalled

occur

economy

southern

produce

commit

lifting

embassies

serving

indeedaiding

ratified

cold

halt

wealthy

discuss

drop

entirely

fired

happen

calling

space

increase

receiving

rebuilding

cars

marine

card

tries support

sprawling

flying

impossible

message

size

silent

atrocities

caught

national

friend

televisiontangible

contentious

remained

overnight

equipmenttourist

begin

deterioration

price

rationale

slavesprofessional

typically

discovered

excerpts

ground

title

developed

scientist

electedconcern

import

justified

blame

comes

nearby

rebuild

suspected

commander

media

inquiry

figures closestcomplained

charges

foot

speed

struck momentum

real

laundering

ruledheightened

pounds

passions

deficit

business

sixth

equivalent

strained

unease

negotiated

comparison

central

supervised greatly

testimonyrolling

involvement

intervention

prepare

start

low

lot

delayed

trying

moved

sales

moves

intercontinental

forcing

pool

surrounded

month

unrest

deadline

clerics

study

fence

streets

witnessed

learned

strong

ahead

inspired

soldierbase

family

aimed

trained

gunmen

excusebroke

coalition

nine historyboycottpushed

reject

tried

terrorist

invasion

banks

dream

help

soon

actually systems

foodsweeping

traveled

imprisoned

stoppeddie

referred

heavy

inspections

event

safety

favored

heroin

launch

liaison

scheduled

reflected

station

hundred

sentenced

alongside

approved

unreasonable

northern

assertion

flights

circle

ouster

engineers

close

probably

conditions

missing

abruptly

collaborators

sensitive

forgotten

headed

contras

vessel

described

stamp

dams

exert

maintenance

airstrikescollected

threaten

territory

empty

lives

pact

look

pace

endanger

voters

rulers

ready

conflict

anxious

obviously

ambassador

settlements

reviewedwounding

cemetery

march

outright

provoking

popular

parade

antiaircraft

fathers

urgent

economic

deliveredrun

stem

abroad

troopsrivalries

collapsed

properly

doesn politicianselectrical

recognition

reportedly

visitsregulations

approving

requiredorbit

depth

airliners

excluded

friendly

positions

enforce

commercial

radicals

kidnapped

collaborating

addressed

buildup

remains formally

started

missile

appointed

crosses

announcing

sanctions

crossed

meet

links

arrangement

farm

peace

gunman

mentioned

helicopters

guerrilla

hands

front

universitymagnitude

crossing

globe

constitute

variety

special

activist

cause

alleged

completely

resumed

hostile

route

times

counterpart

management

finalexactly

ultimatum

providing

border

pursued

instance

tactic

intentions

expiredgesture

businessman

emerging

sent

soldiers

based

earned

bases

employed

joint

procedures

tobacco

accused

written

tens

importance

strains

diplomats

addition

immense

controlled

releasing

guerrillas

smoking

surface

finance

capture

party

effect

frequently

indigenous

destruction

trials

restore

sources

fate

propelled

historic

immediately

loss

lost

payments

lose

refugees

home

broad

reaching

monetaryexpansion

celebrations

previously additional

inner

north

evacuate

blaming

demilitarized

signedlimit

urging

reservationsoffensive

deter

friends

persistent

portion

aided

protest

seriously

misunderstanding

toppling

vast

enhance

clothes

force

prisons

spokesmandeemed

met

governing

detonated

economics

credit

permit

campaign

guests

call calm

survive

tell

disruption

advancing

answer

coup

president

attempt

maintain

operate

operations

personal

weeks

overcome

meat

wage

lawyers

crucial

struggle

arrived

girls matter

emphasizing

seen

regular

don

medical

dog

principle

consumer

jurisdiction

treaties

attempts

judges

jailed

exports

conductstop

comply

fields

bad

troop

bantesting

decided

restricting

suggested

appeared

initiative

basis

patrolling

basic

deeperprosperous

aviation

tank

economies

nearmoratorium

seven

grown

potentially

advocate

declaring

opportunity

unconditionally

failing

punishment

materials

claims

investments

left

candidate

assassination

save

boasted

interior

destroy

www

deal

dead

devastationlies

distributing

initial

form

failure

stages

occupiers

grenades

sale

ship

marking

handed

fell

authorities

weekend

billion

assume

precise

mileassailants

bomber

hijacked

suicide

answered

finally

marks

advised

governments

round

fearing

targets

suspect

international

refused

earthquake

wait

box

shift

bow

convoys

sympathetic

wealth

visit

sharing

rigid

effort

fly

soul

arrive

agent

council

chemical hostage

sworn

map

designed

hailed

stability

argued

policy

main

boatskilled

possess

living

interference

emerged

spies

advance

language

frequent

suspended

carry

visas

speaking

venture

voices

occupied

fortified

consortium

channel

investigations

arguing

paid

assault

hardships

especially

massacres

shop

shot

sites

impunity

black

enthusiasm

contractsnearly

morning

declared

seal

indicted

checks

infrastructure

killing

nationality

according

tour

unusual

capable

borders

gripped

wake

sound

coca

coffee

pressing

traffickers pay

speech

running

roughly

substantive

money

resolution

grid

identifying

maintainedimages

colonial

critical

expressing

island

mixed

road

decisive

deepened

deadlock

powerless

personnel

narcotics

starting dollar

expressions

children

fall

neighborhood

returned

issues

movement

promising

speeches

significantly

published

buses

establish

controlling

particular

town

hour

intercepted

guards

remain

abandon

purchases

share

purchased

acts

advice

response

bleak

responsibilitysect

playing

inspectors

industrial

existence

late

smuggle

curfews

seeking

compound

easilyspy

deteriorated

house

energy

hard

oil reparations

beginning

computersconductedembraced

militant

assumption

statement

rates

doctors

contrary

treated

mountain

build

deepening

province

significant

joining

particularly

giant

resolve

common

scenes

surrendered

forthcoming

feared

expert

fans

aircraft

solely

annual

simply

throughout

expensive

create

secret

collaborator

meeting

gas

moment

understand

resisted

politician

bill

elections

cluster

replaced

consular century

suggests

corridor

development

yesterday

helped

spent

flagswithdraw

entryfundamentalists

shape

mob

alternative

cutdanger

source

deliberately

bid

matters

foreigners

bit

absolutely

scale

contacts

decision

remembered

hostages

forward

invite

immigration

hopes

subsidiarydirected

requesting

planes

autonomous

units

expedition

stripped

single

desert

vehicles

notion

brigade

warheads

rivers

givingaccessbody

exchange

jointlyhonor

richest

privately

resolutions

breakup

themselves

account

tunnel

praise

closing

effectively

reserves native

holds

regions

responsibilities

profile

collection

chief

bridges

labor

criminal

extradition

descendants

mandate

day

warned

identified

approachedallies

mortar

allied

reached

gear

acquiredoctorate

unlikely

apparently

mid

mix

eight

prisoner

enthusiastic

stranded request

normally

staff

controls

regularly

teams

shellingincluding

intensity

instant equal

passing

commentdisarmament

proposal

curtail

assert

finished

assessments

appearance

value

captors

arrangements

claimed

administration

injured

material

cubic

insure

executive

interim

delegations

resolved

tests

journalist

authority

proper students

panelactual

baseless

billions

buy

bus

repeated

dangerous

minutes

supreme

deaths

shooting

misstated

weaponry

stirredaccidents

limited

facilities

violate

rise

encounter

school

reciprocal

barracks

enjoy

leaders

consistent

direct

street

estimated

supplied

endorse

supplies

generals

changedhospital

assessment

casualties

shoot

joinestate

illicit

accusations

widespread

badly

stronghold

amidacknowledged

enter

executed

expectsexpectations

destroyed

prevented

tourism

notified

leading

victory

satellite

laid

newly

closings

days

relations

militancy

murder

servenegotiating

abandoning

tourists

bridge

ran

relatively

broadly

flow

reputation

pope

radio

earth

delays

announce

criticized

despite

reportcountries

twice

corrupt

capacityimprove

protect

games

expense

quickly

confident

expected

drugs

teachers

stopping

beside

complex

bands

action

insisted

shortly

forceful

detained

humanity

cultural

judge

advanced

apart

gift

officer

meters

election

escape

armed

espionage

core

attacks

efforts

civil

laborers

rely

fought

head

medium

differences

hear

removed

suspicions

setting

bullet

vigorously

backward

leads

environment

charge

tanksdenounced

level

posts

brother

water

baseballguilty

shootings

crisishandled

memory

ferry

figure

accepts

unarmed

critic

dropped

requirementsvital

information

tipped

succeededmainly

coordinated

happened

intended

tensions

factory

state

view

blood

suspend

requirement

season

copy

population

wide

require

officials

cards

outcomepre

armored

blunt

urge

stormed

consideredsenior

convoy

allegations

gulf

crime

cautioned

embassy

imposing

college

insuring

farmers

federal

review

commitments

weapons

outside

missions

region

provocation

period

insistpilgrims

peacefuldirection

minister

spirit

cash

tentatively

couples

postwarevents

status

driver

vowed

relief

inability

model

justify

guided

violent

cease

captured

announcement

disrupt

inflation

acceptedpicture

competition

table

provided

legal

discussed

stand

tariffs

communication

rising

determine

strict

regard

promote

strongly

spends

deserve

briefly

detaineessymbol

bullets included

callsfollow

settlement

decisions

acting

opportunities

program

activities

woman

worse

worst

list

prolonged

rate

design

supposedly

options

proceed

markets

minor

flag

glad

division

breaking

short

retaliate

mission

outraged

pretext

clashes

resort

soccer

framework

signing

instructions

friendship

expect

threat

feel

linkingstory

option

exploded

king

double

arsenal

outstanding

strengthen

chairman

donor

added

electric

measures

enduring

measured

stands

institutions

lying

industrialized

hit

militants

investigate

retaliated

declined

youth

unions

determined

various

numerous

corruption

recently

sold

attentionrapid

opposition

oversee

reasons

village

declinepolitical

due

strategy

flight

demand

plants

frozen

warrant

obstacle

widely

protestersflown

stating

questions

workers

amended

refuses

pipeline

firing

shed

question

fast

repeatedly

liberty

called

humanitarian

commissioner

warning

allowing

fertilizer

issued

resistance

disposal

dissidents languages

stable

include

fences

seized

deals

underscored

settlers

reunion

growers

universities

waiting

capital

chose

degree

pushing

larger

leaving

deputy

apply

timetable

journalistscooperate

women

tax

sitsix

struggled

attend

ethnic

lightsuperior

ships

permanent

covered

crash

flew

satellites

flee

fled

successor

articles

transitional

artists

related

doubled

disclose

averted

promptly

conflicts

proposing

reversed

institution

isolation

linked

afraid

agency

appealing

combat

alarmedneighboring

gain

electoral

agreed

fearoperational

local

ones

married

barrels

hardship

closer

favor

identification

closed

ability

agencies

job

grain

tactical

unclear

wall

withdrawn

improved

cross

unity

largest

diplomat

student

correspondent

fighting

rocket

heavily

obtain

republics

press

taxpayers

agents

employment

ravaged

lead

leader

murdered

extension

getting

spilled

sectarian

shipping

surge

carries

promisetransfer

intention

funds

challenges

record

demonstrate

firm

percent

book

conclusion

illegally

−0.5 0.0 0.5

Experiments

• Randomly select 3607 paragraphs discussing pairs of 245countries and territories.

• Label each of these paragraphs’ sentiment with two ratingsfrom Amazon Mechanical Turk.

• Hold out 42 random pairs (244 paragraphs) for testing.

• Fit sentiment model parameters β on training paragraphs.

• Infer the spatial sentiment model with these parameters on all257,472 paragraphs from 1988 to 2008.

Analysis with this model

To perform analysis with this model:

1. Fit the posterior (we used MAP)

2. Inspect countries’ means x̄c,·

3. Inspect the relationship between countries’ means x̄c1,t x̄c2,t

Results: selected countries’ latent positions

Ip1

Ip2

−0.4

−0.2

0.0

0.2

0.4

0.6

0.8

afghanistan

canada

china

france

indiairaq

israel

japannorth_koreapakistanrussia/ussr

south_korea

switzerland

united_states

●

−0.2 0.0 0.2 0.4 0.6 0.8 1.0Ip1

Ip2

−0.2

0.0

0.2

0.4

0.6

0.8

1.0

afghanistancanada

china

franceindia

iraq

israeljapan

north_korea

pakistan

russia/ussr

south_koreaswitzerland

united_states

●

0.0 0.5 1.0

1987 2007

Results: selected countries’ mutual sentiment with the U.S.

Date

Mut

ual s

entim

ent w

ith th

e U

nite

d S

tate

s

−0.6

−0.4

−0.2

0.0

0.2

0.4

−0.6

−0.4

−0.2

0.0

0.2

0.4

−0.6

−0.4

−0.2

0.0

0.2

0.4

afghanistan

iraq

russia/ussr

1990 1995 2000 2005

canada

israel

switzerland

1990 1995 2000 2005

china

japan

yugoslavia

1990 1995 2000 2005

Results: selected countries’ mutual sentiment with Spain

Date

Mut

ual s

entim

ent w

ith S

pain

−0.3

−0.2

−0.1

0.0

0.1

0.2

0.3

−0.3

−0.2

−0.1

0.0

0.1

0.2

0.3

−0.3

−0.2

−0.1

0.0

0.1

0.2

0.3

afghanistan

great_britain

portugal

1990199520002005

china

greece

united_states

1990199520002005

france

iraq

1990199520002005

germany

italy

1990199520002005

Evaluation

The model does better than text regression and individualMechanical Turk workers compared against one another.

Model Mean Squared Error Mean Absolute Error

Inter-rater agreement 1.77 (7.11) 1.037 (2.07)

Text regression 5.53 1.94

Reversion variance 0.1 2.36 1.09Reversion variance 1 2.32 1.07

Reversion variance 10 2.32 1.08Reversion variance 100 2.34 1.09

Reversion variance 1000 2.33 1.08

Current work and future directions

• Sentiment intercepts for each country

• Infer asymmetric relationships

• Application to other dyads

• Infer unsupervised relations• Sentiment is only one dimension• Similar to relational topic models [1]

Thank you

• Sean Gerrish (sgerrish@cs.princeton.edu)

• David Blei (blei@cs.princeton.edu)

Bibliography

Chang, J., D. M. Blei.

“Relational topic models for document networks.”

Proceedings of the 12th International Conference on ArtificialIntelligence and Statistics (AIStats) 2009, 5, 2009.