Can Deep Learning solve the Sentiment Analysis Problem

Can Deep Learning solve the

Sentiment Analysis Problem?

Mark CieliebakZurich University of Applied Sciences

Annual Meeting of SGAICO – Swiss Group for Artificial Intelligence and Cognitive Science18.11.2014

Outline

1. What is sentiment analysis?

2. How good are "classical" approaches?

3. Does deep learning solve the problem?

Mark Cieliebak 218.11.2014

About Me

18.11.2014 Mark Cieliebak 3

Mark CieliebakInstitute of Applied Information Technology (InIT)

ZHAW, Winterthur

Email: [email protected], Website: www.zhaw.ch/~ciel

Text

Analytics

Open

Data

Automated

Test

Generation

Research

Interests

Software

Engineering

mailto:[email protected]

What is Sentiment Analysis

"… WiFi Analytics is a free Android app that I find very

handy when it comes to troubleshooting and monitoring

a home network. "[1]


Sample Application:

Social Media Monitoring

Text Analytics Components:

• Find relevant documents

• Hot topic Analysis

• Sentiment analysis


[7]

Flavours of Sentiment Analysis

• Document Based

• Sentence Based

• Target-Specific

• Rating Prediction


Classic Approaches to Sentiment Analysis

Rule-Based Corpus-Based


Predicted

Label

[3][4]

Simple Sentiment Analysis

Idea: Count number of positive and negative words

"This camera is great[+1]." +1 (pos)

"I find it beautiful[+1] and good[+1]." +2 (pos)

"It looks terrible[-1]." -1 (neg)

"This car has a blue color." 0 (neu)

POSITIVE:

great

love

nice

...

NEUTRAL:

hello

see

I

…

NEGATIVE:

bad

hate

ugly

...

Use Sentiment-Dictionary:


Sample Rules


• Detect Booster Words: "The car is really very expensive[-1 -1 -2] ."

• New Category "Mixed": "This car has an appealing[+1] design and

comfortable[+1] seats, but it is expensive[-1]. "

• Negation: Invert only score of words occuring after the negation:

"The car is appealing[+3] and I do not[*-1] find it expensive[-2]"

• I do not find the car expensive and it is appealing.

Need to “understand” the sentence

Linguistic Analysis

-> RULE: Invert scores of words being in the same phrases as negation.

“I do not find the car expensive[+2]

and it is appealing[+3].” → +5 (pos)

Sentence

Sentence Conj. Sentence

Noun

Phrase

Verb Phrase

Verb Adverb Verb Noun

Phrase

Adj. Noun

Phrase

Verb Phrase

Det. Det Noun Det. Verb Participle

I do not find the car expensive and it is appealing


Rule-Based Sentiment Analysis

Most Important Issues:

- Requires good hand-crafted rules

- Hard to transfer to new tasks

or languages

- Does not work well for texts with bad grammer (Twitter)


[5]

Classic Approaches to Sentiment Analysis



Predicted

Label

[3][4]

Corpus-Based Sentiment Analysis


Predicted

Label

[4]


Annotated Corpus

Sentence Polarity

This analysis is good. Pos

It looks awful. Neg

This car has a blue color. Neu

This car has an appealing design, comfortable seats, but it is expensive. Mix

This car has a very appealing design, comfortable seats, but it is really expensive. Mix

This analysis is not good. Neg

This car has an appealing design, comfortable seats and it is not expensive. Mix

This movie was like a horror event. Neg

This car is appealing and is not expensive. Mix

... ...


Sample Features for Tweets

• Word ngrams: presence or absence of contiguous sequences of 1, 2, 3,

and 4 tokens; noncontiguous ngrams

• POS: the number of occurrences of each part-of-speech tag

• Sentiment Lexica: each word annotated with tonality score (-1..0..+1)

• Negation: the number of negated contexts

• Punctuation: the number of contiguous sequences of exclamation marks,

question marks, and both exclamation and question marks

• Emoticons: presence or absence, last token is a positive or negative

emoticon;

• Hashtags: the number of hashtags;

• Elongated words: the number of words with one character repeated (e.g.

‘soooo’)from: Mohammad et al., SemEval 2013



Most Important Issues:

- Requires large annotated corpora

- Depends on good features


[6]

How good are Sentiment Analysis Tools?


Quick Poll

• Short texts: 1-2 sentences from

Twitter, news, reviews etc.

• Three-class classification: positive,

negative, other

• Accuracy =# 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑜𝑐𝑠

# 𝑑𝑜𝑐𝑠

Mark Cieliebak 21

Accuracy Votes

<50%

50-60%

60-70%

70-80%

80-90%

>90%

"How good are state-of-the-art

sentiment analysis tools?"

18.11.2014

Tool Accuracy

0,2

0,3

0,4

0,5

0,6

0,7

0,8

Accu

rac

y

Best Tool per Corpus

Worst Tool per Corpus

22

61%

40%

Avg.

Mark Cieliebak18.11.2014

[14]

Tool Accuracy

0,2

0,3

0,4

0,5

0,6

0,7

0,8

Accu

rac

y

Best Tool per Corpus

Worst Tool per Corpus

Overall Best Tool

23

61%

40%

59%

Avg.

Mark Cieliebak18.11.2014

Take-Home Lesson

Accuracy of best commercial tool on

arbitrary short texts is 59%


Approaches to Sentiment Analysis



Pr

edi

cte

d

La

bel

[9]

Deep Learning

[8]

Deep Learning on Text

It's all about Word Vectors!


Word2Vec

• Huge set of text samples

(billions of words)

• Extract dictionary

• Word-Matrix: k-dimensional

vector for each word (k typically

50-500)

• Word vector initialized randomly

• Train word vectors to predict

next words, given a sequence of

words from sample text


Major contributions by Bengio et al. 2003, Collobert&Weston 2008, Socher et al. 2011, Mikolov et al. 2013

[9]

The Magic of Word Vectors


King - Man + Woman ≈ Queen

Live Demo on 100b words from Google News dataset: http://radimrehurek.com/2014/02/word2vec-tutorial/

[10]

http://radimrehurek.com/2014/02/word2vec-tutorial/

Relations Learned by Word2Vec


[11]

Using Word Vectors in NLP


Collobert et al., 2011:

• SENNA: Generic NLP System based on word vectors

• No manual feature engineering

• Solves many NLP-Tasks as good as benchmark systems

[12]

Deep Learning and Sentiment

Maas et al., 2011

• Enrich word vectors with sentiment context

• Capture semantic of words (unsupervised) and sentiment (supervised) in

parallel, using multiple learning tasks

wonderful

amazing

terrible

awful



Socher et al. 2013:

• Word Vectors do not help for Sentiment Analysis

• Recursive Neural Tensor Networks

• Representing sentence structures as trees while adding sentiment

annotations at same time

• Restricted to single, well-structured sentences

•


[13]


Quoc and Mikolov, 2014:

• "Paragraph Vectors"

• Add context (sentence, paragraph, document) to word vectors during

training

• Improves many existing approaches


[9]

Does Deep Learning solve the

Sentiment Analysis Problem?


Conclusion: Deep Learning for Sentiment

• Small improvements, not revolution

• Very recent research, not yet "end of the story"

• SemEval 2015 will be benchmark


Talk in Short!

1. Classic approaches are rule-based

or corpus-based

2. State-of-the-art tools classify 4

out of 10 docs wrong

3. Deep Learning does not need

hand-crafted features

4. Deep Learning improves

existing benchmarks


Thank You!

Mark CieliebakZurich University of Applied Sciences (ZHAW)

Winterthur, Switzerland

Email: [email protected], Website: www.zhaw.ch/~ciel


[15]

Can Deep Learning solve the Sentiment Analysis Problem

Data & Analytics

Transcript of Can Deep Learning solve the Sentiment Analysis Problem