Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use...

27
Using Sentiment Analysis to Fill in the Gaps in User Surveys That’s a really long title

Transcript of Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use...

Page 1: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Using Sentiment Analysis to Fill in the Gaps in User

SurveysThat’s a really long title

Page 2: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

YOW! Data 2017My lack of expertise

Confusing setup

Flaunted my poor understanding of underlying tech

Audience understood less than when they went in

Dance number

Page 3: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

TodayLower expectations

Confusing setup

Poor explanation of the technology

Ambiguous results

Silent weeping

Page 4: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Confusing Setup

Page 5: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

SuperannuationWe all love superannuation

Can we measure just how much we love it?

Page 6: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Two SourcesPhone calls with consultants Web chat logs

Page 7: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Phone callsSpeech-to-text

Poor quality, single-channel

Comical results

Much funnier presentation

Page 8: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Transcript

“hope hi there John my name is Spider-Man advise of the strange episode advice can you please get some of that your separation under some fantastic now we all in 20 minutes according to that and so that I can include details Yu-Gi-Oh counterpart of a conversation”

Page 9: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Web chat logsAgent: Hello, I am totally a human being. My name is Agent.

Customer: Yes, indeed. May I just say how much I love Super?

Agent: Me too, after all, it is called “super” for a reason.

Customer: How can I get more Super into me?

Agent: Would you be interested in our new “Super Super”?

Customer: Here is all my money, it is yours.

Page 10: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Web chat logs13 months JSON data

100k conversations

1 million lines of dialogue

Metadata

Page 11: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Web chat logsMeaningful Connection Score

Agent-supplied rating

-100 to +100

Almost every chat has one

Exit Survey

User-supplied rating

10% get asked

10% of those will do it

~1000 conversations have one

0 to 10

Page 12: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Web chat logsNobody wants to fill in a survey

Can we use Sentiment Analysis as an indicator?

Page 13: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Poor Technical Explanation

Page 14: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Sentiment AnalysisClassification problem

Page 15: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

First ProblemHumans only agree on the sentiment of text about 80% of the time

Page 16: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

ComplicatedI like cheese.

I do not like cheese.

I like cheese, but I’d rather eat biscuits.

I used to like cheese.

Page 17: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

SarcasmWell, that’s great.

Vs.

Well, that’s great.

Page 18: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

ModelsLexical - word frequencies

Supervised - Naive Bayes, SVM

Unsupervised - Latent Dirichlet Allocation, word2vec

Neural Networks

Page 19: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

LevelsDocument

Paragraph

Sentence

Entity

Page 20: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Google Natural Language APIParser + NN

Sentence and entity level

Sentiment score and magnitude

Cheap compared to writing your own

Other services are available

Build your own (can use SyntaxNet if you want)

Page 21: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Ambiguous Results

Page 22: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

MCS vs Sentiment

Page 23: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Exit Scores vs Sentiment

Page 24: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Entity Sentiment

Page 25: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Silent Weeping

Page 26: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Please stop nowSentiment analysis is a reasonable indicator of customer satisfaction

Sentiment analysis is slightly more accurate than astrology

Probably best to train your own model for entity sentiment

Refunds available from the organisers

Page 27: Using Sentiment Analysis to Fill in the Gaps in User Surveys€¦ · Build your own (can use SyntaxNet if you want) Ambiguous Results. MCS vs Sentiment. Exit Scores vs Sentiment.

Questions?Please don’t.

Nobody wants this to go on any longer than it has to.

@nomiddlename


apr 2014maps sentiment 0 o O O o o 00 0 0 00 sentiment opage maps sentiment opage maps maps sentiment sentiment ©page maps sentiment opage maps maps i sent sentiment ...

apr 2014maps sentiment 0 o O O o o 00 0 0 00 sentiment opage maps sentiment opage maps maps sentiment sentiment ©page maps sentiment opage maps maps i sent sentiment ...

Compositional Sentiment AnalysisI Pleasure vs. displeasure ... use the syntax to do ‘compositional’ sentiment analysis: S H HH HH NP This product VP H HH Adv ... with gesture and

Compositional Sentiment AnalysisI Pleasure vs. displeasure ... use the syntax to do ‘compositional’ sentiment analysis: S H HH HH NP This product VP H HH Adv ... with gesture and

WELCOME - PSG€¦ · Trade war Having a big impact on prevailing market sentiment Source.Policyuncertainty.com. Page 52 ... Russian Ruble vs US$ Indian Rupee vs US$ US Dollar Against

WELCOME - PSG€¦ · Trade war Having a big impact on prevailing market sentiment Source.Policyuncertainty.com. Page 52 ... Russian Ruble vs US$ Indian Rupee vs US$ US Dollar Against

Practical Sentiment Analysis Tutorial - BIUu.cs.biu.ac.il/~89-680/sentiment.pdf · Practical Sentiment Analysis Tutorial Jason Baldridge @jasonbaldridge Sentiment Analysis Symposium

Practical Sentiment Analysis Tutorial - BIUu.cs.biu.ac.il/~89-680/sentiment.pdf · Practical Sentiment Analysis Tutorial Jason Baldridge @jasonbaldridge Sentiment Analysis Symposium

Data vs Sentiment: The Great Divide

Data vs Sentiment: The Great Divide

Sentiment Analysis of Greek Tweets and Hashtags using ...hashtag.nonrelevant.net/Sentiment Analysis of Greek... · sentiment rating for the Greek tweets, for a variety of sentiment

Sentiment Analysis of Greek Tweets and Hashtags using ...hashtag.nonrelevant.net/Sentiment Analysis of Greek... · sentiment rating for the Greek tweets, for a variety of sentiment

Riding the tide of sentiment change: sentiment analysis ... · Riding the tide of sentiment change: sentiment analysis with evolving online reviews ... market intelligence, and customer

Riding the tide of sentiment change: sentiment analysis ... · Riding the tide of sentiment change: sentiment analysis with evolving online reviews ... market intelligence, and customer

Twitter Sentiment & Investing - modeling stock price movements with twitter sentiment.

Twitter Sentiment & Investing - modeling stock price movements with twitter sentiment.

Unpaired Sentiment-to-Sentiment Translation: A Cycled … · 2018-10-10 · Unpaired Sentiment -to Sentiment Translation: A Cycled Reinforcement Learning Approach 15 07 2018 4 of

Unpaired Sentiment-to-Sentiment Translation: A Cycled … · 2018-10-10 · Unpaired Sentiment -to Sentiment Translation: A Cycled Reinforcement Learning Approach 15 07 2018 4 of

Election 2016 Twitter Sentiment Map - web.stanford.edu · (2.3) Python script analyzes each tweet for sentiment (2.4) Sentiment returned to server (2.5) Tweet and sentiment sent to

Election 2016 Twitter Sentiment Map - web.stanford.edu · (2.3) Python script analyzes each tweet for sentiment (2.4) Sentiment returned to server (2.5) Tweet and sentiment sent to

Tree Communication Models for Sentiment Analysis · sentiment analysis over Stanford Sentiment Treebank, which allows the sentiment signals over hierarchical phrase structures to

Tree Communication Models for Sentiment Analysis · sentiment analysis over Stanford Sentiment Treebank, which allows the sentiment signals over hierarchical phrase structures to

Sentiment Homework

Sentiment Homework

Sentiment Analysis and Subjectivityfaculty.baruch.cuny.edu/dluna/consling/sentanalysisch.pdf(called sentence-level sentiment classification). 3. Feature-based sentiment analysis: This

Sentiment Analysis and Subjectivityfaculty.baruch.cuny.edu/dluna/consling/sentanalysisch.pdf(called sentence-level sentiment classification). 3. Feature-based sentiment analysis: This

Weekly Sentiment

Weekly Sentiment

The disruption-vs-value debate: Defining the pace of your ... Group Presentations... · Yammer sentiment analysis • Set up a Flow to evaluate sentiment of Yammer group posts •

The disruption-vs-value debate: Defining the pace of your ... Group Presentations... · Yammer sentiment analysis • Set up a Flow to evaluate sentiment of Yammer group posts •

Introduction to Sentiment Analysis - ETH Z · sentiment ! Sentiment analysis is also known as opinion mining L Sanders 3 What is Sentiment Analysis Sentiment analysis is the operation

Introduction to Sentiment Analysis - ETH Z · sentiment ! Sentiment analysis is also known as opinion mining L Sanders 3 What is Sentiment Analysis Sentiment analysis is the operation

Deep-Sentiment: Sentiment Analysis Using …Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models Shervin Minaee , Elham Azimi , AmirAli Abdolrashidiy New York

Deep-Sentiment: Sentiment Analysis Using …Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models Shervin Minaee , Elham Azimi , AmirAli Abdolrashidiy New York

Tutorial on Universal Dependenciesuniversaldependencies.org/eacl17tutorial/applications.pdf · • SyntaxNet • Cross-lingual parsing • Usually motivated by a low-resource scenario

Tutorial on Universal Dependenciesuniversaldependencies.org/eacl17tutorial/applications.pdf · • SyntaxNet • Cross-lingual parsing • Usually motivated by a low-resource scenario

Sentiment Analysis & Opinion Mining€¦ · Sentiment Analysis Sentiment Classification System Experimente Perspektiven * Abbildung dem Sinn nach entnommen aus Heyer (2006: 5). Sentiment

Sentiment Analysis & Opinion Mining€¦ · Sentiment Analysis Sentiment Classification System Experimente Perspektiven * Abbildung dem Sinn nach entnommen aus Heyer (2006: 5). Sentiment

Sentiment Analysis and the Consumer Genome2012w.sentimentsymposium.com/presentations/SAS12w-Vaidyanath… · Sentiment Analysis and the Consumer Genome Sentiment Analysis Symposium

Sentiment Analysis and the Consumer Genome2012w.sentimentsymposium.com/presentations/SAS12w-Vaidyanath… · Sentiment Analysis and the Consumer Genome Sentiment Analysis Symposium