Best Practices for Sentiment Analysis
-
Upload
francisco-javier-sibaja-hurtado -
Category
Documents
-
view
222 -
download
0
Transcript of Best Practices for Sentiment Analysis
-
8/6/2019 Best Practices for Sentiment Analysis
1/25
B e s t P r a c t i c e sf o r
S e n t i m e n tA n a l y s i s
Presented by:John HoskinsAmazon Mechanical Turk
Max YankelevichFreedom OSS CrowdControl
-
8/6/2019 Best Practices for Sentiment Analysis
2/25
2011 Amazon.com, Inc. or its Affiliates.
Introductions
John HoskinsAmazon Mechanical Turk
Sr. Manager, Business Development
Max YankelevichFreedomOSSChief Architect
-
8/6/2019 Best Practices for Sentiment Analysis
3/25
2011 Amazon.com, Inc. or its Affiliates.
Welcome Logistics
How to ask a question:.
Ask a question with the questionpanel.
We will moderate questions.
Agenda
Introductions
Sentiment Analysis
Applying Mechanical Turk
CrowdControl optimized for Sentiment
Q & A
-
8/6/2019 Best Practices for Sentiment Analysis
4/25
2011 Amazon.com, Inc. or its Affiliates.
What is Sentiment Analysis? Keeping your finger on the pulse of your market in
near real timeWith the explosion in use of Facebook, Twitter, and blogging it isessential to hear the true Voice of the Customer. Yet keeping up withthe round-the-clock information torrent is impossible with old,manual methods. As businesses look to automate the process offiltering out the noise, understanding the conversations, identifyingthe relevant content and actioning it appropriately, many are nowlooking to the field of sentiment analysis[
-
8/6/2019 Best Practices for Sentiment Analysis
5/25
2011 Amazon.com, Inc. or its Affiliates.
Why Sentiment Analysis?
The ability to lead the competition in customer satisfaction and support,
brand and reputation management
product design and marketing
With the proliferation of reviews, ratings,recommendations and other forms of online expression,
online opinion has turned into a kind of virtual currencyfor businesses looking to market their products, identifynew opportunities and manage their reputations.
-
8/6/2019 Best Practices for Sentiment Analysis
6/25
2011 Amazon.com, Inc. or its Affiliates.
How to properly analyzeSentiment.
-
8/6/2019 Best Practices for Sentiment Analysis
7/25 2011 Amazon.com, Inc. or its Affiliates.
Context & Cognitive Recognitionare the keys to an accurate analysisHuman analysis provides more accurate assessment. Complex emotions such as sarcasm are presented in tweets and
blog posts.
Wit, sarcasm and complex emotions are difficult to analyze withtechnology alone
Processing the unnatural language of text messaging: lol, omg
Positive and negative emotions are difficult to accurately assess
Avoid misleading classification of SPAM
-
8/6/2019 Best Practices for Sentiment Analysis
8/25 2011 Amazon.com, Inc. or its Affiliates.
Is human judgmentaffordable and scalable?
Yes.
-
8/6/2019 Best Practices for Sentiment Analysis
9/25 2011 Amazon.com, Inc. or its Affiliates.
Introducing Mechanical TurkAn affordable solution for human judgment.
-
8/6/2019 Best Practices for Sentiment Analysis
10/25 2011 Amazon.com, Inc. or its Affiliates.
Mechanical Turk is amarketplace for work.Mechanical Turk gives businesses and developers access to
an on-demand, scalable workforce. Flexibility: Scale your workforce up and down quickly
Accuracy: Get high quality, efficient and cost effectiveresults.
Price: Pay only when you are satisfied with the results. Speed: Start receiving results in minutes
-
8/6/2019 Best Practices for Sentiment Analysis
11/25 2011 Amazon.com, Inc. or its Affiliates.
Workforce
500,000 Workers
190+ Countries
Who are the Workers? Workers are global: 24X7 Follow the Sun
Managing Your Workforce Can narrow to US based (i.e. when you need western
culture competence)
Leverage qualifications to find your best Workers
-
8/6/2019 Best Practices for Sentiment Analysis
12/25 2011 Amazon.com, Inc. or its Affiliates.
How it Works
Design Your Task
(HIT)
Publish your HIT tothe marketplace
You TheRequester
Accepts your HIT
Submits an answer
The Worker Validate the Results
Pay Workers
Go!
You TheRequester
-
8/6/2019 Best Practices for Sentiment Analysis
13/25 2011 Amazon.com, Inc. or its Affiliates.
Popular Use Cases
Data Management Data Verification
Data Entry & Collection
Data De-duplication
Algorithm Training
Content & Media Moderate Photos & Content
Content Creation & Editing
Transcription
Categorization Classification
Tagging
Sentiment Analysis
Business Services Search Relevancy
Product Usability Testing
Research
-
8/6/2019 Best Practices for Sentiment Analysis
14/25 2011 Amazon.com, Inc. or its Affiliates.
Case Study Sentiment Analysis Problem:
A large consumer brand reporting customer need 10s ofthousands of human coded samples to establish a baseline for each project
Solution:Code them through Mechanical Turk.
Details:
Projects are completed in hours providing faster to marketproduction of broad analysis.
By integrating Mechanical Turk into their process, they freed upanalysts time to do value add work and can start projects in a dayinstead of weeks.
-
8/6/2019 Best Practices for Sentiment Analysis
15/25 2011 Amazon.com, Inc. or its Affiliates.
Strategies for Analyzing Results
PluralityAsking multiple Workers to do assignments (iftheir answers agree then the result is validated)
Qualifying & Training WorkersAssess competence on your coding instructionsbefore allowing them to work for you
Known Data Sets
Include work with known answers to quicklyassess worker accuracy.
-
8/6/2019 Best Practices for Sentiment Analysis
16/25 2011 Amazon.com, Inc. or its Affiliates.
Implementing a SentimentAnalysis Workflow
-
8/6/2019 Best Practices for Sentiment Analysis
17/25 2011 Amazon.com, Inc. or its Affiliates.
Freedom at a Glance Long term AWS Partner
Enterprise Cloud Business
CrowdControl fastest growing division
Corporate Office: Newtown PA Regional Offices: Newark, NJ. Reston, VA, Seattle, WA
Offshore Engineering Centers
Global operations centers
Privately held
Established in 2008
300% annual growth
-
8/6/2019 Best Practices for Sentiment Analysis
18/25 2011 Amazon.com, Inc. or its Affiliates.
Workflow, Adjudication,Worker Management.Programmatic connection to crowdsourcing. How do I know I have the best answer?
How do I break my workflow into discrete tasks
Who are my best workers?
How do I put it all together?
-
8/6/2019 Best Practices for Sentiment Analysis
19/25 2011 Amazon.com, Inc. or its Affiliates.
CrowdControl for Sentiment Analysis Provides High Sentiment Quality Data at Lower Cost
Ability to Process Large Number of UGC
Combines best of breed Artificial Intelligence to handle mostchallenging nuances of Crowdsourcing for efficientSentiment Analysis
Worker Management
Adjudication strategies Workflows
Can easily retrieve and send data to and from any datasource (e.g. Database, File ,etc.)
-
8/6/2019 Best Practices for Sentiment Analysis
20/25 2011 Amazon.com, Inc. or its Affiliates.
What Can CrowdControl Do for Me?
Mechanical Turk Portal
Turkers
Mechanical Turk Portal
Turkers
ManualSteps
CustomerIT
SystemsStep
Create sentiment coding templates
Step Come up with opinion adjudication rules
Step Extract data from Internal and External IT Systems
Step Review responses
Step Ban bad workers
Step Accept or reject answers
Step Pay workers
Step Assign special worker qualifications
Step
Import data back into IT Systems
CustomerIT
Systems
Complete
Automation
QualityInformation
-
8/6/2019 Best Practices for Sentiment Analysis
21/25 2011 Amazon.com, Inc. or its Affiliates.
Sentiment Analysis Process Setup
-
8/6/2019 Best Practices for Sentiment Analysis
22/25 2011 Amazon.com, Inc. or its Affiliates.
Brand Sentiment Coding Template
-
8/6/2019 Best Practices for Sentiment Analysis
23/25 2011 Amazon.com, Inc. or its Affiliates.
-
8/6/2019 Best Practices for Sentiment Analysis
24/25 2011 Amazon.com, Inc. or its Affiliates.
John Hoskins, Amazon Web Services:
Max Yankelevich, FreedomOSS:
@amazonmturk
Facebook.com/amazonmturk
Mechanicalturk.typepad.com
-
8/6/2019 Best Practices for Sentiment Analysis
25/25
Q & AYour Questions. Answered.