Best Practices for Sentiment Analysis Webinar

25
Best Practices for Sentiment Analysis Presented by: John Hoskins Amazon Mechanical Turk Max Yankelevich Freedom OSS CrowdControl

description

An informative session on Amazon Mechanical Turk where you will learn how your company can leverage the human crowd for human sentiment analysis of content such as tweets, articles, RSS feeds and blog posts. This session digs into the details of getting started and provides information on how to be successful so you get accurate results. Additionally, FreedomOSS will share their experiences designing and managing sentiment tasks and demo's their CrowdControl crowdsourcing platform that is built on top of Mechanical Turk.

Transcript of Best Practices for Sentiment Analysis Webinar

Page 1: Best Practices for Sentiment Analysis Webinar

Best Practices for Sentiment Analysis

Presented by:John HoskinsAmazon Mechanical Turk

Max YankelevichFreedom OSS CrowdControl

Page 2: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Introductions

Page 3: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Welcome Logistics

How to ask a question:. Ask a question with the question

panel. We will moderate questions.

Agenda Introductions Sentiment Analysis Applying Mechanical Turk CrowdControl – optimized for Sentiment Q & A

Page 4: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

What is Sentiment Analysis? Keeping your finger on the pulse of your market –

in near real timeWith the explosion in use of Facebook, Twitter, and blogging – it is essential to hear the true Voice of the Customer. Yet keeping up with the round-the-clock information torrent is impossible with old, manual methods. As businesses look to automate the process of filtering out the noise, understanding the conversations, identifying the relevant content and actioning it appropriately, many are now looking to the field of sentiment analysis[

Page 5: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Why Sentiment Analysis?The ability to lead the competition

in customer satisfaction and support, brand and reputation managementproduct design and marketing

With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.

Page 6: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

How to properly analyze Sentiment.

Page 7: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Context & Cognitive Recognition are the keys to an accurate analysisHuman analysis provides more accurate assessment.Complex emotions such as sarcasm are presented in tweets and blog posts.Wit, sarcasm and complex emotions are difficult to analyze with technology aloneProcessing the unnatural language of text messaging: lol, omgPositive and negative emotions are difficult to accurately assessAvoid misleading classification of SPAM

Page 8: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Is human judgment affordable and scalable?

Yes.

Page 9: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Introducing Mechanical TurkAn affordable solution for human judgment.

Page 10: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Mechanical Turk is a marketplace for work. Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.

Flexibility: Scale your workforce up and down quickly

Accuracy: Get high quality, efficient and cost effective results.

Price: Pay only when you are satisfied with the results.

Speed: Start receiving results in minutes

Page 11: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Workforce500,000 Workers190+ Countries

Who are the Workers? Workers are global: 24X7 Follow the Sun

Managing Your Workforce Can narrow to US based (i.e. when you need

western culture competence) Leverage qualifications to find your best Workers

Page 12: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

How it Works

Page 13: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Popular Use Cases

Data Management Data Verification Data Entry & Collection Data De-duplication Algorithm Training

Content & Media Moderate Photos &

Content Content Creation &

Editing Transcription

Categorization Classification Tagging Sentiment Analysis

Business Services Search Relevancy Product Usability Testing Research

Page 14: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Case Study – Sentiment Analysis Problem:

A large consumer brand reporting customer need 10’s of thousands of human coded samples to establish a baseline – for each project

Solution: Code them through Mechanical Turk.

Details: Projects are completed in hours providing faster to

market production of broad analysis. By integrating Mechanical Turk into their process, they

freed up analysts time to do value add work and can start projects in a day instead of weeks. 

Page 15: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Strategies for Analyzing Results Plurality

Asking multiple Workers to do assignments (if their answers agree then the result is validated)

Qualifying & Training WorkersAssess competence on your coding instructions before allowing them to work for you

Known Data Sets Include work with known answers to quickly assess worker accuracy.

Page 16: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Implementing a Sentiment Analysis Workflow

Page 17: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Freedom at a Glance Long term AWS Partner

Enterprise Cloud Business CrowdControl – fastest growing division

Corporate Office: Newtown PA Regional Offices: Newark, NJ. Reston, VA, Seattle,

WA Offshore Engineering Centers Global operations centers

Privately held Established in 2008 300% annual growth

Page 18: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Workflow, Adjudication, Worker Management. Programmatic connection to crowdsourcing.How do I know I have the best answer?How do I break my workflow into discrete tasksWho are my best workers?How do I put it all together?

Page 19: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

CrowdControl for Sentiment Analysis Provides High Sentiment Quality Data at Lower

Cost

Ability to Process Large Number of UGC

Combines best of breed Artificial Intelligence to handle most challenging nuances of “Crowdsourcing” for efficient Sentiment Analysis Worker Management Adjudication strategies Workflows Can easily retrieve and send data to and from any data

source (e.g. Database, File ,etc.)

Page 20: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

What Can CrowdControl™ Do for Me?

Mechanical Turk Portal

Turkers

Mechanical Turk Portal

Turkers

Manual S

teps

CustomerIT

Systems

CustomerIT

Systems

Com

ple

te

Auto

matio

n

Quality Information

Page 21: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Sentiment Analysis Process Setup

Page 22: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Brand Sentiment Coding Template

Page 23: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Page 24: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

John Hoskins, Amazon Web Services: [email protected] Yankelevich, FreedomOSS: [email protected]

@amazonmturk

Facebook.com/amazonmturk

Mechanicalturk.typepad.com

Page 25: Best Practices for Sentiment Analysis Webinar

© 2011 Amazon.com, Inc. or its Affiliates.

Q & AYour Questions. Answered.