AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Transformational Impact of Cloud Labor

John Hoskins & Daniel Grayjhoskins@amazon.com

djgray@amazon.com

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Crowdsourcing Best Practices

amazonweb services

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Crowdsourcing myths

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

• It’s cheaper– It’s actually more efficient

• It’s faster– It’s actually more scalable

• It’s not accurate– It’s actually more accurate

The Myths[ ]

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

5

Crowdsourcing Best Practices

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

• Consider the question carefully– Workers answer what you ask

• Select your workers– Perspective and skills vary

• Iterate and Optimize– Adjust for optimal results

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

The question[ ]7

You will get an answer to the question that you ask. Focus on asking the right question

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Choosing your workers[ ]8

Workers are different– from language and cultural differences – to varying skills. Test and monitor.

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Monitor and Improve[ ]9

Monitor key metrics, adjust and measure key attributes impact on those metrics.

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

• Accuracy– Know your current

• Throughput– Understand both turnaround and scale requirements

• Cost– Measure against a budget – as cost can impact the other two

Key Metrics[ ]10

“Great service, Good food, Friendly staff – you can choose two”

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Cost[ ]11

Cost is impacted most by the efficiency of the other two metrics.

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Accuracy[ ]12

Error has two sources: human and systematic. Isolating human error and solving for systematic error gives a better chance for long term success.

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Accuracy[ ]13

After solving for systematic error choosing the best workers and monitoring those workers provides the next step towards high accuracy and lowering costs.

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Throughput[ ]14

Many factors impact throughput;

Reputation

Ergonomics

Clarity

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

http://www.mturk.com

15

John Hoskins, Amazon Mechanical Turk

hoskins@amazon.com

amazonweb services

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Cost[ ]16

Cost is impacted most by the efficiency of

the other two metrics.

Optimization of task and workers lowers

both the cost of getting it done and

adjudicating a result.

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

Thank You