Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task...
Transcript of Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task...
![Page 1: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/1.jpg)
CrowdsourcingMarkus Rokicki
![Page 2: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/2.jpg)
FOR CERTAIN TASKS, THE CORTEX STILL BEATS THE CPU
“We’re going to consider the human brain as an extremely advanced processor that can solve problems that computers cannot yet solve. Even more, we’re going to consider all humanity as an extremely advanced and large scale distributed processing unit that can solve large scale problems that computers cannot yet solve. …. What I want to advocate ... is a symbiotic relationship, a symbiosis in which humans solve some problems, computers solve others, and together we work to create a better world.”
--- Luis von Ahn, 2006
Luis von Ahn - one of the pioneers of crowdsourcing
2
![Page 3: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/3.jpg)
● Human computation is a computer science technique in which a machine performs its function by outsourcing certain steps to humans (Wikipedia).
● Social computing is related to humans in social role where their communication is mediated by technology (Parameswaran and Whinston, 2007).
● Crowdsourcing is the act of conducting traditional human work with ordinary people (Howe 2008).
● Collective intelligence is a superset of social computing, human computation and crowdsourcing. This term is defined by Malone et al. (2009) as groups of individuals doing things collectively that seem intelligent.
Definitions
Romani R. and Calani Baranauskas M..Exploring Human Computation and Social Computing to Inform the Design Process. (ICEIS-2013)
![Page 4: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/4.jpg)
Collective Intelligence
4
The crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged (the average was closer to the ox's true butchered weight than the estimates of most crowd members)
![Page 5: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/5.jpg)
Collective Intelligence
5
4,000 experts80,000 articles200 years to developAnnual Updates
>~100,000 amateurs1.6 Million articles5 years to developReal-Time Updates
Expert $$$$
Util
ity
Masses $
# of contributors10 10,000+
![Page 6: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/6.jpg)
Example: Product Categorization
6Figure: Product categorization task on Amazon Mechanical Turk1
1https://www.mturk.com
![Page 7: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/7.jpg)
Human Computation FrameworkCustomer
● Researcher
● Online store
● Manufacturer
● etc.
Crowd
Result
How to obtain knowledge from the crowd?
7
?
Problem
● Monitor animal population
● Improving IR systems
● Translations● Sentiment
analysis● Video surveillance● Quality check● Audio
Surveillance
![Page 8: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/8.jpg)
Human Computation FrameworkCustomer Problem Microtasks
● Monitor animal population
● Improving IR systems
● Translations● Sentiment
analysis● Video surveillance● Quality check● Audio
Surveillance
● Catch a fly
● Judge relevance of
a document
● Classify item
● Translate a
sentence
● Solve capture
● Detect damage
● Tag an image
● Identify persons
Design
● UI
● Prevent spam
● Engage
● Facilitate
● Platform
● Bias
● Motivation
● Size
● Give feedback
● Pay
● Researcher
● Online store
● Manufacturer
● etc.
● Filter
● Judge
● Weight
Recruit Process Aggregate
Result
Questions to be answered● How to create meaningful tasks?
● How to find suitable crowd?
● How to motivate to carry out my task?
● How to aggregate the results?8
![Page 9: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/9.jpg)
Human Computation FrameworkCustomer Problem Microtasks
● Monitor animal population
● Improving IR systems
● Translations● Sentiment
analysis● Video
surveillance● Quality check● Audio
Surveillance
● Catch a fly
● Judge relevance of
a document
● Classify item
● Translate a
sentence
● Solve capture
● Detect damage
● Tag an image
● Identify persons
Design
● UI
● Prevent spam
● Engage
● Facilitate
● Platform
● Bias
● Motivation
● Size
● Give feedback
● Pay
● Researcher
● Online store
● Manufacturer
● etc.
● Filter
● Judge
● Weight
Recruit Process Aggregate
Result
● HIT: Human Intelligence Task
9
![Page 10: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/10.jpg)
Crowdsourcing infrastructures
● Paid Crowdsourcing○ Crowdflower○ Mechanical Turk
● Virtual Citizen Science○ Zooniverse○ Crowdcrafting
● Hybrid Systems○ Crowd Databases
10
![Page 11: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/11.jpg)
11
Paid Microtask Crowdsourcing
![Page 12: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/12.jpg)
Paid Microtask Crowdsourcing
12
Figure: Paid Crowdsourcing Scheme2
2Figure source: http://dx.doi.org/10.1155/2014/135641
![Page 13: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/13.jpg)
Gamification
13
Task design & quality control
Important for quality, worker retention and throughput:
● Task pricing
● Worker competence/screening
● Requester reputation
● Task packaging
● Task framing & priming
● Task clarity
● ...
![Page 14: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/14.jpg)
Gamification
14
Task design & quality control
Important for quality, worker retention and throughput:
● Task pricing
● Worker competence/screening
● Requester reputation
● Task packaging
● Task framing & priming
● Task clarity
● ...
vague, blank, unclear, inconsistent,
imprecise, ambiguous, or poor
too many words, high standard of English, broken English, spelling
![Page 15: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/15.jpg)
Gamification
15
Task design & quality control
Important for quality, worker retention and throughput:
● Task pricing
● Worker competence/screening
● Requester reputation
● Task packaging
● Task framing & priming
● Task clarity
● ...
![Page 16: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/16.jpg)
Paper 1: Dynamics and task types on MTurk
Data-driven analysis of microtask crowdsourcing platform. Based on scraping information:
16
Djellel Eddine Difallah et al. “The dynamics of micro-task crowdsourcing: The case of amazon mturk”. In: Proceedings of
the 24th International Conference on World Wide Web. ACM. 2015, pp. 238–247.
![Page 17: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/17.jpg)
Paper 1: Dynamics and task types on MTurk
17
Djellel Eddine Difallah et al. “The dynamics of micro-task crowdsourcing: The case of amazon mturk”. In: Proceedings of
the 24th International Conference on World Wide Web. ACM. 2015, pp. 238–247.
Figure: Distributions of rewards per HIT between 2009 and 2013.
![Page 18: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/18.jpg)
Gamification
18
Quality control
asdf asdf asdf asdf
asdf asdf asdf asdf
asdf asdf asdf asdf
Paid Microtask Crowdsourcing
![Page 19: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/19.jpg)
Gamification
19
Gold standard data
● Relying on questions with priorly known answers to filter out low quality workers
○ Gold standard questions are mixed with “real” tasks
○ Workers usually receive feedback on answers to gold standard questions
● Good gold standard data○ Unambiguous○ Score-able automatically○ Not easily detectable
![Page 20: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/20.jpg)
Gamification
20
Qualification tests & pre-screening
Qualification Tests
● Using a sample/simulating real task as a qualification test.
Pre-screening
● Using specific criteria to select workers who are potentially more suitable to provide high quality responses (eg. personality type, demographic traits, etc.).
● Reputation systems
![Page 21: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/21.jpg)
Paper 2: Postprocessing results for quality
● How to learn from noisy human (expert and crowd) labels?
● Estimate worker reliability and true labels using EM○ Each worker is assigned a reliability○ Iterate:
■ E-step: Estimate hidden true labels of the data given current estimated worker reliability■ M-step: Estimate worker reliability given current estimated labels
● Scenarios:○ Binary classification○ Multi-class classification○ Ordinal regression○ Regression
21
Vikas C Raykar et al. “Learning from crowds”. In: Journal of Machine Learning Research 11.Apr (2010), pp. 1297–1322.
![Page 22: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/22.jpg)
Paper 2: Postprocessing results for quality
22Vikas C Raykar et al. “Learning from crowds”. In: Journal of Machine Learning Research 11.Apr (2010), pp.
1297–1322.
![Page 23: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/23.jpg)
Paper 2: Postprocessing results for quality
23Vikas C Raykar et al. “Learning from crowds”. In: Journal of Machine Learning Research 11.Apr (2010), pp.
1297–1322.
![Page 24: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/24.jpg)
Motivation
Classic motivations:
● Paid Crowdsourcing● Gamification● Contribution to Society
Other aspects:
● Curiosity● Team responsibility● Social pressure● etc.
24
![Page 25: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/25.jpg)
Financial Methods of Motivation
25
Employment regulation
Are the payment rules clear?
Recruitment & Retention
Is the salary competitive?
Pay linked to performance
Is the salary fair?
Individual vs. Group incentives
What is the trade off?
![Page 26: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/26.jpg)
Financial Rewards: Crowdsourcing Apps
26
Influence of Payment
➢ 611 participants sorted 36,000 images sets of varying size, for varying payments.
Winter Mason and Duncan J Watts. “Financial incentives and the performance of crowds”. In: ACM SigKDD Explorations Newsletter 11.2 (2010), pp. 100–108.
![Page 27: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/27.jpg)
Financial Rewards: Influence of payment
27Winter Mason and Duncan J Watts. “Financial incentives and the performance of crowds”. In: ACM SigKDD Explorations Newsletter 11.2 (2010), pp. 100–108.
Figure: Accuracy (left) and number of completed tasks (right) in relation to payment
![Page 28: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/28.jpg)
Financial Rewards: Influence of payment
28Winter Mason and Duncan J Watts. “Financial incentives and the performance of crowds”. In: ACM SigKDD Explorations Newsletter 11.2 (2010), pp. 100–108.
Figure: Post-hoc survey of perceived value
![Page 29: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/29.jpg)
Paper 3: Influence of compensation and payment on quality
● Empirical study on influences on overall judgment quality
● Varying○ payment○ worker qualifications○ Required effort
● Scenario: relevance assessments for information retrieval
29
Gabriella Kazai. “In search of quality in crowdsourcing for search engine evaluation”. In: European Conference
on Information Retrieval. Springer. 2011, pp. 165–176.
![Page 30: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/30.jpg)
Paper 3: Influence of compensation and payment on quality
30
Gabriella Kazai. “In search of quality in crowdsourcing for
search
engine evaluation”. In: European Conference on Information
Retrieval.
Springer. 2011, pp. 165–176.
![Page 31: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/31.jpg)
Gamification
31
Complex Crowdsourcing and workflow design
1. Task decomposition: Complex tasks can be decomposed into smaller independent or dependent tasks (multi-stage workflows)
2. Managing dependencies between sub-tasks: Decomposed tasks can be setup and deployed in parallel or in sequence (different UIs, data, etc.)
3. Assembling results
JOB 1 JOB 2 JOB ‘n’
![Page 32: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/32.jpg)
Paper 4: Collaborative workflows
Word processing interface that integrates workers to edit parts of documents on demand.
Find-Fix-Verify: split tasks into generation and review stages.
32Bernstein, Michael S., et al. "Soylent: a word processor with a crowd inside." Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 2010.
![Page 33: Crowdsourcing - KBS€¦ · of crowdsourcing 2 Human computation is a computer science ... Task design & quality control Important for quality, worker retention and throughput: ...](https://reader034.fdocuments.in/reader034/viewer/2022050102/5f414abb81ffa77c6815e923/html5/thumbnails/33.jpg)
Thank you!
33