CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility Alex...
-
date post
21-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility Alex...
CrowdFlowIntegrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility
Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin
Human Computation
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
Translation
Photo tagging
Face recognition
Human detection
Speech recognition
Text analysis
Planning
Human Computation
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
Translation
Photo tagging
Face recognition
Speech recognition
Human detection
Text analysis
Planning
Example: Human detection
Trade-off space
Quality
Spee
d, A
fford
abili
ty
Computers
Human Workers(traditional)
Human Computation
Trade-off space
Quality
Spee
d, A
fford
abili
ty
Computers
Human Computation
Human Workers(traditional)
Man-Computer Symbiosis
Automation with human
post-correction
Supervisedmachinelearning
humans
computer
speedcostquality
computer
humans
speedcostquality
Man-Computer Symbiosis
CrowdFlowAutomation with human
post-correction
Supervisedmachinelearning
humanscomputer
speedcostquality
humans
computer
speedcostquality
computer
humans
speedcostquality
Mechanical Turk
Human Detection – Starting point
Human Detection – Task
Human Detection – Results
Quality
Spee
d, A
fford
abili
ty
60% 90%
119 images took3 hrs 50 mins
and cost $2.38
Human Detection – Scenarios
Quality
Spee
d, A
fford
abili
ty
60% 90%
1000 photos at 72% accuracy would take 12 hrs 20 mins and cost $8.00
119 images took3 hrs 50 mins
and cost $2.38
Vision: Richer modelInput with computer results
Validator
Appraiser
Fixer Worker
Correct Incorrect
Fix Start over
Output
Lessons Learned Design for overall needs/constraints
Practical advice: Pay consistently and reasonably Reject only work that is definitely cheating Build in fair cheating deterrence from the start Keep instructions short, but always clear
Contact: Alex Quinn [email protected]