Work Like a BeeTaking Advantage of Diligent Crowdsourcing Workers

Presentation: Michael Riegler, Simula Research Laboratory, Norway Preben N. Oslen, Simula Research Laboratory Pål Halvorsen, Simula Research Laboratory michael@simula.no

The Idea• Use “good” workers

• Like the task or good with it

• Will do it much better

• Diligent workers

• Do most of the work

• Extra: additional crowd workers

The Runs

• 4 different runs

• Run 1: MJV with additional crowdsourcing

• Run 2: Diligent Workers vote only

• Run3: MJV with weighted diligent workers

• Run4: Run3 with additional crowdsourcing

The ResultsRun WF1 score True Labels

Predicted Labels

R1 0.7207 183, 63, 291 192, 68, 279

R2 0.6919 183, 63, 291 208, 95, 234

R3 0.6912 183, 63, 291 208, 87, 242

R4 0.6919 183, 63, 291 208, 95, 234

Baseline 0.3809 183, 63, 291 0, 0, 537

The Analysis

• Every run is better than the baseline

• R1 is the best performing run

• Difference between R1 and R2 shows that diligent workers perform better

• Can for example be used to save costs

• R2, R3 and R4 perform similar

• Additional Crowdsourcing does not improve results

The Conclusion

• Very simple approach

• Showed that it can work

• No gain trough additional crowdsourcing

• Extending work on larger datasets

Tusen takk og ha det bra!

The End