Intervention Strategies for Increasing Engagement in Crowdsourcing
-
Upload
smart-society-project -
Category
Science
-
view
36 -
download
2
Transcript of Intervention Strategies for Increasing Engagement in Crowdsourcing
![Page 1: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/1.jpg)
1
Intervention Strategies for Increasing Engagement in Crowdsourcing: Platform, Predictions and Experiments
Avi Segal, Ben-Gurion University, Israel
Ya’akov (Kobi) Gal, Ben-Gurion University, Israel
Ece Kamar, Microsoft Research, Redmond WA, USA
Eric Horvitz, Microsoft Research, Redmond WA, USA
Alex Bowyer, University of Oxford, UK
Grant Miller, University of Oxford, UK
![Page 2: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/2.jpg)
Citizen Science (Zooniverse.org)
Wikipedia: scientific research conducted, in whole or in part, by amateur or nonprofessional scientists.
2
![Page 3: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/3.jpg)
Galaxy Zoo
Conclusions
3
![Page 4: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/4.jpg)
The Disengagement Problem
4
Eveleigh, Alexandra, et al. "Designing for dabblers and deterring drop-outs in citizen science.” CHI 2014Sauermann, Henry, and Chiara Franzoni. "Crowd science user contribution patterns and their implications." PNAS 2015
![Page 5: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/5.jpg)
The Approach: Intervention
5
Dimension Description Examples
Target Target population for intervention all, newcomers, heavy users
Time Intervention timing periodical; as soon as risk factor is identified; one day after disengagement
Duration Intervention (interruption) duration for x minutes; until intervention is acknowledged; while user is on webpage
Media Type Method of presentation text; graphics; audio; video; mixed
Channel Medium of delivery email; web page; modal message
Mechanism The method used for this intervention
Messages; help system; task routing; competition; achievements; …
Content Content of the intervention communication
“...If GalaxyZoo didn't suit you, then…”; checkmark image;
![Page 6: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/6.jpg)
Disengagement Prediction Intervention Evaluation
Dimensions: When & What
6
![Page 7: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/7.jpg)
The Intervention Server Platform
7
![Page 8: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/8.jpg)
Predicting Disengagement
8
Predict Disengagement
Intervene
Evaluate
Supervised Learning
Supervised Learning
User will disengage within given window
Session & History
1M records13.5K users
Andrew Mao, Ece Kamar, and Eric Horvitz. "Why stop now? predicting worker engagement in online crowdsourcing.” HCOPM 2013
![Page 9: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/9.jpg)
Intervention Message: Helpful
9
Predict Disengagement
Intervene
Evaluate
Please don’t stop just yet. You’ve been extremely helpful so far. Your votes are really helping us to understand deep
mysteries about galaxies.
Ryan, Richard M., and Edward L. Deci. "Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being." APA(2000)
![Page 10: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/10.jpg)
Intervention Message: Community
10
Predict Disengagement
Intervene
Evaluate
Thousands of people are taking part in the project every month. Visit Talk at
talk.galaxyzoo.org to discuss the images you see with them.
Kraut, Robert E., and Paul Resnick. "Encouraging contribution to online communities." Building successful online communities: Evidence-based social design 2011
![Page 11: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/11.jpg)
Intervention Message: Anxiety
11
Predict Disengagement
Intervene
Evaluate
We use statistical techniques to get the most from every answer; So, you don’t
need to worry about being “right”. Just tell us what you see.
Segal, Avi, et al. "Improving productivity in citizen science through controlled intervention." WWW 2015
![Page 12: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/12.jpg)
Intervention GUI
12
Predict Disengagement
Intervene
Evaluate
![Page 13: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/13.jpg)
Conditions
13
Predict Disengagement
Intervene
Evaluate
# Type
1. No Intervention
2. Random Timing, Helpful Message
3. Random Timing, Community Message
4. Random Timing, Anxiety Message
5. Prediction Based Timing, Helpful Message
6. Prediction Based Timing, Community Message
7. Prediction Based Timing, Anxiety Message
![Page 14: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/14.jpg)
Impact on Contributions
14
Predict Disengagement
Intervene
Evaluate
![Page 15: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/15.jpg)
Impact of History
15
Predict Disengagement
Intervene
Evaluate
![Page 16: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/16.jpg)
Conclusion
Increasing engagement = intervention design + intelligent timing
Code is open sourcedSequential decision making with uncertainty
Off policy policy evaluation Engagement Importance
16
![Page 17: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/17.jpg)
Thank You & [email protected]
![Page 18: Intervention Strategies for Increasing Engagement in Crowdsourcing](https://reader033.fdocuments.in/reader033/viewer/2022052514/5888d57e1a28aba1058b56c1/html5/thumbnails/18.jpg)
Human Aware AI?
30
Learn
Intervene
Evaluate
“Intervention” title 1Papers this morning ~100Learning ~80? 10Intervention? ~10Large scale int. ~2
% 2%
Is this what we want? Is this what we want?