DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL...
Transcript of DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL...
![Page 1: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/1.jpg)
DELIVERING WEB EXPERIMENTS USINGAMAZON MECHANICAL TURK
CHUN CHAN
18 OCT 2017
![Page 2: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/2.jpg)
INTRODUCTION
¡ Software developer with the Linguistics Department and CSD
¡ Developing MTurk based experiments for about 3 years
![Page 3: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/3.jpg)
MTURK URLS
¡ Sandbox
¡ https://requestersandbox.mturk.com/begin_signin
¡ https://workersandbox.mturk.com/mturk/beginsignin
¡ Production
¡ https://requester.mturk.com/mturk/beginsignin
¡ https://www.mturk.com/mturk/beginsignin
![Page 4: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/4.jpg)
https://acceptability-judgements-demo.firebaseapp.com/
![Page 5: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/5.jpg)
1. External Link 2. Internal Single Page Survey 3. Embedded
Easy setupDifficult close out
Complex setupEasier close out
![Page 6: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/6.jpg)
EXTERNAL LINK – WORKER’S VIEWS
¡ Live demo
![Page 7: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/7.jpg)
EXTERNAL LINK – REQUESTER’S VIEW
¡ Tutorial
¡ https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-qualtrics-f5366f0bd880
![Page 8: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/8.jpg)
![Page 9: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/9.jpg)
![Page 10: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/10.jpg)
![Page 11: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/11.jpg)
![Page 12: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/12.jpg)
![Page 13: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/13.jpg)
![Page 14: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/14.jpg)
![Page 15: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/15.jpg)
![Page 16: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/16.jpg)
![Page 17: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/17.jpg)
![Page 18: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/18.jpg)
1. External Link 2. Internal Single Page Survey 3. Embedded
Easy setupDifficult close out
Complex setupEasier close out
![Page 19: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/19.jpg)
SINGLE PAGE SURVEY
¡ Worker doesn’t have to visit an external URL
¡ No confirmation codes to validate
¡ Involves writing some HTML
¡ https://bradylab.ucsd.edu/ttt/
¡ Data gets saved in Mturk
![Page 20: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/20.jpg)
SINGLE PAGE SURVEY – WORKER’S VIEWS
¡ Live demo
![Page 21: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/21.jpg)
![Page 22: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/22.jpg)
![Page 23: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/23.jpg)
![Page 24: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/24.jpg)
![Page 25: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/25.jpg)
![Page 26: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/26.jpg)
1. External Link 2. Internal Single Page Survey 3. Embedded
Easy setupDifficult close out
Complex setupEasier close out
![Page 27: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/27.jpg)
EMBEDDED
¡ Worker doesn’t have to visit external URL
¡ No confirmation codes to validate
¡ Data is saved to Firebase or Qualtrics
¡ Better functionality from worker’s perspective
![Page 28: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/28.jpg)
EMBEDDED SURVEY – WORKER’S VIEWS
¡ Live demo
![Page 29: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/29.jpg)
EMBEDDED
¡ Mturk Command Line Tools (CLI)
¡ https://requester.mturk.com/developer/tools/clt
¡ http://web.mit.edu/cathywu/Public/AMT/aws-mturk-clt-1.3.0/UserGuide.html#externalhit
¡ AMT Manager
¡ https://nusubdb.ling.northwestern.edu/AMTManager/
¡ Invite only for now
![Page 30: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/30.jpg)
EMBEDDED
¡ Firebase or Qualtrics compatible
¡ Some special code “hooks” required
¡ Templates available if interested
¡ Speech In Noise frameworks
![Page 31: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/31.jpg)
https://acceptability-judgements-demo.firebaseapp.com/
![Page 32: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/32.jpg)
![Page 33: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/33.jpg)
![Page 34: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/34.jpg)
![Page 35: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/35.jpg)
![Page 36: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/36.jpg)
![Page 37: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/37.jpg)
workerId = ABC1234567
https://www.mturk.com/mturk/externalSubmit?workerId = ABC1234567
![Page 38: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/38.jpg)
IN SUMMARY
External URL¡ Worker must visit an external URL
¡ Can’t close original window
¡ Worker enters confirmation code (possible typos)
¡ Data is saved to Firebase or Qualtrics
¡ Requester matches confirmation codes to workers
Single Page Survey¡ Worker doesn’t have to visit an external URL
¡ Long scrolling page, difficult for worker
¡ No confirmation codes to validate
¡ Involves writing some HTML
¡ Data gets saved in Mturk
![Page 39: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/39.jpg)
IN SUMMARY
Embedded Survey¡ Worker doesn’t have to visit external URL
¡ No confirmation codes to validate
¡ Data is saved to Firebase or Qualtrics
¡ Better functionality from worker’s perspective
¡ Little more work to get set up initially (Firebase/Qualtrics templates)
¡ Can embed games, other types of interesting tasks
¡ Speech in Noise frameworks
¡ https://northwestern.box.com/s/9g2rigz1iqh4ymfkgunq6t31u3iycbpr
![Page 40: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/40.jpg)
THE LIFE OF AN MTURK WORKER
![Page 41: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/41.jpg)
https://turkopticon.ucsd.edu/
![Page 42: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/42.jpg)
![Page 43: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/43.jpg)
![Page 44: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/44.jpg)
WHY USE MTURK
¡ Easy set up
¡ Diverse population
¡ More data over a short period of time
¡ Everyone else is doing it
![Page 45: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/45.jpg)
REASONS NOT TO USE MTURK
¡ You have to manage everything
¡ Payments
¡ Technical issues
¡ Angry workers
¡ Preventing the same workers from repeating your HIT
¡ Can get expensive
¡ Not all workers are naive
¡ https://doi.org/10.1016/j.tics.2017.06.007)
¡ Not all languages and countries are evenly represented
¡ https://www.semanticscholar.org/paper/The-Language-Demographics-of-Amazon-Mechanical-Tur-Pavlick-Post/034bd3a6f63caf2bb2dc2762e43401cd204b86b5)
![Page 46: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/46.jpg)
ALTERNATIVES
¡ Turkprime
¡ https://www.turkprime.com
¡ Prolific
¡ https://www.prolific.ac/
![Page 47: DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL …groups.linguistics.northwestern.edu/documentation/v2/... · 2017-10-18 · DELIVERING WEB EXPERIMENTS USING AMAZON MECHANICAL](https://reader034.fdocuments.in/reader034/viewer/2022042303/5ece9baaad639c66df582a30/html5/thumbnails/47.jpg)
QUESTIONS?