The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and...
-
Upload
lily-oneal -
Category
Documents
-
view
213 -
download
0
Transcript of The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and...
![Page 1: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/1.jpg)
The Start of the ArtIntroduction to the Workshop on Crowdsourcing Technologies for
Language and Cognition Research
Robert Munro and Hal TilyStanford University and MIT
![Page 2: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/2.jpg)
Acknowledgments
University of Colorado & the LSA David Clausen, StanfordCrowdflowerReview committee PresentersBeth Levin and Tom Wasow
![Page 3: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/3.jpg)
Daily potential language exposure
How many languages could you hear on any given day?
How has this changed?
Year
# o
f la
nguages
![Page 4: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/4.jpg)
Daily potential language exposure
Year
# o
f la
nguages
![Page 5: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/5.jpg)
Daily potential language exposure
Year
# o
f la
nguages
![Page 6: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/6.jpg)
Daily potential language exposure
Year
# o
f la
nguages
![Page 7: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/7.jpg)
Daily potential language exposure
Year
# o
f la
nguages
Our potential communications will never be so diverse as right now
![Page 8: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/8.jpg)
Crowdsourcing (microtasking)People completing short tasks
◦typically online for a few cents eachWho logs on to complete
microtasks?◦~1,000,000 people daily
Who can create tasks for workers?◦Anyone (on many platforms, Amazon
Mechanical Turk from Nov 2005)What kind of tasks can you create?
◦Anything embeddable in a browser or phone
![Page 9: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/9.jpg)
Typical tasks
Transcription of a business card
![Page 10: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/10.jpg)
Typical tasks
Quality control for OCR
![Page 11: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/11.jpg)
Typical tasks
Write a comment on a blog post
![Page 12: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/12.jpg)
Crowdsourcing and languagePowerset (2007~8)
◦Semantic annotation (Snow et al. 2008)
◦Linguistic research (Munro et al. 2010)
◦Quality control across multiple worker platforms (CrowdFlower)
![Page 13: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/13.jpg)
Crowdsourcing and languageCommercial transcription and
translation
![Page 14: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/14.jpg)
Who are the workers?
(source: http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html)
![Page 15: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/15.jpg)
Who are the workers?
(source: http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html)
![Page 16: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/16.jpg)
Who are the workers?
(source: http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html)
![Page 17: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/17.jpg)
Who are the workers?
(source: http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html)
![Page 18: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/18.jpg)
Who are the workers?
(source: http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html)
![Page 19: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/19.jpg)
What languages do they speak?
![Page 20: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/20.jpg)
What languages do they speak?
![Page 21: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/21.jpg)
Why people work
(source: http://waxy.org/2008/11/the_faces_of_mechanical_turk/)
![Page 22: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/22.jpg)
Crowdsourcing and language
![Page 23: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/23.jpg)
Crowdsourcing and research
![Page 24: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/24.jpg)
Will people enjoy language tasks? “It’s really nice to remind me what I am” “I am hoping that MTurk will provide more opportunities for
translations in Spanish!” “What about localizing mturk in different languages so
that any one can easily work in mturk” “It is a very enjoyable thing to do research in linguistics.” “I’m an outlier, because I love languages so much, but you
can count me in, I suppose. Good luck with your research.” “Used to be fluent in Latin, but it’s hard to stay in practice
with a dead language.” “Tamil is my mother tongue ... the creativity will shine in
mother tongue only.” “This is not only work. It can improve our knowledge also”
![Page 25: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/25.jpg)
Language and cognition researchCultural transmission of grammatical
structure: Introducing a web-based iterated language learning paradigm with human participants◦ Jaeger, Tily, Frank, Gutman and Watts
Assessing the pragmatics of experiments with crowdsourcing: The case of scalar implicature◦ Anand, Andrews and Wagers
Collecting task-oriented dialogues◦ Clausen and Potts
Trivial Classification: What features do humans use for classification?◦ Boyd-Graber
Language Learning
Pragmatics
Dialogue
Intuition
![Page 26: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/26.jpg)
Language and cognition researchA crowdsourcing study of logical metonymy
◦Zarcone and PadoBalancing experimental lists without
sacrificing voluntary participation◦Watts and Jaeger
Creating illusory social connectivity in Amazon Mechanical Turk◦Duran and Dale,
A case study in effectively crowdsourcing long tasks with novel categories◦de Marneffe and Potts
Semantics
Balancing experiments
Controlling social context
Running large studies
![Page 27: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/27.jpg)
“Arm-chair” linguist
Circa1950-2000
![Page 28: The Start of the Art Introduction to the Workshop on Crowdsourcing Technologies for Language and Cognition Research Robert Munro and Hal Tily Stanford.](https://reader037.fdocuments.in/reader037/viewer/2022110213/56649e2f5503460f94b1fa42/html5/thumbnails/28.jpg)
“Arm-chair” linguist
Circa 2010+