Wisdom and Folly of the Crowds
-
Upload
big-design-conference -
Category
Design
-
view
1.524 -
download
2
description
Transcript of Wisdom and Folly of the Crowds
![Page 1: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/1.jpg)
Wisdom (and Folly) of the CrowdsJessica Keup
![Page 2: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/2.jpg)
OutlineDefinitionsUsesPros/ConsCrowdsourcing MarketplacesCrowdsourcing Research
![Page 3: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/3.jpg)
History
“Crowdsourcing is a method of solving problems with, and outsourcing work, to the collective online intelligence”
James Surowiecki - The Wisdom of Crowds, 2005
![Page 4: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/4.jpg)
History
“Remember outsourcing? Sending jobs to India and China is so 2003. The new pool of cheap labor: everyday people using their spare cycles to create content, solve problems, even do corporate R & D.”
Jeff Howe - The Rise of Crowdsourcing, 2006
![Page 5: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/5.jpg)
Crowdsourcing vs . . .Outsourcing
◦work given to pre-decided person or group
Open Source◦initiated by public, not a client
RequestSubmission
s/Work
Reward
![Page 6: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/6.jpg)
Examples?Digg/Reddit iStockPhoto Netflix prizeThreadlessReCAPTCHAGoogle image labelerEquitySplash / Piqqem
![Page 7: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/7.jpg)
Best UsesImage taggingSearch relevance evaluationContent generationProduct categorizationCollecting feedback and reviewsTranscriptions
![Page 8: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/8.jpg)
ProsLabor at reduced costAround-the-clockHuge potential workforceVariety of opinions, ideas,
experienceConsensus and better answers
![Page 9: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/9.jpg)
ConsCheating/manipulation
Inexperience/ignorance
Confidentiality
![Page 10: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/10.jpg)
FixesCheating/manipulation
◦corroboration, ratingsInexperience/ignorance
◦qualification testsConfidentiality
◦?
![Page 11: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/11.jpg)
Crowdsourcing MarketplacesCrowdFlowerMechanical TurkCloud Crowd BootbcrowdSPRING Innocentive TopCoder Kluster
![Page 12: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/12.jpg)
mTurk “Artificial Artificial Intelligence” Human Intelligence Tasks (HITs) $0.00 and up, mostly a few cents SDKs in Java, Ruby, Perl, .NET Or, create through website UI*
![Page 13: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/13.jpg)
Training AI“Computer Music Composition
using Crowdsourcing and Genetic Algorithms”
DarwinTunes
![Page 14: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/14.jpg)
![Page 15: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/15.jpg)
![Page 16: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/16.jpg)
Research Examples collecting creative drawings rating audio and video sample
quality tagging location-sensitive queries
and points of interest with mobile devices
remote massage task delegation in Wikipedia collecting volunteer expertise to
help the homeless annotating political campaign ads
![Page 17: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/17.jpg)
More Research examples describing and organizing
geometric shapes* tracking popular music rating search result relevance collecting user feedback extracting product and brand
information tagging specific objects in an
image training speech recognition
software
![Page 18: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/18.jpg)
Questions?
![Page 19: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/19.jpg)
References Alonso, O., Rose, D.E., and Stewart, B. (2008). Crowdsourcing for Relevance
Evaluation. SIGIR Forum 42(2) 9-15. Amazon.com, Inc. (2010). Amazon Mechanical Turk.
https://www.mturk.com/mturk/welcome. CastingWords (2010). Audio transcription services: plus video, CDs, podcasts and
more. http://castingwords.com/ Chung, K., Chiu, C., Xiao, X., & Chi, P. (2009). Stress outsourced: A haptic social
network via crowdsourcing. Paper presented at the CHI EA '09: Proceedings of the 27th International Conference Extended Abstracts on Human Factors in Computing Systems, Boston, MA, USA. 2439-2448.
DarwinTunes (2010). Welcome to DarwinTunes. http://darwintunes.org/ Ganjisaffar, Y., Javanmardi, S., & Lopes, C. (2009). Leveraging crowdsourcing
heuristics to improve search in wikipedia. Paper presented at the WikiSym '09: Proceedings of the 5th International Symposium on Wikis and Open Collaboration, Orlando, Florida. 1-2.
Harper, F. M., Raban, D., Rafaeli, S. & Konstan, J. A. (2008). Predictors of answer quality in online Q & A sites. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 865-874.
Hintikka, K. A. (2008). Web 2.0 and the collective intelligence. Paper presented at the MindTrek '08: Proceedings of the 12th International Conference on Entertainment and Media in the Ubiquitous Era, Tampere, Finland. 163-166.
Howe, J. (2006). The Rise of Crowdsourcing. Wired, 14(6). http://www.wired.com/wired/archive/14.06/crowds.html.
![Page 20: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/20.jpg)
More References Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A
study of annotation selection criteria. Paper presented at the HLT '09: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, Boulder, Colorado. 27-35.
Jagadeesan, A. P., Lynn, A., Corney, J. R., Yan, X. T., Wenzel, J., Sherlock, A., et al. (2009). Geometric reasoning via internet CrowdSourcing. Paper presented at the SPM '09: 2009 SIAM/ACM Joint Conference on Geometric and Physical Modeling, San Francisco, California. 313-318.
Kittur, A., Chi, ,E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 453-456.
Koblin, A. M. (2009). The sheep market. Paper presented at the C&C '09: Proceeding of the Seventh ACM Conference on Creativity and Cognition, Berkeley, California, USA. 451-452.
Krieger, M., Stark, E. M., & Klemmer, S. R. (2009). Coordinating tasks on the commons: Designing for personal goals, expertise and serendipity. Paper presented at the CHI '09: Proceedings of the 27th International Conference on Human Factors in Computing Systems, Boston, MA, USA. 1485-1494.
Ledlie, J., Odero, B., Minkov, E., Kiss, I., & Polifroni, J. (2009). Crowd translator: On building localized speech recognizers through micropayments. SIGOPS Operating.Systems Review, 43(4), 84-89.
Li, C., Buyuktur, A. G., Hutchful, D. K., Sant, N. B., & Nainwal, S. K. (2008). Portalis: Using competitive online interactions to support aid initiatives for the homeless. Paper presented at the CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy. 3873-3878.
![Page 21: Wisdom and Folly of the Crowds](https://reader033.fdocuments.in/reader033/viewer/2022051014/54be2db64a7959ee018b469f/html5/thumbnails/21.jpg)
Even More References Mannes, A. E. (2009). Are we wise about the wisdom of crowds? the use of group
judgments in belief revision. Management Science, 55(8), 1267-1279. Mason, W., & Watts, D. J. (2009). Financial incentives and the performance of
crowds. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France. 77-85.
Milne, D., & Witten, I. H. (2008). Learning to link with wikipedia. Paper presented at the CIKM '08: Proceedings of the 17th ACM Conference on Information and Knowledge Management, Napa Valley, California, USA. 509-518.
Roman, D. (2009). Crowdsourcing and the question of expertise. Commnuications of the.ACM, 52(12), 12-12.
Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the enterprise. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France. 50-53.
Su, Q., Pavlov, D., Chow, J., & Baker, W. C. (2007). Internet-scale collection of human-reviewed data. Paper presented at the WWW '07: Proceedings of the 16th International Conference on World Wide Web, Banff, Alberta, Canada. 231-240.
Surowiecki, J. (2005). The wisdom of crowds. Anchor. Tanaka, A., Tokui, N., & Momeni, A. (2005). Facilitating collective musical
creativity. Paper presented at the MULTIMEDIA '05: Proceedings of the 13th Annual ACM International Conference on Multimedia, Hilton, Singapore. 191-198.
Von Ahn L., & Dabbish, L. (2004). Labeling images with a computer game. Paper presented at the CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria. 319-326.