TurKit: A Toolkit for Human Computation Algorithms
-
Upload
greg-little -
Category
Technology
-
view
1.563 -
download
10
Transcript of TurKit: A Toolkit for Human Computation Algorithms
![Page 1: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/1.jpg)
1
TurKit: A Toolkit for Human Computation Algorithms
Rob Miller & Greg LittleUser Interface Design Group
MIT CSAIL
Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein
June 10, 2010
![Page 2: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/2.jpg)
Outline
• Iterative human computation– TurKit: a toolkit for human
computation algorithms
• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word
June 10, 2010 2
![Page 3: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/3.jpg)
What is TurKit?
![Page 4: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/4.jpg)
2 Ways
What is TurKit?
![Page 5: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/5.jpg)
Way 1
![Page 6: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/6.jpg)
Way 1
ADD 1, 2
![Page 7: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/7.jpg)
Way 1
ADD 1, 2
JMP somewhere
![Page 8: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/8.jpg)
Way 1
ADD 1, 2
JMP somewhere…
![Page 9: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/9.jpg)
Way 1
ADD 1, 2
JMP somewhere…
TURK “How do you feel?”
![Page 10: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/10.jpg)
Way 1
ADD 1, 2
JMP somewhere…
TURK “How do you feel?”
![Page 11: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/11.jpg)
Way 2
![Page 12: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/12.jpg)
Way 2
![Page 13: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/13.jpg)
Way 2
![Page 14: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/14.jpg)
Way 2
![Page 15: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/15.jpg)
Way 2
![Page 16: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/16.jpg)
Way 2
![Page 17: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/17.jpg)
Way 2
![Page 18: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/18.jpg)
Way 2
![Page 19: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/19.jpg)
Way 2
![Page 20: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/20.jpg)
Way 2
![Page 21: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/21.jpg)
Demo
![Page 22: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/22.jpg)
Demo
TurKit
![Page 23: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/23.jpg)
What can you do with TurKit?
![Page 24: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/24.jpg)
x50
![Page 25: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/25.jpg)
31
19
![Page 26: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/26.jpg)
![Page 27: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/27.jpg)
![Page 28: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/28.jpg)
Improve
![Page 29: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/29.jpg)
Improve
![Page 30: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/30.jpg)
Improve
Vote
![Page 31: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/31.jpg)
Improve
Vote
![Page 32: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/32.jpg)
Handwriting Recognition
![Page 33: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/33.jpg)
Image Description
![Page 34: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/34.jpg)
Progression of Other Offer Description:• $10,000 more• better package• far more competitive offer• more competitive offer• competing offer
Outline to Prose
![Page 35: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/35.jpg)
Picture Sorting
![Page 36: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/36.jpg)
x
Subjective Ratings
![Page 37: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/37.jpg)
Subjective Ratings
![Page 38: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/38.jpg)
Subjective Ratings
![Page 39: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/39.jpg)
Workflow Comparisons
![Page 40: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/40.jpg)
Workflow Comparisons
![Page 41: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/41.jpg)
Workflow Comparisons
![Page 42: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/42.jpg)
Workflow Comparisons
![Page 43: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/43.jpg)
Workflow Comparisons
![Page 44: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/44.jpg)
Iterative
![Page 45: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/45.jpg)
Iterative
Parallel
![Page 46: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/46.jpg)
Iterative
Parallel
![Page 47: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/47.jpg)
Iterative
Parallel
![Page 48: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/48.jpg)
Which is better?
![Page 49: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/49.jpg)
Which is better?
• 30 images
![Page 50: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/50.jpg)
Which is better?
• 30 images• 6 iterations
![Page 51: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/51.jpg)
Which is better?
• 30 images• 6 iterations
![Page 52: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/52.jpg)
Which is better?
![Page 53: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/53.jpg)
Which is better?
![Page 54: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/54.jpg)
Which is better?
![Page 55: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/55.jpg)
Larger Algorithms
![Page 56: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/56.jpg)
Larger Algorithms
![Page 57: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/57.jpg)
Larger Algorithms
![Page 58: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/58.jpg)
Outline
• Iterative human computation– TurKit: a toolkit for human
computation algorithms
• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word
June 10, 2010 58
![Page 59: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/59.jpg)
VizWiz: Helping the Blind See
59
Which door is the women’s restroom?
joint work with Jeff Bigham (University of Rochester)
June 10, 2010
![Page 60: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/60.jpg)
Helping the Blind See
60
MechanicalTurk
1. the left one2. LEFT3. on the left
June 10, 2010
![Page 61: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/61.jpg)
Field Study
• Field deployment with 11 blind iPhone users– Answers received within a minute for ~5 cents a question– Latency can be reduced to less than 30 seconds by keeping
workers warmed up (at $4 per hour)
June 10, 2010 61
What color is this pillow? What denomination is this bill?
Do you see picnic tables across the parking lot?
What temperature is my oven set to?
Can you please tell me what this can is?
What kind of drink does this can hold?
(89s) I can’t tell.(105s) multiple shades of soft green, blue and gold
(24s) 20(29s) 20
(13s) no(46s) no
(69s) it looks like 425 degrees but the image is difficult to see.(84s) 400(122s) 450
(183s) chickpeas.(514s) beans(552s) Goya Beans
(91s) Energy(99s) no can in the picture(247s) energy drink
![Page 62: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/62.jpg)
All Mobile Users Are Situationally Disabled
62
When is Barack Obama speaking at MIT?
MechanicalTurk
joint work with Rajeev Najak
June 10, 2010
![Page 63: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/63.jpg)
Soylent: Putting a Crowd inside MS Word
June 10, 2010
joint work with Michael Bernstein
63
![Page 64: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/64.jpg)
Crowd-Driven Proofreading
June 10, 2010 64
![Page 65: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/65.jpg)
Crowd-Driven Shortening
June 10, 2010 65
![Page 66: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/66.jpg)
Bigger Idea: REAL Wizard of Oz
• Wizard of Oz is a tried-and-true prototyping technique in AI and HCI– Putting a human behind the curtain until we figure out how to
put software there
• Crowd computing enables Wizard of Oz systems that are useful and deployable– So we can start collecting data about how the system is
really used in practice– Adding the AI backend becomes a performance or cost
optimization
66June 10, 2010
![Page 67: TurKit: A Toolkit for Human Computation Algorithms](https://reader033.fdocuments.in/reader033/viewer/2022060111/55649416d8b42ab8278b48a6/html5/thumbnails/67.jpg)
Outline
• Iterative human computation– TurKit: a toolkit for human
computation algorithms
• Systems with human computation inside– VizWiz: vision for blind users with camera phones– Soylent: putting a crowd inside Microsoft Word
• Funded in part by
June 10, 2010 67