Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE...
-
Upload
caroline-christine-dawson -
Category
Documents
-
view
220 -
download
1
Transcript of Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE...
![Page 1: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/1.jpg)
![Page 2: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/2.jpg)
Crowdsourcing Complexity:Lessons for Software Engineering
Lydia Chilton2 June 2014 ICSE Crowdsourcing Workshop
![Page 3: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/3.jpg)
Clarification: Human Computation
Mechanical Turk
Microtasks:
![Page 4: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/4.jpg)
2007: JavaScript Calculator
![Page 5: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/5.jpg)
2007: JavaScript Calculator
![Page 6: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/6.jpg)
Evolution of Complexityin Human Computation
Task Decomposition: Cascade & Frenzy
![Page 7: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/7.jpg)
Evolution of Complexity
![Page 8: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/8.jpg)
1. Collective Intelligence
1906: 787 aggregated votes averaged 1197 lbs. Actual answer: 1198 lbs.
![Page 9: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/9.jpg)
1. Collective Intelligence
Principles:– Small tasks– Intuitive tasks– Independent answers– Simple aggregation
Application: - ESP Game
![Page 10: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/10.jpg)
2. Iterative Workflows
Collective Intelligence
work improve vote improve vote
![Page 11: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/11.jpg)
2. Iterative Workflows
Principles:– Use fresh eyes– Vote to ensure
improvement
Application:- Bug finding
“given enough eyeballs, all bugs are shallow”
![Page 12: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/12.jpg)
3. Psychological Boundaries
![Page 13: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/13.jpg)
3. Psychological Boundaries
Applications:• Manager / programmer• Writer / editor• Write code / test code• Addition / subtraction
Principle:– Task switching is hard– Natural boundaries for tasks
![Page 14: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/14.jpg)
4. Task Decomposition
Legion:Scribe Real Time Audio Captioning on MTurk
![Page 15: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/15.jpg)
4. Task Decomposition
Principles:– Must be able to break apart tasks AND put them
back together.– Complex aggregation – Hint: Solve backwards. Find what people can do,
and build up from there.
![Page 16: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/16.jpg)
5. Worker Choice
Mobi: Trip Planning on Mturk with an open UI.
![Page 17: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/17.jpg)
5. Worker Choice
Applications:• Trip planning• Conference time table• Conference session-making
Principles:- Giving workers freedom relieves requesters’ burden of
task decomposition. - Workers feel more involved and empowered.- BUT complex interface that is difficult to scale.
![Page 18: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/18.jpg)
6. Learning and Doing
![Page 19: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/19.jpg)
6. Learning and Doing
Applications:• Peer assessment • Do grading assignments before you do your own
assignment• Task Feedback
Principles:- Teaching workers makes them better.- How long will they stay?
![Page 20: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/20.jpg)
Lessons for Software Engineering
1. Propose and vote 2. Find natural psychological
boundaries between tasks3. Find the tasks people can do,
then assemble them using complex aggregation techniques.
4. Teach.
221+ 473
-221+ 473
![Page 21: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/21.jpg)
Evolution of Complexityin Human Computation
Task Decomposition: Cascade & Frenzy
![Page 22: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/22.jpg)
Task decomposition is the key to crowdsourcing software engineering
![Page 23: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/23.jpg)
23
Lydia Chilton (UW), Greg Little (oDesk), Darren Edge (MSR Asia), Dan Weld (UW), James Landay (UW)
CascadeCrowdsourcing Taxonomy Creation
![Page 24: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/24.jpg)
24
Problem
![Page 25: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/25.jpg)
26
• 1000 eGovernment suggestions• 50 top product reviews• 100 employee feedback comments• 1000 answers to “Why did you decide to
major in Computer Science?”
Machines can’t analyze it
People don’t have time to analyze it1. time consuming2. overwhelming3. no right answer
![Page 26: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/26.jpg)
27
Solution
![Page 27: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/27.jpg)
28
Solution: Crowdsourced Taxonomies
![Page 28: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/28.jpg)
29
Toy Application: Colors
![Page 29: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/29.jpg)
30
Initial Prototypes
![Page 30: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/30.jpg)
31
Problems1. The hierarchy grows and
becomes overwhelming2. Workers have to decide
what to do
Lesson Break up the task more
Iterative Improvement
![Page 31: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/31.jpg)
32
Initial Approach 2:Category Comparison
ProblemWithout context, it’s hard to judge relationships• flying vs. flights• TSA liquids vs. removing liquids• Packing vs. what to bring
LessonDon’t compare abstractions to abstractionsInstead compare data to abstractions
![Page 32: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/32.jpg)
Use Lesson #3
Find the tasks people can do.Assemble them using complex
aggregation techniques.
Generate Labels Select Best LabelsCategorize
![Page 33: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/33.jpg)
34
Cascade Algorithm
Generate Labels Select Best Labels
Categorize
For a subset of items {good labels}
For all items, for all good labels,
Blue
Light Blue
Green
Greenish
Red
Gold
Then recurse
![Page 34: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/34.jpg)
35
Aggregate Data into TaxonomyBlue
Light Blue
Green
Greenish
Red
Gold
Blue
Light Blue
GreenBlue:
Light Blue:Green:Other:
redundant
nested
singletons
![Page 35: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/35.jpg)
36
Cascade Results: 100 Colors
![Page 36: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/36.jpg)
37
How can we get a global picture from workers who see only subsets of the data?
![Page 37: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/37.jpg)
Propose, Vote, Test1. Workers have good heuristics.2. Let them propose categories.3. Vote on categories to weed out bad ones.4. Test the heuristics by verifying it on data.
Propose Vote Test
![Page 38: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/38.jpg)
39
Lesson
Propose, Vote, Test.
![Page 39: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/39.jpg)
40
Deploy Cascade to Real Needs
1. CHI 2013 Program CommitteeOrganize 430 accepted papers to help session making
2. 40 CrowdCamp Hack-a-thon Participants Organize 100 hack-a-thon ideas to help organize teams
![Page 40: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/40.jpg)
41
Patina: Dynamic Heatmaps for Visualizing Application Usage',Effects of Visualization and Note-Taking on Sensemaking and Analysis',Contextifier: Automatic Generation of Messaged Visualizations',Interactive Horizon Graphs: Improving the Compact Visualization of Multiple Time Series',Quantity Estimation in Visualizations of Tagged Text',Motif Simplification: Improving Network Visualization Readability with Fan, Connector, and Clique Glyphs',Evaluation of Alternative Glyph Designs for Time Series Data in a Small Multiple Setting',Individual User Characteristics and Information Visualization: Connecting the Dots through Eye Tracking',"Without the Clutter of Unimportant Words": Descriptive Keyphrases for Text Visualization']],Direct Space-Time Trajectory Control for Visual Media EditingYour eyes will go out of the face: Adaptation for virtual eyes in video see-through HMDsSwifter: Improved Online Video ScrubbingDirect Manipulation Video Navigation in 3DNoteVideo: Facilitating Navigation of Blackboard-style Lecture VideosOwnership and Control of Point of View in Remote AssistanceEyeContext: Recognition of High-level Contextual Cues from Human Visual BehaviourYour eyes will go out of the face: Adaptation for virtual eyes in video see-through HMDsStill Looking: Investigating Seamless Gaze-supported Selection, Positioning, and Manipulation of Distant TargetsIndividual User Characteristics and Information Visualization: Connecting the Dots through Eye TrackingQuantity Estimation in Visualizations of Tagged Text
430 CHI Papers: Good Results, but…
• Visualization (19)• evaluating infovis (9)
• text (2)• video (6)• visualizing time data (5)• gaze (4)
• gaze tracking (3)• user requirements (3)• color schemes (2)
![Page 41: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/41.jpg)
42
“Don’t treat me like a Turker.”
“I just want to see all the data”
![Page 42: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/42.jpg)
43
Lesson
Authority and Responsibility should be aligned.
![Page 43: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/43.jpg)
44
Frenzy: Collaborative Data Organization for
Creating Conference Sessions
Lydia Chilton (UW), Juho Kim (MIT), Paul Andre (CMU), Felicia Cordeiro (UW), James Landay (Cornell?), Dan Weld (UW),
Steven Dow (CMU), Rob Miller (MIT), Haoqi Zhang (NW)
![Page 44: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/44.jpg)
45
![Page 45: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/45.jpg)
46
Groupware
Creating conference sessions is a social process. Grudin: Social process are often guided by personalities, tradition, convention. Challenge: support to the process without seeking to replace these behaviors. Challenge: remain flexible and do not improve rigid structures.
![Page 46: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/46.jpg)
47
DEMO
![Page 47: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/47.jpg)
48
Label
VoteCategorize
Light-weight contributions
![Page 48: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/48.jpg)
49
2-Stage Workflow
Stage 1 Stage 2
Set-up Collect Meta Data • 60 PC members• Low authority• Low responsibility
Session Making• 11 PC members• High authority• High responsibility
![Page 49: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/49.jpg)
50
Goals
Collect data: labels, votes Session-Making
![Page 50: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/50.jpg)
Results
Sessions created in record-setting 88 minutes.
![Page 51: Crowdsourcing Complexity: Lessons for Software Engineering Lydia Chilton 2 June 2014 ICSE Crowdsourcing Workshop.](https://reader035.fdocuments.in/reader035/viewer/2022062314/56649dac5503460f94a9c6c4/html5/thumbnails/51.jpg)
Lessons for Software Engineering
1. Propose and vote. 2. Find natural psychological
boundaries between tasks.3. Find the tasks people can do,
then assemble them using complex aggregation techniques.
4. Teach.5. Propose, vote, test.6. Align authority and
responsibility.
221+ 473
-221+ 473
Low Hi