Slide 1

Stanford HCI Group Adobe Advanced Technology Labs Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code Joel Brandt with Philip Guo (S), Joel Leweinstein (S), Mira Dontcheva (A), Scott Klemmer (S), http://hci.stanford.edu/opportunistic

Slide 2

2 good grief, I dont even remember the syntax for forms!

Slide 3

3

Slide 4

4

Slide 5

5

Slide 6

6

Slide 7

7

Slide 8

8

Slide 9

9

Slide 10

10

Slide 11

11

Slide 12

12

Slide 13

13

Slide 14

14

Slide 15

The Web has fundamentally changed the cost structure of information access. Enables an opportunistic approach to programming 15

Slide 16

People program opportunistically to prototype, ideate, and discover Emphasize speed and ease of development over code robustness and maintainability 16

Slide 17

http://flickr.com/photos/royalsapien/2387707860/ 17 Designers

Slide 18

http://flickr.com/photos/niosh/2492023651/ 18 Scientists

Slide 19

http://www.flickr.com/photos/samthor/3198975900/ 19 Makers

Slide 20

By 2012: 13 million who program as part of their job but only 3 million who are professional programmers [Scaffidi 2005] 20

Slide 21

How do programmers use online resources? 21

Slide 22

Related Work Web Search for Programmers [Stylos 06, Hoffmann 07] Code Cloning in Software Engineering [Kim 04, Ducasse 99] Learning Barriers of Programming [Ko 04] Learning on the Web [Torrey 09] 22

Slide 23

23 Study 1: Prototyping a Web-based chat room Study 2: Log Analysis of Programmers Web Queries

Slide 24

Study 1: Web Chat Room 20 participants (9 Graduate, 11 Undergrad) basic knowledge of PHP, JS, and HTML 2.5 hours each use any resources 3 researchers observed, one asked questions to prompt think aloud 24 M ETHOD

Slide 25

Chat Room Specifications 1.Set username 2.Post messages 3.Update without page reload 4.Messages have timestamp 5.Limited chat history 25 M ETHOD

Slide 26

26

Slide 27

Most Subjects Met All Specs 27 username post message AJAX update timestamphistory Specification Met Specification Attempted subjects R ESULTS

Slide 28

Frequent Web Use 28 subjects time (minutes) R ESULTS

Slide 29

Frequent Web Use 29 subjects time (minutes) R ESULTS

Slide 30

Frequent Web Use 30 subjects time (minutes) R ESULTS

Slide 31

Frequent Web Use 31 subjects time (minutes) R ESULTS

Slide 32

The Web plays multiple roles 32 R ESULTS length (seconds) session (sorted by length)

Slide 33

The Web plays multiple roles 33 R ESULTS length (seconds) session (sorted by length)

Slide 34

The Web plays multiple roles 34 R ESULTS length (seconds) session (sorted by length)

Slide 35

Three Intentions Behind Web Use Learning just-in-time acquisition of new skills Clarification connecting high-level knowledge to implementation details Reminder offloading memory to external resources 35 R ESULTS

Slide 36

Just-in-Time Learning-by- Doing Copy and modify code before reading prose 36 R ESULTS

Slide 37

Theres some stuff in this code that I dont really know what its doing, but Ill just try it and see what happens. 37

Slide 38

Just-in-Time Learning-by- Doing Copy and modify code before reading prose Not concerned with deep learning 38 R ESULTS

Slide 39

I think its less expensive for me to just take the first code I find and see how helpful it is at a very high level as opposed to just reading all these descriptions and text 39

Slide 40

Just-in-Time Learning-by- Doing Copy and modify code before reading prose Not concerned with deep learning Learn new terminology from result snippets 40 R ESULTS

Slide 41

Example Query Terms Time Number of Refinements Number of Result Clicks Types of pages visited Lines of code copied 41

Slide 42

Learning Example Query Terms Time Number of Refinements Number of Result Clicks Types of pages visited Lines of code copied 42

Slide 43

Learning Example Query ajax tutorial Terms Time Number of Refinements Number of Result Clicks Types of pages visited Lines of code copied 43

Slide 44

Learning Example Query ajax tutorial TermsEnglish Time Number of Refinements Number of Result Clicks Types of pages visited Lines of code copied 44

Slide 45

Learning Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements Number of Result Clicks Types of pages visited Lines of code copied 45

Slide 46

Learning Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks Types of pages visited Lines of code copied 46

Slide 47

Learning Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited Lines of code copied 47

Slide 48

Learning Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied 48

Slide 49

Learning Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 49

Slide 50

Clarification of Existing Knowledge Web search enables translations and language analogies Know exactly what code does when they see it Copied code often not immediately tested 50 R ESULTS

Slide 51

Learning Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 51

Slide 52

LearningClarification Example Query ajax tutorial TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 52

Slide 53

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 53

Slide 54

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutes Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 54

Slide 55

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-3 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 55

Slide 56

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+ Types of pages visited tutorials, how-to articles Lines of code copied dozens 56

Slide 57

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles Lines of code copied dozens 57

Slide 58

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozens 58

Slide 59

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 59

Slide 60

Reminding about forgotten details Explicitly choose not to learn things Search serves as Web-enabled auto-complete 60 R ESULTS

Slide 61

61

Slide 62

62

Slide 63

Reminding about forgotten details Explicitly choose not to learn things Search serves as Web-enabled auto-complete Despite utility, can break flow 63 R ESULTS

Slide 64

LearningClarification Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 64

Slide 65

LearningClarificationReminder Example Query ajax tutorial javascript thread TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 65

Slide 66

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 66

Slide 67

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 67

Slide 68

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute less than 1 minute Number of Refinements 2-30-1 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 68

Slide 69

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute less than 1 minute Number of Refinements 2-30-10 Number of Result Clicks 5+1-2 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 69

Slide 70

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute less than 1 minute Number of Refinements 2-30-10 Number of Result Clicks 5+1-20-1 Types of pages visited tutorials, how-to articles API docs, blogs, forums Lines of code copied dozensseveral 70

Slide 71

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute less than 1 minute Number of Refinements 2-30-10 Number of Result Clicks 5+1-20-1 Types of pages visited tutorials, how-to articles API docs, blogs, forums API docs, result snippets Lines of code copied dozensseveral 71

Slide 72

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute less than 1 minute Number of Refinements 2-30-10 Number of Result Clicks 5+1-20-1 Types of pages visited tutorials, how-to articles API docs, blogs, forums API docs, result snippets Lines of code copied dozensseveralnone to several 72

Slide 73

How do these results generalize? 73

Slide 74

Study 2: Query Log Analysis Adobe Community Help logs from July 2008 ~25,000 programmers performing 101,289 queries about Adobe Flex Framework Grouped queries into 65,955 search sessions Analyzed three properties of sessions: Format of query strings Process of query refinement Types of Web pages visited Hand-coded 300 sessions for intent 74 M ETHOD

Slide 75

Query format predicts types of pages visited 75

Slide 76

Query format predicts types of pages visited Page typecode onlyEnglish + codeEnglish only Adobe API s 38%16%10% Adobe tutorials 31%33%39% blogs 15%22%19% forums 3%7%6% unclassified 13%22%27% 76 All inter-row differences significant at p < 0.001 R ESULTS

Slide 77

Query format predicts types of pages visited Page typecode onlyEnglish + codeEnglish only Adobe API s 38%16%10% Adobe tutorials 31%33%39% blogs 15%22%19% forums 3%7%6% unclassified 13%22%27% 77 R ESULTS All inter-row differences significant at p < 0.001

Slide 78

Query format predicts types of pages visited Page typecode onlyEnglish + codeEnglish only Adobe API s 38%16%10% Adobe tutorials 31%33%39% blogs 15%22%19% forums 3%7%6% unclassified 13%22%27% 78 R ESULTS All inter-row differences significant at p < 0.001

Slide 79

Query format predicts types of pages visited Page typecode onlyEnglish + codeEnglish only Adobe API s 38%16%10% Adobe tutorials 31%33%39% blogs 15%22%19% forums 3%7%6% unclassified 13%22%27% 79 R ESULTS All inter-row differences significant at p < 0.001

Slide 80

Reminder sessions are typically code-only queries 80

Slide 81

Query format indicative of intention query formatremindinglearning code only 56%21% English + code 10%29% English only 34%50% 81 R ESULTS All inter-row differences significant at p < 0.05

Slide 82

Programmers rarely need to refine queries 82

Slide 83

Programmers refine rarely 83 R ESULTS number of queries query number in session

Slide 84

and they start out with either words or code 84 R ESULTS percent of queries query number in session

Slide 85

Five Key Insights for Tool Design 85

Slide 86

Just-in-time learning is common Create tutorials that teach how the code works 86

Slide 87

Copied code often not immediately tested Demarcate copied code, highlight what is executed 87

Slide 88

Web search often used as a translator Improve search ranking with synonyms, perform searches automatically 88

Slide 89

Programmers choose not to learn syntax Make Web-enabled auto-complete 89

Slide 90

Query refinement is often predictable Automatically augment queries 90

Slide 91

Preview of Ongoing Work 91 Blueprint Enabling Example-Centric Programming

Slide 92

92

Slide 93

Slide 94

Stanford HCI Group Adobe Advanced Technology Labs http://hci.stanford.edu/opportunistic

Slide 95

95

Slide 96

LearningClarificationReminder Example Query ajax tutorial javascript thread fetch_array php TermsEnglish mix of code and English mostly code Time10s of minutesabout 1 minute less than 1 minute Number of Refinements 2-30-10 Number of Result Clicks 5+1-20-1 Types of pages visited tutorials, how-to articles API docs, blogs, forums API docs, result snippets Lines of code copied dozensseveralnone to several 96