GoHaskell! Mobile Application Contextual Inquiry and Analysis

Group G: Tatiana Cooper, Lody Isaac, Elena McIntosh, Laura Morgan, Jason Smith

Table of Contents

Product Concept Statement Tailoring the Scope Front­end Analysis

Meeting with Aaron Kent, Co­Founder of Isomorf Meeting with Aaron Webb, Software Developer Summary

Contextual Inquiry Process Overview Interviews

Isomorf Interview Sample Interview Questions

Potential End­User Interviews Sample Interview Questions

Observations Isomorf Observation Potential End­User Observations Work Artifacts

Survey Responses Competitive Analysis: Website Evaluations Results

Contextual Analysis Data Collection Protocols WAAD Development Process Final WAAD

Data Analysis Competitive Analysis Matrix

Flow Models Isomorf’s Programming Education Flow Model Learning to Program Flow Model

Appendix Appendix A: Interview Script/Questions and Observation Prompt for Isomorf Appendix B: Interview Script/Questions and Observation Prompt for non­Haskell Programmers Appendix C: Consent Form Appendix D: Online Survey (Google Forms) Appendix E: Website Evaluation Criteria Appendix F: Raw Data Collected

1

Product Concept Statement The GoHaskell! mobile application will provide on­the­go learning for experienced imperative language (C++, Java, Python) programmers who need to learn the functional language Haskell quickly. The design will incorporate short lessons/modules in the “microlearning model” that can be taken separately or all at once with flexibility in accommodating various learning on­the­go timeframes. Compared to traditional classroom lessons and current online lessons, this product will provide the convenience of the mobile platform and 24/7 access to lessons, practice and immediate feedback. Unlike conventional programming practice, this product will not require the learner to download or install additional software for testing or running practice code. The innovative design will improve job productivity, confidence and satisfaction for people who desire to learn to code using Haskell. The GoHaskell! mobile application will be developed in collaboration with Isomorf, a Washington, D.C.­based technology startup company. Isomorf will provide Haskell subject matter expertise and serve as a model for the growing group of companies that require employees to understand and use functional programming.

Tailoring the Scope Based on the responses from the surveys, interviews and observations we conducted, we have determined that our project scope will need to be narrowed. Rather than attempt to create an all­purpose “learn to code” application for a diverse audience, we will narrow our intended audience to people who currently know how to program in imperative languages. Because we will no longer be creating an application for first time programmers, we will be able to focus in on content that experienced programmers are already familiar with. Our content will focus on the aspects of programming in Haskell that are different from programming in other languages. For the prototype version, we will not cover all of the aspects unique to coding in Haskell, rather we will focus only on a few concepts such as “monads,” “types and typeclasses,” “higher­order functions” and “referential transparency.” The application will emphasize the advantages of this pure functional programming language in order to persuade more programmers to learn to code in Haskell.

Front­end Analysis

Meeting with Aaron Kent, Co­Founder of Isomorf The design team met with Aaron Kent early in the application development process to discuss his views on programming, including which languages he believes to be most useful to current

2

and future programmers. Aaron K. is the co­founder of our partner company Isomorf, which builds all of its systems using functional programming languages and concepts. He was formerly a director at an energy trading firm where he wrote programs to analyze complex data environments. He also has experience training employees on how to code in new languages. Aaron K. explained that there are two ways to classify programming languages: interpreted vs. compiled and imperative vs. functional. He believes that differentiating between imperative and functional is the most interesting because of the clear benefits and drawbacks of each. He explained that programming with imperative languages (such as C++, Java, Perl, PHP, Python, and Ruby) leads to “inter­related” code in which every line of the program is mapped to every other line. This leads to exponential growth in complexity. However, programming with functional languages (such as Haskell) creates code that is “self­contained.” Every function is independently meaningful, which leads to linear­­and therefore more manageable­­growth in complexity. This also leads to code that is easier to share and reuse, which Aaron anecdotally cited as a frequent desire of many programming professionals. He warned that functional programming is currently not a popular way to code, but is possibly the future of coding. He cited Facebook writing their Sigma security program in Haskell as a great example. He also mentioned his own company: the Isomorf programming platform would not be possible without functional programming concepts. When asked about potential audience, Aaron K. explained that it would be challenging to learn Haskell without prior experience with imperative programming.

Meeting with Aaron Webb, Software Developer Based on our group’s meeting with Aaron Kent, we met with another programming professional Aaron Webb to cross­reference our perspective on functional programming. (He requested that he not disclose his place of employment.) Before the meeting, we conducted independent research online to better inform ourselves about programming concepts and jargon. Aaron Webb has over 30 years of experience as a programmer and has coded using both imperative and functional paradigms. He has learned and worked with Haskell recently. He explained that using a functional programming language is “radically different” in comparison with imperative languages. He emphasized the need to understand the foundational concepts of functional programming, not just the syntax of a single language. He concurred that learners of Haskell would likely already know to code with imperative languages.

Summary Both meetings with Aaron Kent and Aaron Webb contributed to the design team’s understanding of the “big picture” of functional programming and Haskell, as well as our

3

understanding of the GoHaskell! app’s potential end­users. These perspectives helped us to structure our contextual inquiry plan.

Contextual Inquiry The design team determined that our partner company Isomorf is representative of an organization that requires employees to use functional programming. An investigation of Isomorf, then, would reveal the context of using Haskell in the workplace. Because GoHaskell! will be a commercial product, the design team needed to go beyond inquiry of our partner company. We also investigated the context of learning to program so that we understand both our competitive space and the experiences and preferences of our potential end­users. To capture this rich context, we planned and conducted interviews, observations, online surveys, and competitive analyses of two coding tutorial websites and a microlearning mobile application:

1 Interview with Isomorf co­founders 1 Observation with Isomorf co­founders 7 Interviews with potential end­users 2 Observations with potential end­users 2 Online Survey Responses from potential end­users 3 Competitive Analyses

We implemented several Interview techniques from Convivial Toolbox (Sanders & Stappers, 2012) including the use of ambiguity, association, context mapping, experience and service design, day­in­the­life and survey questions.

Process Overview 1. Identified potential end­user characteristics based on front­end analysis:

Experience with imperative programming Experience with functional programming, but not with Haskell

2. Identified tutorial websites and mobile app to evaluate as competitors: CodeAcademy (coding tutorial) Khan Academy (coding tutorial) Duolingo (microlearning app)

3. Developed interview script and observation prompt for Isomorf co­founders 4. Developed interview script and observation prompt for potential end­users 5. Developed online survey questions for potential end­users 6. Developed website and app evaluation criteria

4

7. Contacted Isomorf co­founders and potential end­users and arranged appointments 8. Finalized interview script and observation prompts for Isomorf co­founders (Appendix A) 9. Finalized interview script for potential end­users based on SME input (Appendix B) 10. Finalized Consent Form (Appendix C) 11. Finalized online survey based on SME input (Appendix D) 12. Finalized website evaluation criteria (Appendix E) 13. Sent online survey to potential end users 14. Conducted Isomorf interview and observation 15. Conducted potential end­user interviews and observations 16. Conducted website evaluations

Interviews

Isomorf Interview During our interview with the Isomorf co­founders, we wanted to know about how functional programming was being used in the private sector.

Sample Interview Questions Isomorf Purpose and Structure

1. Could you tell me a bit about Isomorf? (If needed: Why did you found it? What are you building? What’s your message?)

2. Can you explain to me why functional programming is so important to your company? 3. Once you begin to grow, what do you expect the structure of your company to look like?

a. What teams might you have and for what purpose? b. Who will need to understand functional programming?

4. Who do you envision as your future users? In 2 years? In 10 years? a. Do your users need to understand functional programming? b. What language will users code in?

Learning Functional Programming

5. Can you walk me through how you learned to code using functional programming? (If needed: Did you use any of the following resources? Formal classroom environments, formal documentation, free or purchased online programs, online forums, colleagues, on­the­job training, mobile app, book or textbook?)

6. Do your future employees need to know functional programming to be hired? a. If not, do you plan to teach them about functional programming? b. Which resources would you recommend to them?

7. How long do you think it would take someone to learn Haskell with no prior experience with functional programming?

5

Role Names and Positions

Date and Time Team Members Present

Client Stakeholders/ Functional Programming SMEs

Aaron Kent, CEO and Co­Founder of Isomorf Brian Kent, CTO and Co­Founder of Isomorf

5 OCT 2015, 1500­1600

Laura Morgan

Interview Summary: Aaron and Brian described the purpose of Isomorf: to make software development easier and faster. The platform they are developing would be impossible to create without functional programming because of its ability to analyze code as data. This makes code reusable and shareable. They expect all future employees to at least be able to understand and explain why functional programming is important to the company, even the accounting team, for example. They also described other teams that will be essential to their company. They would ideally form a learning team responsible for developing training materials and answering questions on online forums. They do not expect their users to know functional programming, but users could use the site more easily if they did. Both Aaron and Brian agreed to the importance of realistic, perhaps on­the­job practice when learning functional programming, but would prefer if developers already knew some basics. They both learned functional programming skills by solving real problems at Isomorf. They referenced articles, blogs, research papers, forums, and presentations from universities.

Potential End­User Interviews During interviews with our potential end­users, we wanted to know about their experiences in learning to program. Based on our front­end analysis, we knew we needed to talk to people with experience in imperative programming. The team reached out for interviews to current and former professional colleagues, as well as personal acquaintances. We determined that seven potential end­users’ viewpoints would provide us with sufficiently diverse data points for the scope of the project.

Sample Interview Questions 1. Can you tell me how many years of experience you have in computer programming? 2. Can you tell me some of the feelings that you associate with programming?

a. What aspects of it do you like? (can you give me an example?) b. What aspects of it do you NOT like? (can you give me an example?)

3. Can you remember a day in the last month in which you had to write code? Walk me through that day (were you at work? who did you communicate with? etc.). (Try to pick up on the high points/low points and inquire about them.)

4. Have you used any of the following resources to learn to program (I’ll read one at a time)?

6

a. Formal classroom environments (high school, college, grad school) i. Which?

b. Formal documentation of the language (online) c. Free online programs (CodeAcademy, Khan Academy, etc.)

i. Which one(s)? d. Online programs that you paid for (Lynda.com, etc.)

i. Which one(s)? e. Online forums (StackOverflow, GitHub, etc.)

i. Which one(s)? f. People you know (friends and family)

i. How did you learn together? g. On­the­job training at work

i. What was the nature of your on­the­job training? h. A mobile app

i. Which one(s)? i. A book or textbook

5. What are some of the things that you liked about learning with these resources? 6. What are some of the things that you did NOT like about learning with these resources? 7. I’m particularly interested in finding out about people’s memories of their experiences,

especially stories that people tell about their experiences. Can you remember the last time you learned a new programming language? Walk me through the steps. (Try to pick up on the high points/low points and inquire about them.)

8. What helps you to retain information when learning programming skills? 9. What causes you to forget information about programming? What do you do when you

forget newly learned information about programming? 10. In order to learn a new programming language, I will need ____________________. 11. The thing I find hardest about learning a new language is_______________________. 12. Have you ever used a mobile app to learn anything?

a. What did you learn? Name some apps if you can. b. What was it like? What features did it have that you liked/disliked?

13. Have you heard of the programming language Haskell? If so, what sorts of ideas and feelings do you associate with it?

Role Names and Positions

Date and Time Team Members Present

Potential End­User Michael Cobb, IT Director at Inter­American Defense College

2 OCT 2015, 1600­1700

Laura Morgan

Interview Summary: Mike has over 30 years of experience as a programmer, but has not heard of Haskell. He described programming as frustrating, but very satisfying because of

7

problem­solving. He said that using books was his preferred method to learn new languages, but admitted that this has changed over the years. He now uses online forums and web­based tutorials more. For Mike, face­to­face classes are good because there is an expert in the room, but are also generally either too fast or slow­paced. He does a lot of copying and pasting of other people’s code, although this needs to be edited. He stressed the need of repeated hands­on practice in order to learn to code. He mentioned his enjoyment of competition­based and time­based (Learn X in 21 Days) learning activities, but recognized that it is difficult to stick with it after these are finished. The one learning app he mentioned was Duolingo. He loved the repetition and frequency.

Potential End­User Marshall Koch, Systems Engineer at MITRE

3 OCT 2015, 1200­1300

Laura Morgan

Interview Summary: Marshall has 10 years of imperative programming experience. He is aware of Haskell, but fears that learning it would require him to return to step one in learning to code because it is so different. He does recognize the benefits of functional programming to be purity, speed, and less need for debugging. He described programming as fun and satisfying because of accomplishing goals and engaging in “battle of wits.” He learned to code in formal classroom environments using books, but has done most on­the­job training via problem­solving and referencing online forums and formal documentation. He liked that classroom training was more conceptual, but did not like the forced pace. He finds online forums to be useful “in the moment,” but oftentimes ends up figuring out better solutions on his own. He stressed the need to apply learning code to a real problem. He suggested that an app like Duolingo could be good for maintaining his knowledge. He also mentioned that he enjoys games and competitions.

Potential End­User Andrew Obrien, Senior Developer at Bloomberg Government

5 OCT 2015, 1100­1200 7 OCT 2015, 1300­1400

Elena McIntosh

Interview Summary: Andrew has some Haskell experience and would be very interested in coding in Haskell, but feels the language is too unknown for him to advertise his services as a Haskell developer. The streamlined nature of this functional programming language has helped him become a better developer in other programming languages. He emphasized that problem­solving and developing solutions is his main goal which he can achieve across languages. He would prefer to use the language best suited for the solution to the programming problem at hand. He does not typically learn on mobile devices, but he can see how we could use a cloud­based remote server for our application as he has used his iPad to access a remote server to run code for projects he has worked on in the past. His preferred methods of learning are research in books or online forums combined with mentoring from an expert in that programming language and working through realistic problems.

Potential End­User Landon Gibbs, Senior IT Specialist and Program

25 SEP 2015, 1500­1540

Tatiana Cooper

8

Manager at XYZ (requested privacy)

Interview Summary: Landon is an experienced imperative programmer. He frequently associated coding with “solving problems.” As relatively new to Haskell, Landon was skeptical about the idea of learning to code on an app. He preferred not to master new languages “on the go,” especially an advanced one like Haskell. But taking into consideration the fact that he has never attempted to learn code this way before, he was considering trying it out to see it it fits his learning style. According to Landon, the success of the Haskell app would depend on how well the modules were designed.

Potential End­User Serghey Prytkov, Software Developer at XYZ (requested privacy)

29 SEP 2015, 1900­1945

Tatiana Cooper

Interview Summary: Serghey has 10 years of experience with imperative programming. He has worked in both Russia and the U.S. Serghey was familiar with Haskell and described functional programming as used for “expanding your mind” and “solving problems outside of the box”. He learned to program the “old school” way with books and a computer, which is still his preferred method. However, he would be open to using an app although it might not fit his “learning style.” In other words, the app would be something to get used to.

Potential End­User Pranav R., IT Support and Software Developer at XYZ (requested privacy)

4 OCT 2015, 1830­1915

Tatiana Cooper

Interview Summary: Pranav has 5 years of experience with imperative programming and considers himself “a newcomer.” He has heard of Haskell and believes that it was created for “future research in functional­language design,” although he also acknowledges its practical use. Pranav prefers learning by solving problems. He also mentioned that he likes “on the go” style instead of spending all of his time in front of a computer.

Potential End­User John Frye, IT Director at FAA

5 OCT 2015, 1500­1600

Lody Isaac

Interview Summary: Mr. Frye was interviewed with the use of interview questions developed by the team. He consented to have the observation portion of the field visit video and audio recorded and signed the consent form. Mr. Frye has 30 years of experience in programming. He is an IT Director for the FAA. He has never programmed in Haskell and had never heard of it prior to the interview. Mr. Frye likes programming because of the problem­solving aspect. His least favorite aspect of programming is forgetting what he has learned. His prefers books for learning programming and uses a Kindle to read them. Applying what he learns is most helpful to him in retaining information. His biggest hurdle in learning new languages is time. He thinks the most time­consuming part of learning new languages is the integrated development environment (when required).

9

Observations

Isomorf Observation Because Isomorf is a small startup consisting of only two employees, we could not literally observe the interactions between teams. We obtained information about what team interactions could look like in the future from the Isomorf interview. We also foresaw difficulty in the design team comprehending a “think aloud” programming observation because of the high skill level required. Instead, we asked the Isomorf team to perform the task of teaching functional programming. We used the following observational scenario script: I’d like you to pretend that you have just hired an intern who is unfamiliar with functional programming. For the next 30 minutes, please show me how you would begin to explain and demonstrate functional programming to your intern. You can mention basic concepts, common errors, etc. You can use your computer if you’d like.

Role Names and Positions

Date and Time Team Members Present

Client Stakeholders/ Functional Programming SMEs

Aaron Kent, CEO and Co­Founder of Isomorf Brian Kent, CTO and Co­Founder of Isomorf

5 OCT 2015, 1330­1500

Laura Morgan

Observation Summary: Aaron and Brian first focused on explaining functional programming through the lens of imperative programming. It seemed like this was a natural approach. For example, in imperative programming, you are allowed to make errors, but functional programming does not let you make errors. Also, in imperative programming, “everything is a string (words),” but with functional programming, you must define what everything is. Aaron explained that with functional programming, you must define everything in your universe and slowly refine it over time. A key concept here is types. He said that “helping someone define types would be very useful” for a learning experience. Brian gave an example using bears at a zoo, but backtracked saying that more realistic examples would be better. They both agreed that modeling is another key concept with functional programming. Modeling is “how to bucket things into different ideas.” Brian said he would approach this one of two ways: either start with a model and ask the intern to figure something out based on the model; or start with the data and ask the intern to make a type out of it. For example, a restaurant owner wants to know if they have a table available at a certain time, or how many cans of tomatoes he should order. There could be given assumptions like “a server can handle five tables.” Both Aaron and Brian agreed that realistic problem­solving was a good idea. Finally, they mentioned the key concept recursion, which involves anything that references itself. They warned that this is the “mathy” part of functional programming and probably requires background training.

10

Potential End­User Observations For the potential end­user observations, we wanted to see how programmers would approach learning a new language, specifically Haskell. We used the following observational scenario script: I’d like you to pretend that you have just been given an assignment that requires you to code in Haskell. For the next 10 minutes, please show me how you would begin to learn about Haskell. You can use this computer. Also, I’d like you to think aloud as you do it. In other words, try to verbalize your reasoning behind the decisions you make. Questions to ask as they perform the activity:

1. What kinds of resources do you use/prefer to use? (videos, forums, books, etc.?) Why? 2. Are you learning anything surprising or interesting about Haskell? 3. How do you feel as you’re learning about Haskell? 4. How long do you think it would take you to learn Haskell?

Role Names and Positions

Date and Time Team Members Present

Potential End­User Michael Cobb, IT Director at Inter­American Defense College

2 OCT 2015, 1600­1700

Laura Morgan

Observation Summary: Mike explained that his first step for learning anything new about programming would be to Google it, which he did. He generally uses the top Google sites because he figures they are the best, but he avoids Wikipedia. He also mentioned that “YouTube is always good.” Mike encountered and began to use a free tutorial that did not require him to download anything new. He liked being able to do something right away like display “Hello World,” but later contradicted himself by saying that he prefers to start with the big picture instead of diving right in with commands. He advised that it would be a good idea to compare functional programming with imperative programming. If he had more time, he would have gone to the official Haskell website next. At the end of the observation, he said he got an idea for functional programming, but realized that it would be very different from other languages he has used in the past.

Potential End­User John Frye, IT Director at FAA

5 OCT 2015, 1500­1600

Lody Isaac

Observation Summary: During the observation portion of the field visit John Frye demonstrated and explained his preferred methods of finding programming learning resources and which learning resources he preferred. Mr. Frye demonstrated using his go­to method of organizing information, including learning resources: OneNote. OneNote is an electronic notebook application. When finding resources on the internet, Mr. Frye is

11

concerned with viruses and expressed that he always checks resources before opening them. Mr. Frye’s preferred programming learning website, W3Schools.com, was searched during the field visit and did not yield results on the Haskell topic.

Work Artifacts The following list of artifacts were collected to represent work context. They are electronic resources used in the work of learning programming and the work of using Haskell in the workplace. Learning Programming:

Search Engines: Google and Bing OneNote Application Kindle Internet W3Schools.com Learn You A Haskell For Great Good book Haskell website AppMkr app Sparkle app TopCoder

Using Haskell in the Workplace: Isomorf website, https://isomorf.io/#/

Survey Responses The design team tailored the potential end­user interview script to fit the form of an online survey. We believed that this would be an efficient way to obtain additional data from potential end­users. We contacted related LinkedIn groups and MeetUp groups with the survey URL. We had initial success in establishing relationships with these groups for potential focus groups, but when we reoriented our plan towards a survey, they did not respond to our request. We received two responses from professional acquaintance interviewees (see Appendix F).

Competitive Analysis: Website Evaluations Results The website evaluation form was developed to provide the evaluator with a standardized tool to measure the user experience for websites that teach users new skills. We evaluated three very popular learning websites, all of which also have a mobile version, to analyze the usefulness of the components of these instructional websites and to collect information about their instructional and assessment methods and the fun factor of the sites. Codecademy, KhanAcademy and Duolingo each provided slightly different services for very different content. Codecademy focuses on teaching users how to code, program and create

12

websites. KhanAcademy offers diverse content for the K­12 audience, but the evaluator focused on the Computer Programming lessons for the purpose of this evaluation. Duolingo teaches foreign languages, not coding. All three sites offered some type of orientation, tour or introduction to the site to direct users about how to get around and where to find resources. KhanAcademy was the only site that did not offer a placement test for the content to help the user determine where to start learning. Codecademy offers a placement test, but only if the user signs up for the paid version of the site. All three sites included user­led discussion boards, but again, Codecademy charged for the access to the discussion boards. Two of the websites used a character to provide hints or guide the user through activities. All three websites provided immediate feedback throughout the modules as users were asked to perform tasks or mini­assessments. Duolingo provided badges and awards for completing modules and to keep track of your progress. All three sites allowed the user to create a profile in order to save the user’s progress. This gave the websites a personalized feel as well as allowing users to access the same content at the same place they left off from any device they use. The mobile platform for KhanAcademy only provided a limited view of the page, which made it difficult to complete the tasks. Duolingo had a streamlined user interface for the mobile platform that allowed for full participation without feeling like something was missing.

Example of a character guide from Khan Academy

13

Contextual Analysis

Data Collection Protocols Raw data from the contextual inquiry was recorded in notes, video with audio, and on website evaluation forms (Appendix F).

WAAD Development Process The focus of our research was on the work activity of learning a new programming language and using programming languages to code. To develop our work­activity affinity diagram, we each took the notes from interviews that had been conducted. Team members read the interview notes and began to record work activities on sticky notes. Notes were written in the user’s voice, paraphrased and separated into independent thoughts. Until we ran out of color choices, each interviewee’s work activity notes were written on different color sticky notes. One color was duplicated, but the penmanship of the writers made it easy to distinguish between the two interviews. As we conducted this scan for information, we discussed some of our general findings. Group members then worked quietly to write down relevant work­activity notes.

As group members finished their selection of work activity notes they began to place the sticky notes at random on the sheet on the wall. Once several interviews were up on the wall, we began to read the notes to identify trends. One group member pointed out that it seemed that the interview questions dictated which work­activity notes would be grouped together. For example, we began to notice that several interviewees said they use books to research new programming languages, but all of the responses about books came from the same question asking about what resources the interviewees used when learning a new programming language. We wondered if the categories and trends would all fall under predetermined groupings based on interview questions. We continued to look for connections between the work activities.

14

As we moved work activity notes around, we sometimes disagreed about which loose grouping of related ideas a work activity would fall under. Categories and sub­categories began to form and membership in those informal categories and sub­categories was debated as disagreements arose. We settled on several headings such as “Benefits of Functional Programming,” “Components of Functional Programming,” “Learning Resources,”“Retaining,” “Motivation,” and “Time.”

We continued to discuss where the remaining work activities should go. It was especially difficult to determine where the start­up company staff interview data fit in as this data represented a different type of work. We discovered that we had two different concepts of work for our project. The first concept was the work of learning to code. The second concept was the work of developing an application that teaches code. Our final WAAD demonstrated this distinction in the categories of ‘learning’ and ‘programming’ with the work activity data from the start­up company falling under ‘programming.’ We further elaborate on this distinction with two distinct work­flow models to represent the work of learning to program and the work of programming.

15

Final WAAD

Data Analysis In the creation of the data representation, we examined the WAAD and the various identified learning­oriented categories/sub­categories listed. Several themes emerged from the interview responses. These themes were vetted amongst team members during our WAAD assembly. Similar or re­occurring categorical responses were tallied within these themes by recognizing patterns in word identifiers or phrases. The Data Analysis allowed us to further explore some of the strings of information within several WAAD categories and identify additional connections between data and ideas pertaining to our effort. Within the sub­category “Mobile Apps”, responses related to "Online Tutorial Apps” contained a total of 15 responses and any word identifiers contained in the represented responses were used to create a data category (represented in table 1a) which would benefit or provide an advantage to users. Any similar responses were added to that theme (for example if there were 4 responses which related to the characteristic of “recalling information” as a benefit, they were added to the category "recall"). The tallied responses were then entered into the pie chart and represented as a percentage out of 100.

16

Fig. 1a In the pie chart (Fig. 1b) "Use of Books to learn programming/coding/computer languages" , interview data retrieved a "yes/no" response. This was the simplest to represent visually as a pie graph with responses either being affirmative or negative (yes 7, no 1). The tallied responses were then entered into the pie chart and represented as a percentage out of 100.

17

Fig. 1b The column chart below (Fig. 1c) represents WAAD interview response data related to the users’ “primary motivation in learning programing”. The tallied responses fell into three distinct categories: Big Picture, Problem Solving, and Professional Development. Two remaining responses were unrelated and are represented by the category “Other”.

Fig. 1c

Competitive Analysis Matrix Within the Competitive Inquiry and using our collaboratively generated criteria, we decided to evaluate three companies, which currently provide online instructional products and services across a variety of portable modalities. More than one of the companies identified in our competitive analysis was mentioned throughout these user interviews and appeared in several of the created learning sub­categories reflected in our WAAD. The Competitive Analysis Matrix (Table 1a) provides a visual representation of the categorical data gathered from the navigational exercises conducted on the sites for the three companies examined. The analysis provided valuable contextual insights about the features and environment available to users seeking to conduct a web­based learning activity using a variety of hardware (both portable and non­portable).

18

Category Codeacademy DuoLingo Khan Academy

Main Subject Audience Coding/Programming

X

Orientation X X X

Placement Test X X

User Driven­Discussion Boards

X X X

Characters/Avatars used as user guide

X

Immediate Feedback provided on Learning Activities

X X X

Badges and Awards for learner progress

X

User profiles created to track progress

X X X

Personalized feel throughout navigation

X X X

Mobile Platform Ease of Use

X

Table 1a

19

Flow Models

Isomorf’s Programming Education Flow Model The flow model below depicts the work roles and workflow of programming language education within the Isomorf company. Although the company is still developing its structure, they envision having the following work roles and information flow related to programming education in the future.

Work Roles

Owner/CEO: The Owner/CEO directs marketing goals, collaborates with the Product Development Specialist and communicates the company’s talent needs to the Learning Specialist.

Marketing Specialist: The Marketing Specialist collaborates with the Salesperson to accomplish sales goals.

Salesperson: The Salesperson conducts community outreach to sell the company’s products.

Product Development Specialist: The Product Development Specialist works with the UX Designer to develop products and collaborates with the Owner/CEO on the talents

20

needed for new products. The Product Development Specialist also directs the Developer to learn new programming languages as needed for new product development.

UX Designer: The UX Designer collaborates with the Product Development Specialist and Developer in the development of products and helps identify new programming languages needed for the development.

Developer: The developer programs products and learns new programming languages when needs have been identified and directed by the Product Development Specialist.

Learning Specialist: The Learning Specialist supports the Developer in learning new programming languages by providing and arranging for learning resources.

Data Security and Network Infrastructure Specialist: The Data Security and Network Specialist provides and maintains the framework for safe storage of employee records.

Accounting Specialist: The Accounting Specialist makes payments to educational vendors for ongoing service contracts.

Learning to Program Flow Model Because GoHaskell! will be a commercial application, we also needed to capture the work roles of anyone that wants to learn Haskell and their envisioned relationship to the app. We thought it would be helpful to demonstrate the “after” flow model diagram to show the effect of GoHaskell! on the work environment. Based on our contextual inquiry data, we found the following types of users and information flow:

21

Work Roles Haskell expert Experienced imperative programmer, Haskell novice Experienced functional programmer, Haskell novice Inexperienced programmer Companies that need employees to learn Haskell Isomorf startup team

All of the programmers learn from non­mobile resources mentioned in the contextual inquiry, but only Haskell experts, experienced imperative programmers, and experienced functional programmers are envisioned as future users of GoHaskell! based on our contextual inquiry. All of the programmers write code to create computer programs that increase efficiencies for the companies at which they work. The Isomorf start­up team will assist in developing the app and participate in a continual improvement process based on user feedback.

22

Appendix Appendix A: Interview Script/Questions and Observation Prompt for Isomorf Appendix B: Interview Script/Questions and Observation Prompt for non­Haskell Programmers Appendix C: Consent Form Appendix D: Online Survey Appendix E: Website Evaluation Criteria Appendix F: Raw Data Collected

________________________________________________

23

Appendix A: Interview Script/Questions and Observation Prompt for Isomorf Part I: Interview Questions (30 minutes) Scripted Introduction Thanks so much for meeting with me. I hope this will be fun and interesting for both of us. As you know, the aim of the research is to find out a little bit more about functional programming and the way it’s being used in the private sector. The reason that I’ve asked you is because I’m aware that your company uses functional programming and I would like to find out a little bit more about what your company’s experience has been about so far, and what you see for the future. I’m expecting the interview to last about 30 minutes or so. If you want to stop at any time or take a break, that’s fine. And if I ask a question that you don’t feel like answering, that is fine, too. Sign consent form. Do you have any questions for me before we get started?

1. I’d like to know a bit about [company]. a. Why did you found [company]? (What do you believe in?) b. What sets you apart from other companies in this field? What are your core

values? How will you solve problems differently? c. What are you building? d. How do you plan to market [company]? What’s your message?

2. Once you begin to grow, what do you expect the structure of your company to look like? What teams might you have and for what purpose?

3. Can you explain to me why functional programming is so important to your company? 4. I understand that Facebook recently rewrote their system Sigma, which protects against

spam, malware, etc., in Haskell. Are there certain types of programs that functional programming is best for?

5. In the context of [company], who needs to understand functional programming? Employees? Users? Anybody else?

6. What are you looking for in future programming employees? What experience and skills must they have?

a. Do they need to know functional programming to be hired? If not, how do you plan to teach them about functional programming?

b. What language will employees code in? c. Are there languages future employees can practicing using now?

7. Who do you envision as your future users? In 2 years? In 10 years? a. How will you attract users? b. Do your users need to understand functional programming?

24

c. What language will users code in? Now we are going to talk more specifically about functional programming and how people can learn it.

8. Can you tell me some of the feelings that you associate with functional programming? 9. Can you walk me through how you learned to code using functional programming?

a. (If needed): Did you use any of the following resources? (formal classroom environments, formal documentation, free or purchased online programs, online forums, colleagues, on­the­job training, mobile app, book or textbook?)

10. How long do you think it would take someone to learn Haskell with no prior experience with functional programming?

11. Have you learned anything on a mobile app? It does not have to be related to programming.

a. What did you learn? Name some apps if you can. b. What was it like? What features did it have that you liked/disliked?

Part II: Observation (15­30 minutes) Materials Computer with an Internet connection Video recorder (if interviewee consents) Scripted Directions Now we’ll move to the observation portion of the interview. I’d like you to pretend that you have just hired an intern who is unfamiliar with functional programming. For the next 30 minutes, please show me how you would begin to explain and demonstrate functional programming to your intern. You can mention basic concepts, common errors, etc. You can use this computer if you’d like.

25

Appendix B: Interview Script/Questions and Observation Prompt for non­Haskell Programmers Part I: Interview Questions (30 minutes) Scripted Introduction Thanks so much for meeting with me. I hope this will be fun and interesting for both of us. As you know, the aim of the research is to find out a little bit more about what people’s experience of programming and learning to program is like. The reason that I’ve asked you is because I’m aware that you’re a programmer and I would like to find out a little bit more about what your personal experience of learning to program has been about. I’m expecting the interview to last about 30 minutes or so. If you want to stop at any time or take a break, that’s fine. And if I ask a question that you don’t feel like answering, that is fine, too. Sign consent form. Do you have any questions for me before we get started?

1. Can you tell me how many years of experience do you have in computer programming? 2. Can you tell me some of the feelings that you associate with programming?

a. What aspects of it do you like? (can you give me an example?) b. What aspects of it do you NOT like? (can you give me an example?)

3. Can you remember a day in the last month in which you had to write code? Walk me through that day (were you at work? who did you communicate with? etc.). (Try to pick up on the high points/low points and inquire about them.)

4. Have you used any of the following resources to learn to program (I’ll read one at a time)?

a. Formal classroom environments (high school, college, grad school) Which? b. Formal documentation of the language (online) c. Free online programs (CodeAcademy, Khan Academy, etc.) Which one(s)? d. Online programs that you paid for (Lynda.com, etc.) Which one(s)? e. Online forums (StackOverflow, GitHub, etc.) Which one(s)? f. People you know (friends and family) How did you learn together? g. On­the­job training at work. What was the nature of your on­the­job training? h. A mobile app. Which one(s)? i. A book or textbook

5. What are some of the things that you liked about learning with these resources? 6. What are some of the things that you did NOT like about learning with these resources? 7. What motivates you to learn a new programming language? (required by current job?) 8. I’m particularly interested in finding out about people’s memories of their experiences,

especially stories that people tell about their experiences. Can you remember the last

26

time you learned a new programming language? Walk me through the steps. (Try to pick up on the high points/low points and inquire about them.)

9. What helps you to retain information when learning programming skills? 10. What causes you to forget information about programming? What do you do when you

forget newly learned information about programming? 11. About how long does it take you to learn a new programming language?

a. If it depends, what factors are there? 12. In order to learn a new programming language, I will need ____________________. 13. The thing I find hardest about learning a new language is_______________________. 14. One of the most time­consuming components of learning to code in a new language

is____________________. 15. If you were to learn a new programming language, when would you probably do it?

(leave open­ended, but use the following if need be: Weekends, Weekdays, Before work, At work, After work, Sporadically throughout the day, On vacation

16. Why did you choose this (these) time(s)? (Trying to see if they are limited by needing a computer)

17. Have you ever used a mobile app to learn anything? a. What did you learn? Name some apps if you can. b. What was it like? What features did it have that you liked/disliked?

18. Have you heard of the programming language Haskell? If so, what sorts of ideas and feelings do you associate with it?

19. Is there anything else that you would like to add that could help us develop an application to teach programming? (or “about your experience learning to program?”)

Part II: Observation (15 minutes) Materials Computer with an Internet connection Video recorder (if interviewee consents) Scripted Directions Now we’ll move to the observation portion of the interview. I’d like you to pretend that you have just been given an assignment that requires you to code in Haskell. For the next 10 minutes, please show me how you would begin to learn about Haskell. You can use this computer. Also, I’d like you to think aloud as you do it. In other words, try to verbalize your reasoning behind the decisions you make. Questions to ask as they perform the activity:

1. What kinds of resources do you use/prefer to use? (videos, forums, books, etc.?) Why? 2. Are you learning anything surprising or interesting about Haskell? 3. How do you feel as you’re learning about Haskell? 4. How long do you think it would take you to learn Haskell?

27

Appendix C: Consent Form

28

Appendix D: Online Survey (Google Forms) The online survey was built in Google Forms. It is accessible from the following URL: http://goo.gl/forms/3VVa2JzCic. Please provide responses about your programming experience and learning experiences.

1. How many years of computer programming experience do you have? a. 0­5 b. 6­15 c. 16­25 d. 26+

2. Which of the following programming languages do you know (would be comfortable putting on a resume)? (check all that apply):

a. C b. C++ c. C# d. Go e. Haskell f. Java g. Javascript h. Pascal i. Perl j. PHP k. Python l. Ruby m. Scala n. Swift o. Other:

3. Can you explain the difference between imperative programming and functional programming?

a. Yes b. No c. Not sure

4. If you are familiar with functional programming, what sorts of ideas and feelings do you associate with it? You can mention pros and cons, likes and dislikes, etc.

5. Have you used any of the following resources to learn to program? (check all that apply): a. Formal classroom environments (e.g., high school, college, graduate school, etc.) b. Books and/or textbooks c. Formal documentation of languages d. Online forums (e.g., StackOverflow)

29

e. Free online programs (e.g., CodeAcademy, Khan Academy, etc.) f. Online programs that you paid for (e.g., Lynda.com) g. People you know (friends and family) h. On­the­job training i. A mobile app j. Other:

6. What are some of the things that you LIKED about learning with these resources? 7. What are some of the things that you did NOT LIKE about learning with these

resources? 8. What helps you to retain information when learning programming skills? 9. What causes you to forget information about programming? 10. If you wanted to learn a career­advancing skill, when would you probably do it? (check

all that apply): a. Weekends b. Before work c. At work d. After work e. Sporadically throughout the day f. On vacation g. N/A h. Other:

11. Have you ever used a mobile app to learn anything? a. Yes b. No

12. If you have ever used a mobile app to learn, what sorts of things did you learn (not necessarily career­related)? Please include the names of apps if possible. You can also mention features that you liked and disliked.

13. Have you tried using a mobile app to code? a. Yes b. No

14. Is there anything else that you would like to add that could help us develop an application to teach programming?

30

Appendix E: Website Evaluation Criteria

31

Appendix F: Raw Data Collected Interview Notes (Aaron Webb)

Interview Notes ­ Image of Selection (Aaron Kent and Brian Kent)

32

Interview Notes ­ Image of Selection (Michael Cobb)

Interview Notes ­ Image of Selection (Marshall Koch)

Observation Videos (Michael Cobb) Part 1: https://www.youtube.com/watch?v=EWOkmrrVp2Y Part 2: https://www.youtube.com/watch?v=JZiC5jy6fRc Part 3: https://www.youtube.com/watch?v=DyaJ4fngdEw

33

Part 4: https://www.youtube.com/watch?v=rwZY2ivdaRA Part 5: https://www.youtube.com/watch?v=9ai9bUhvOIg Part 6: https://www.youtube.com/watch?v=gPpnG4R_zmc Observation Video and Transcript (John Frye) YouTube Link: https://youtu.be/Dw_Fq46uZFQ

Line #

Transcript

1 Interviewer: This is John Frye and he’s going to tell us

2 about his experience in programming and answer some

3 questions about learning Haskell. John, how would you

4 learn Haskell if you needed to?

5 Interviewee: Well, the first thing, the first thing is, because

6 it’s my career and I tend to manage my career through

7 notes and notebooks and things that I accumulate, what I

8 use typically is this notebook application which is OneNote

9 and within OneNote you can see I have various projects

10 that I keep in here as well as some work notes and

11 general and then I have an engineering section and a

12 career development section, as well as a personal and

13 other section. So in my engineering section I keep

14 running notes of software engineering as well as systems

15 engineering, security and hacking, network engineering,

16 Enterprise IT, etc. The nice thing about the electronic

17 notebook is I can create the notebook, I can create

18 sections, and then I can create pages and I can move

19 them all around search with the easy search function here

20 and I can search all my notebooks for a particular subject

21 or whatever. So, for instance, in software engineering,

22 under software engineering, I have a section called

23 programming and then on the pages I have a number of

24 programming subjects that I have already looked up and

25 stored, taken notes and stored or pulled the information off

26 something. So I have a tools page, I have sequel, which is

27 structured query languages and databases, I have an

28 extreme programming section, as well as Java, .NET

29 services and HTML5. So, one of the sites that I found that

34

30 was really good to obtain a lot of this information is this

31 tutoring site, it’s a programming tutoring site, and it’s called

32 W3schools. com. It allows, you see its got all these

33 different languages that it provides learning sessions on,

34 but let’s see if there’s a Haskell section, and it looks like

35 there may not be. So let’s see if I do a search. And it’s

36 called Haskell? See if anything comes up and it doesn’t.

37 Interviewer: That’s good to know. So what would be your next…

38 Interviewee: So the next one I guess I would have to start

39 searching through Bing. Just to make sure I don’t have

40 any notes on it already that I've forgotten about.

41 [interviewee searches in OneNote] See if anything come

42 up in my notes and no, nothing. So I would start with my

43 search engine, see what comes up, programming in

44 Haskell. Looks like a book comes up. So, I tend to like

45 books also so I have the um. Oh, maybe I don’t have it on

46 this one. What’s the name of the..it’s the Kindle right?

47 Yeah, I don’t have it on this, but i do have a number of

48 Kindle applications on other computers and notepads or

49 iPads and such. So I tend to like books and I would

50 probably first go look at the books. But let’s see what else

51 there is, if there’s anything free. Here’s a PDF. This looks

52 like it might be a free PDF. So I would probably go and

53 explore that, make sure I wasn’t going to get a virus before

54 I started clicking on anything, and that it was legitimate.

55 But it looks like a free pdf that i would probably explore.

56 Interviewer: How would you check to see if it had a virus?

57 Interviewee: Well, main thing I would do is look at some of the links and

58 make sure they’re not going off to some foreign country

59 and that nothing comes up in my virus detection and just

60 look generally to see if there’s anything fishy about it.

35

Observation Notes ­ Image of Selection (Aaron Kent and Brian Kent)

36

Online Survey Results

37

38

39

40

41

Website Evaluations Khan Academy

42

43

44

45

Duolingo Evaluation For all categories the scale of 0­4 will represent 0 (not at all), 1 (mostly false), 2 (somewhat true), 3 (mostly true), 4 (very true)

Feature or Component

Easy to Use (0­4)

Visually Appealing (0­4)

Useful for your purpose (0­4)

Enjoyable or Fun (0­4)

Responsive to user needs (0­4)

Tour or Introduction of Site: a cute owl

4 4 4 4 4

46

guides you through steps and gives directions as you need them

Navigation 4 4 4 4 4

Placement Test (test out option for modules)

4 4 4 4 4

Instructional Modules

4 4 3 3 3

Discussion Boards 3 3 4 3 4

Assessments 4 4 3 4 3

Resources (optional communal translation and editing activities)

4 4 3 3 2

Games/Interactivity Listening activity, translation, badges

4 4 3 4 3

Additional Tools /Resources

0 0 0 0 0

1. Were you able to navigate successfully through the learning modules using a variety of devices (ie. Tablet, mobile /smartphone, etc.)? Yes. Phone app and computer based website. 2. Was the program complimentary to your level of knowledge or experience? Somewhat. It adjusts to your knowledge using assessments. The assessments were pretty limited in scope however and did not seem to get a completely accurate picture of my skills. For example, even though I am very proficient in German and not as much in Spanish, I tested at 56% proficient in Spanish and only 1% proficient in German. Now I can test out of modules in German, but that will take a long time. It varied as to whether it gave me credit for answers that did not match exactly what they intended you to produce. 3. Describe how the content was organized and whether you found that organizational structure helpful or not. The content is organized by topic or theme such as animals, household items, or grammar topics such as the dative case, asking questions and using numbers. 4. Describe the instructional methods used to aid learning and retention.

47

Repetition of sound and visual cue, practice dictation, immediate corrective feedback, frequent assessment. 5. How would you rate the overall experience of using this website or application? Add any additional reflections. (Scale: 1 = worst, 10= best) Because of the lack of accuracy of the levels, I would rate it a 7 because it wasn’t responsive to my exact level, but the modules themselves seems pretty well designed for teaching the concepts they intended to teach. It seemed like only active learning because even the introduction to new material was interactive. Codecademy Evaluation For all categories the scale of 0­4 will represent 0 (not at all), 1 (mostly false), 2 (somewhat true), 3 (mostly true), 4 (very true)

Feature or Component Easy to Use (0­4)

Visually Appealing (0­4)

Useful for your purpose (0­4)

Enjoyable or Fun (0­4)

Responsive to user needs (0­4)

Tour or Introduction of Site (you have to go pro and pay in order to use the personalized learning tool)

2 2 2 2 3

Navigation (found it difficult to scroll in multiple windows during the modules)

3 3 3 2 3

Placement Test (none, or only with payment)

0 0 0 0 0

Instructional Modules (can select modules, can’t skip within one)

3 4 3 3 3

Discussion Boards (could not locate from free service)

0 0 0 0 0

Assessments (embedded in learning process)

4 4 3 3 2

Resources (embedded in the modules)

2 3 2 2 2

48

Games/Interactivity (embedded in modules, more like guess and check)

3 3 3 3 3

Additional Tools /Resources(articles and outside resources, but not many)

2 2 2 2 2

1. Were you able to navigate successfully through the learning modules using a variety of devices (ie. Tablet, mobile /smartphone, etc.)? Yes, but you can’t move forward in a module until you have completed the task. I did not find a mobile app for codeacademy 2. Was the program complimentary to your level of knowledge or experience? Not really. It was very basic and you can’t skip ahead to your level. Unless you pay for a subscription. 3. Describe how the content was organized and whether you found that organizational structure helpful or not. The left window had explanation and instructions. The right window had an input/output window so you could see what the result was of what you were doing with the code. It asked you to do specific tasks and you tried them and watched the result. 4. Describe the instructional methods used to aid learning and retention. The instructional method seems to be in­context learning or guided learning similar to having a mentor walk you through the steps. 5. How would you rate the overall experience of using this website or application? Add any additional reflections. (Scale: 1 = worst, 10= best) I would rate it a 5 because it does not seem to adjust to the level of the user. However, I have not tried the paid version, so I bet it is a lot more adjustable to what you need.

49