Using ADDIE and Systems Thinking as the Framework for Developing a MOOC: A Case Study

Rebecca A. Croxton, MLIS

Instructor and Doctoral Student Department of Library and Information Studies / Department of Teacher Education & Higher

Education School of Education

The University of North Carolina at Greensboro 1300 Spring Garden St.

School of Education Building, #450 Greensboro, NC 27402-6170 Email: racroxto@uncg.edu

Anthony S. Chow, Ph.D. Associate Professor

Department of Library and Information Studies School of Education

The University of North Carolina at Greensboro 1300 Spring Garden St.

School of Education Building, #450 Greensboro, NC 27402-6170

Email: aschow@uncg.edu

Keywords: MOOCs, ADDIE Model, Systemic Change

“In my opinion, the overall instructional design is the magic of a successful MOOC. True quality instruction cannot be just designed on paper, but must be an expression of both the expertise of the instructor as well as the art and scientific expression of creativity and constructivist creation as they design and develop a course in a digital world” (A. Chow, personal communication, October 14, 2014).

Introduction

The Internet has “happened” to education and it is changing the way students and instructors engage with one another and the way the “business” is done. Higher education as we know it is undergoing major paradigm shifts. In order to meet demands of increasing globalization in an era of shrinking budgets, while addressing issues of social mobility and access to education for all (Jongbloed, Enders, & Salerno, 2008), higher education institutions must embrace systemic change and clearly reorganize and align their resources around changing goals, access, and demand. In a recent interview, George Siemens, Associate Director of the Technology Enhanced Knowledge Research Institute at Athabasca University, explained that the “functional elements of higher education have been pulled apart and are waiting to be remixed” in ways not yet realized (Grush, 2012). While this remixing will not take one form or have a single solution, current trends suggest technology and the Internet will play a critical role. Chris Dede, Harvard University professor, explains, “… real gains in the productivity and effectiveness of learning will not come until universities radically reshape those structures and practices to take full advantage of the technology” (Waldrop, 2013).

As universities struggle to re-conceptualize education, innovators in higher education are developing and offering massive open online courses (MOOCs) that provide learners across the globe with free access to university faculty and content. Pappano (2012) explains, “The shimmery hope is that free courses can bring the best education in the world to the most remote corners of the planet, help people in their careers, and expand intellectual and

269

personal networks” (n.p.). From an economic perspective, many people believe that MOOCs will address the fundamental challenge to contain the costs of teaching more students using fewer resources (Fischer, 2014). While MOOC development began in many of the United States’ elite institutions, including Stanford, UC Berkeley, Harvard, and MIT, MOOC development is beginning to occur in mainstream higher education institutions across the world. Though higher education leaders are faced with the pressure of either entering the MOOC environment or running the risk of being left behind, the purpose and goals of offering MOOCs are often unclear.

In their simplest form, MOOCs can be considered higher education courses with open, often massive enrollments that offer free education to all. While the design of MOOCs varies widely, they share some commonalities, including “…blended learning, open educational resources, and crowd-sourced interaction” (Johnson, Adams Becker, Estrada, & Freeman, 2014, p. 11). Because anyone with an Internet connection can enroll in a MOOC, enrollment often exceeds tens of thousands, therefore faculty cannot respond to students individually. Thus, the presentation of material and interactivity built into course design is of utmost importance. Classmates may be required to lean on one another in study groups organized in their towns, in online forums, or even for grading work (Pappano, 2012).

The purpose of this paper is to present a case study of how systems thinking and the ADDIE (analysis, design, development, implementation, and assessment) model were used to design and develop one of the first MOOCs at a mid-sized university in the southeastern United States. Contemporary issues surrounding MOOCs at both the macro university level and the micro instructor level are explored. Further, use of the ADDIE model to bring about a user-centered, meaningful, and well-designed MOOC is examined in this time in which MOOCs and systemic change are being called for in higher education.

Literature Review

Brief History of MOOCs

In their brief existence, MOOCs have received a great deal of attention in both higher education and in mainstream news. The term Massive Open Online Course was originally coined in 2008 by Stephen Downs and George Siemens and came into broad use in 2012 (Johnson et al., 2014). The first MOOC, an online course in “Connectivism and Connective Knowledge,” was offered by the University of Manitoba in 2008. The course enrolled a small number of paying students and was also offered to a group of online “auditors” who participated in the course for free. Unexpectedly, enrollment exceeded 2000 auditors (Sandeen, 2013). The first US-based MOOC was offered in 2011 by the University of Illinois, Springfield and enrollment exceeded 2,500 (Defining a MOOC, 2013).

Since the first MOOCs were launched, prestigious public and private universities as well as for-profit providers (Heller, 2013; Lewin, 2012) have enrolled millions of students around the world in these courses. Rapid growth in enrollment has occurred, in part, by the emergence of three leading platforms in the MOOC market – Coursera (founded by Stanford University professors), Udacity (progeny of Palo Alto), and edX (jointly founded by Harvard University and MIT). Illustrating this trend, when a Harvard University professor offered his popular course, “Concepts of the Hero in Classical Greek Civilization” via HarvardX (Harvard’s proprietary line of edX) in Spring 2013, enrollment exceeded 31,000 (Heller, 2013).

While the number of and enrollment in MOOCs is on the rise, the majority of higher education institutions (53 percent) report they are undecided about MOOCs (Allen & Seaman, 2014). Less than one-third (33%) of these institutions say they have no plans for MOOCs. In their 2014 report on the state of online learning in the United States, Allen and Seaman (2014) noted that the percent of higher education institutions that currently have a MOOC increased from 2.6 percent to 5.0 percent over the previous year. In this same report, Allen and Seaman (2014) noted that only 23 percent of academic leaders believe that MOOCs represent a sustainable method for offering online courses, down from 28% in 2012. Contemporary Issues Surrounding MOOCs

While MOOCs were widely embraced in 2012, the low completion rates (5-16% overall) reported in 2013 brought about a major change in attitude about the quality of education MOOCs can offer (Johnson et al., 2014). MOOC skeptics questioned how engaging such learning environments actually are (Fischer, 2014; Johnson et al., 2014). As leading MOOC providers analyzed the reasons for these low completion rates, they have begun to realize that a “more expansive, human-centered support structure is key to helping students retain information, stick with the course – and finish (Westervelt, 2013). John Dewey (1938), early reformer in education, explained that learning is based on discovery guided by mentoring, rather than transmission of information. Thus, learning approaches must

270

Figure 1: ADDIE Model

move beyond traditional lecture style pedagogies typically found in MOOCs and instead focus on providing students with opportunities to engage with their teachers to feel comfortable, confident, and successful.

Analyzing the movement of a million MOOC users through 16 Coursera courses offered by the University of Pennsylvania from June 2012 to June 2013, researchers noted that only about 50% of those who registered for a course ever viewed lecture materials and only about 4% completed the courses, with user engagement falling off dramatically in the first two weeks of enrollment (Stein, 2013). While many MOOCs are heavy on course content, the engagement piece for learners may be lacking. Ray Schroeder, director of the Center for Online Learning, Research and Service at the University of Illinois, Springfield, suggested three things matter most in online learning: quality of material covered, engagement of the teacher, and interaction among students (Pappano, 2012). While quality content does not appear to be an issue, creating opportunities for instructor connection and student interaction creates a significant challenge when there may be thousands of students enrolled in a single course.

Need for a User-Centered Design Approach

Creating a MOOC that is satisfying, engaging, and meets the needs of learners calls for a user/learner-centered design approach. User-centered design is the process of designing a tool (e.g., course) from the perspective of how it will be understood and used by the user (Nielson & Loranger, 2006; Usability First, n.d.). By employing user-centered design approach to MOOC design, a course can be offered that is more efficient, satisfying, and user-friendly for the learner.

MOOCs, designed largely to be media driven and asynchronous in nature, provide a teaching/learning environment that fits nicely with the application of user-centered instructional design principles. Asynchronous communication is that which does not require all parties involved in communication to be present and available at the same time (e.g., online discussion boards). With a learner-centered rather than teaching-centered approach, the ADDIE model is an ideal framework that can be utilized in MOOC design that can help to make courses more applicable and meaningful for learners (Culatta, 2013; Peterson, 2013). While successful use of the ADDIE model in online course design has been documented in the literature (Evans & Lockee, 2008; Summerville, 2003; Wang & Hsu, 2009), there is currently no reference that has been identified by the study’s authors with regard to the use of the ADDIE model in MOOC design.

The ADDIE framework (Figure 1) is a cyclical, iterative process consisting of five stages: analysis, design, development, implementation, and evaluation. In the analysis phase, the main consideration is the target audience (MOOC students) and the determination of what they know and what they need to know by the end of the course (Peterson, 2013). During the design stage, course developers must determine the objectives of the course, how objectives will be met, how they will be assessed, and the methods that will be most effective in delivery of the objectives (Seels & Glasgow, 1988). If the goals, objectives, and assessments of a course do not align, learners may find themselves losing interest in the course and influencing perceptions of instructional quality (Peterson, 2013). Students’ participation during the design phase can contribute to their overall course satisfaction and plays an important role in determining whether students continue in a course (Murphy, 1999). Referring to results from the analysis and design phases, course designers in the development phase construct the course for delivery of information while, at the same time, determining if students will learn from the course and how it can be improved before implementation (Peterson, 2013). In the implementation phase, course developers, learners, and the instructor actively work together to analyze, redesign, and evaluate the course and make instantaneous modifications to the course to ensure effectiveness. Finally, during the evaluation phase, the course designer must determine if the objectives have been met, evaluate the impact of the course through summative and formative evaluations, and make changes necessary for future delivery of the course.

In a time which MOOCs and systemic change are being called for in higher education, this case study explores how a systems thinking approach and the ADDIE were utilized to develop one of the first MOOCs at a mid-sized university in the southeastern United Stated. More specifically, the authors sought to answer three primary research questions:

271

RQ1: What is the process for designing and developing a MOOC? RQ2: What impact did using the ADDIE model have on MOOC design? RQ3: What is the relationship between instructional design and MOOCs?

Method

The instructor and co-author of this paper were closely followed throughout the MOOC design and development process. Through a lens of instructional systems design and systems thinking, a case study of the development of a MOOC designed to teach web design and usability principles and skills at a public, mid-sized regional university in the southeastern United States was undertaken. Throughout development, interviews were conducted with administrators, the design team, a user group that provided input throughout the process, and the first student completer of the MOOC. The instructor also added his own perspectives throughout the project, detailed the goals and purposes of the MOOC, the process of design and development, and the challenges and implications faced throughout the project.

Results

The university had been following the evolution of the MOOC and wanted to test out the space. Initially, two pilot courses, “Web Design and Usability” and “The Soul & the Search for Meaning” were identified for development, to be offered Spring 2014. This study reports the conceptualization, development, and implementation of the “Web Design and Usability” course. The results of the case are presented according to the three primary research questions for the study, using the key elements of the ADDIE model (analysis, design, development, implementation, and evaluation) throughout. RQ1: What is the process for designing and developing a MOOC?

To develop this course, a full design team was assembled at the onset of the project and included a project manager, instructional designer, graphic designer, video production team, and the lead faculty member, Dr. Anthony Chow. Chow, as the faculty member of record made sure the ADDIE process was followed throughout the project. The university provided $13,000 for the development of this MOOC, which included $3,500 for faculty compensation, with the remaining funds allocated for production costs. Analysis Phase

At the onset of the analysis phase, University leaders, the Chancellor, production team, and lead faculty member met and decided upon the goals of the MOOC. Ultimately, the following two goals were identified that aligned closely to the University’s larger mission.

1. Teach, free of charge, how to design a website and basic usability concepts and methodology. 2. Publicize and brand the university. Because the MOOC was designed to be a free course and statistics

continue to show that retention among students who enroll in MOOCs is not very high, it was decided that the MOOC would be used as a way to brand the university as a place people might consider attending as a campus-based or online student. With this in mind, the production team tried to ensure the university was branded throughout the course through logo placement, video clips recorded throughout the campus with Chow wearing university branded clothing, and a university mascot presence throughout the course to “keep it light.”

3. Serve as test case for different instructional strategies. As a free, open course, the MOOC provides an opportunity to test the efficacy of different instructional strategies with a large number of learners. This provides the opportunity to experiment with different techniques and measure their effectiveness.

Early market analysis identified young, male, tech-savvy professionals as the primary target for this

MOOC, thus the course was developed primarily with this user group in mind. With the goals of the course and the primary target population identified, the design phase of the course was ready to begin.

272

Figure 2 - Outcomes Map refined using Google Drive by the Design Team

Figure 4 - All units had brief one minute introductions

Figure 3: Co-authors Chow and Croxton at a design meeting

Design Phase

With the analysis phase underway, the production team met on two occasions during which time they identified course objectives, created a project timeline, and established the basic architecture of the course using an outcomes map (Figure 2). The production team decided that the course would be comprised of five asynchronous units (Usability, User-Centered Design, ADDIE Process, Evaluation and Testing, and Web Designed) that included a variety of ways of presenting information. On average, each unit was to have between 2 - 5 sections (15 altogether for the course) that would also undergo the ADDIE process and emphasize hands-on application throughout.

All main units and sections were designed to include a video introduction with Chow as the presenter (closed-captions and written transcript provided for all videos), unit review sessions comprised of five multiple-choice questions, online discussion, hands on activities, and readings. Finally, being under the assumption that students would want to know how “…we practiced what we preached,” video case studies were designed for inclusion as final components of each unit.

An important element of instructional design is deciding upon what constitutes acceptable performance. Because the design team wanted to give all individuals who successfully completed the MOOC a certification of completion (no university credit awarded), a basic concept of mastery learning (90% or above) was utilized. Each unit section had five questions and each unit had a test that included the section questions. In order to earn the certificate of completion, students had to earn at least 90% on all unit tests as well as on a final MOOC exam.

Development Phase

A key element of the development phase of the MOOC was creating the introductory videos for all unit and section introductions as well as lectures (Figure 4). Creating the videos were high production affairs. Because the production team staff had expertise in video production, all videos were filmed and edited by expert videographers, using high-end video equipment as well as a teleprompter from which Chow read pre-prepared transcripts. In a recent interview, Chow noted, “…one of the hardest things was writing out the introductions and lectures and reading these off the teleprompter” (A. Chow, personal

273

Figure 5 - User Group Interview

Figure 6 – MOOC Welcome Screen at moocs.uncg.edu/wdu

communication, October 14, 2014). The online software, WordPress, was used for course development because of the flexibility of the program as well as the widgets that were available. The widgets were utilized to create the discussion forums as well as the multiple choice online review sessions.

An essential component of the development phase of any instructional project is to determine if students will learn from the course and how it can be improved before implementation (Peterson, 2013). With this in mind, a group of usability reviewers analyzed the course after the first functional unit that included videos, content, review questions, and a discussion forum was complete (Figure 5). While the usability group should ideally have included young, tech-savvy male professionals (target population identified in the market analysis), the production team did not have ready access to these types of individuals, as most of the users were female.

As a result of the usability reviewer feedback, numerous modifications were made in the MOOC’s design. The most significant changes to the course related to the videos recorded for the course. One of the struggles when designing online content is the ideal length of videos. Initially, videos were shortened to be 15 to 20 second clips, with numerous clips following one after another in a single introduction. According to user feedback, this was too much and too short. As a result, the production team moved away from short clips to longer instructional videos that were kept purposefully, relatively short (average 3-4 minutes, longest was 10 minutes). Additionally, some individuals in the user group indicated they would prefer to read a video transcript instead of watching a video, thus transcripts as well as closed-captioning were created and made available for all videos.

During the implementation phase of the MOOC, the first version in the first section of the course was reviewed by over 60 graduate students enrolled in two courses within the university’s master of library and information studies program, “Instructional Design” and “Library Management.” Overall, the users’ feedback was good, indicating to the production team members that they had heard the user group and had responded positively. The positive feedback received from the reviewers gave the production team the confidence to move forward and create the remaining 14 sections. Figure 7 – Section or Module Home Page

274

Implementation Phase The MOOC was first released with a

quiet rollout to ensure any remaining problems were ironed out which included the entire registration to course completion process.

The MOOC begins with an introduction page where prospective learners can gain a clear overview of what the course has to offer (Fig. 6).

Each unit begins with a brief introductory video along with a full transcript. Feedback suggested that some learners would prefer to read rather than watch a video while other preferred the video over text. In addition, the MOOC navigation was designed to clearly orient the learner to where they were. A right side bar also provided learners with easy access to all other units (Figure 7).

For each of the five unit introduction videos, an attractive area of campus was chosen and also Spyro the UNCG Spartan was introduced as both a way to brand UNCG as well to bring a bit of fun into what otherwise was more academic and intellectual content (Figure 8). Each section or module within a Unit was presented in the right side bar for easy access and orientation.

Each section is comprised of a brief instructional video, a five question review session or assessment, threaded discussion, applied activity (posted on the discussion board), and readings and supplemental resources.

Review sessions involve mostly multiple choice questions (Figure 9). While more interactive questions were intended, most ended up as simply multiple choice due to time and resource limitations. In addition, despite being basic the questions were intended to create reflection and cognitive processing of the main points of each section. The MOOC’s final exam involves all combination of all section review sessions. A learner has to pass this exam at 90% or above in order to receive a Certificate of Completion.

Within each section, learners are encouraged to apply learning concepts to their daily lives or something applied. This helps create a learning community around each section that allows each learner to contribute their individual perspectives and world views to the instructional material and content. While everything is optional, learners are presented with hands-on activities at the end of each section working toward the ultimate goal of creating a website at the end (Figure 10).

Learners are presented with additional resources that are open access. Because learners are not formally UNCG students they are not allowed access to library resources.

Each unit has two to four sections each. At the end of each unit we also provided a case study that detailed how each unit was designed and developed and process that was used. Another goal of the MOOC was to facilitate

Figure 8 – Dr Chow and Spyro the Spartan

Figure 9 – Section Review Session

Figure 10 – Learners present results of applied activity and discuss with other learners

275

the creation of long term learning communities and social networks for all learners. Facebook, Linked-in, and Twitter accounts were created to help support the formation of such communities (Figure 11).

Evaluation Phase

Throughout the MOOC design and development, the usability process reflected the “evaluation” in the ADDIE model, which required that the production team constantly evaluated how they were doing. The MOOC was launched in July 2014 with a soft roll out, with official launch in August 2014.

Anecdotal evidence from a learner who is the first person to earn the certificate of completion suggests the course has been an initial success. In a recent interview, this student indicated she appreciated the flexibility the MOOC offered, noting, “The freedom to pick up and plow through parts of the MOOC at moments where I had free time/during my lunch break at work made it easy to get through the process (A. G. Bradley, personal communication, October 20, 2014). She further stated that the MOOC has helped her gain skills and knowledge that have been applicable to her work, stating, “Thanks to the MOOC (and working with Dr. Chow), I am undertaking a user-centered study with the State Library … I have also implemented the Aunt-ADDIE based user-design principles into my own work…” Reflecting on opportunities for engagement with other students enrolled in the MOOC, this same student noted, “…I admittedly rushed through this [the discussion portion of the course] because of time constrains my summer course load placed on me.” Finally, when asked about her impressions of MOOCs in general, this student explained,

I am admittedly torn on a MOOC; I appreciate the ease of use and ability to complete the assignments at my own pace, but I miss the in-person interaction. It feels a bit detached for me. I am enrolled in an upcoming … MOOC (to keep my skills fresh) and am interested in comparing the two. MOOCs will continue to grow and evolve based on technology and the needs of users. They are still in their relative infancy when compared to other educational technology (A. G. Bradley, personal communication, October 20, 2014).

The end of MOOC evaluation results suggests that learners are for the most part satisfied rating it a 6.5 (n=36) out of 7.0.

Figure 12: Learner Satisfaction with MOOC

Figure 11 – MOOC Social Networking Communities

276

Some user comments include:

“As a "digital native" I was able to figure out the website within about a minute, but those not as comfortable with technology might have some problems without instruction. Having used moodle and blackboard, however, I would say that this is much more intuitive.” “This should be well received by all learner-types.” “I felt the examples were appropriate and relatable. Good ways to illustrate the concept of usability. I think the discussion questions and applied activity question are good ways to further think examples from one's own life.”

RQ2: What impact did using the ADDIE model have on MOOC design?

In direct response to this research question, Chow explained,

The best thing about a model is that it gives you a process by which to structure what otherwise is a very tiring and complex process. In any kind of production, there are timelines, deadlines, and stress involved. The MOOC was an add-on to an already busy schedule” (A. Chow, personal communication, October 14, 2014).

The ADDIE process helped to keep the team focused, because as they developed the course, the ADDIE

process required that they constantly evaluate how they were doing (the “E” in the ADDIE). Essentially, the ADDIE model helped to: A: Ensure goals are clearly defined D: Create a timeline and outline the needs of the project so the production staff knew what resources

(human and financial) they needed to encumber. D: Keep the team to a timeline by which they would have something functional by a certain date. I: Encourage feedback after initial implementation that gave production team confidence to move

forward. E: Create an environment for periodic feedback that allowed for continuous refinement. Dr. Chow was the leader of a production team and the ADDIE model and process allowed him to keep the

entire moving throughout an eight month design and production process. RQ3: What is the relationship between instructional design and MOOCs?

One of the key factors in MOOC design is to accept the fact that a MOOC is intended to have such a large number of students that it eliminates the instructor’s ability to provide feedback to most students. This, Chow explains, “makes the need for careful instructional design very important” (A. Chow, personal communication, October 14, 2014).

In Chow’s opinion, courses that are primarily lecture based, as is typical in many MOOCs, depart from sound instructional design principles. He explains that, while there is something to be said to listening to a lecture from a well-known, captivating speaker,

…no matter how captivating one is, the experience of listening to someone on a video screen as opposed to in person is a tremendous difference. On top of this, lecturing for very long is not very good as far as attention and retention. So, in this course we tried to do something different by applying instructional design principles to the MOOC” (A. Chow, personal communication, October 14, 2014).

The key element of instructional design is placing the learner at the center of what is trying to be

accomplished and designing the learning activity in such a way that is value added and allows the student to learn the material in a way that it is actually used. Instructional design principles require that courses are designed with

277

clear learning objectives and are implemented with activities that include learner engagement, cognitive processing, social interaction, opportunities to review and react, and include a diverse set of learning experiences. In the online context, these learning experiences often include multimedia, traditional reading, some basic assessment, and provide a social context of providing input and sharing ones experiences with others. According to instructional design principles, these elements together create a strong learning environment.

In the MOOC discussed in this case study, all of the instructional design elements noted above were included – video introductions and lectures, readings, assessments, hands on activities, and social interaction via online forums. Because it is essential that students have the opportunity to practice what they have learned, hands-on activities are included throughout each section of the course. Learners are asked to apply some hands on skills and post their results to the discussion board to encourage peer to peer feedback. To encourage social connections, Facebook, LinkedIn, and Twitter groups were created to encourage networking among students. In addition, because literature has found that student retention in courses tends to be higher when people have a start/end date that is formal, a cohort model was initially planned that would require students to have a particular start and end date. This plan, however, fell through in actual implementation. However, there are recommended guided experiences that begin every six weeks, though adherence to this guideline is not required and students may enroll and begin the course anytime they wish.

Implications

Systems thinking and systemic change were used as the guiding frameworks for understanding both the

macro organizational goals and micro issues dealing with design and development involved with creating and offering a MOOC. A critical facet of systems thinking is to ensure there is alignment between institutional goals and organizational elements (Kaufman, 1988). Additionally, it is highly recommended that a well-qualified team of both faculty and production specialists is assembled, properly compensated, and works together throughout a carefully implemented instructional design process such as the ADDIE framework that allows for a constant feedback cycle between users and the production team. This is not always an easy process, as Chow explained in a recent interview,

There is an inherent conflict between how a faculty member intends to deliver instruction and how the production process works. This is very important for those considering developing MOOCs. The production team must work closely with the faculty member in such a way that the faculty member doesn’t feel directed in a way that shifts the paradigm of what he is used to – a great deal of autonomy and creativity” (personal communication, October 14, 2014).

References Culatta, R. (2013). ADDIE model. Retrieved from http://www.instructionaldesign.org/models/addie. Defining a MOOC. (2013). Retrieved from http://wikieducator.org/OER_university/

eduMOOC_planning_group/MOOC_comparison Dewey, J. (1938). Logic: The theory of inquiry. New York, NY: Henry Holt. Fischer, G. (2014). Beyond hype and underestimation: identifying research challenges for the future of

MOOCs. Distance Education,35(2), 149-158. doi:10.1080/01587919.2014.920752 Grush, M. (2012). Seeking systemic change: Higher education in a digital, networked age. Campus Technology.

Retrieved from http://campustechnology.com/home.aspx Heller, N. (2013). LaptopU: Has the future of college moved online? The New Yorker, May 20, pp. 80–91. Johnson, L., Adams Becker, S., Estrada, V., Freeman, A. (2014). NMC Horizon Report: 2014 Higher Education

Edition. Austin, Texas: The New Media Consortium. Jongbloed, B., Enders, J., & Salerno, C. (2008). Higher education and its communities: Interconnections,

interdependencies, and a research agenda. Higher Education, 56, 303-324. Kaufman, R. (1988). Preparing useful performance indicators. Training & Development Journal, September 1988. Lewin, T. (2013). Master’s degree is new frontier of study online. The New York Times. August 18, pp. A-1 & A-

18. Nielsen, J., & Loranger, H. (2006). Prioritizing web usability. San Francisco, CA: Pearson Education. Murphy, T. H. (1999). A quantitative analysis of instructor-student verbal interaction in a two-way audio video

distance education setting. Journal of Agricultural Education, 40(3), 50-60.

278

Pappano, L. (2012, November 2). The year of the MOOC. The New York Times. Retrieved from http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html

Peterson, C. (2003). Bringing ADDIE to life: Instructional design at its best. Journal of Educational Multimedia & Hypermedia, 12(3), 227-241.

Sandeen, C. (2013). Integrating MOOCS into traditional higher education: The emerging “MOOC 3.0” era. Change, 45(6), 34-39. doi:10.1080/00091383.2013.842103

Seels, B., & Glasgow, Z. (1998). Making instructional design decisions (2nd ed.). Upper Saddle River, NJ: Merrill/Prentice Hall.

Skiba, D. J. (2013). On the horizon: The year of the MOOCs. Nursing Education Perspectives, 34(2), 136-137. Stein, K. (2013, December 5). Penn GSE study shows MOOCs have relatively few active users, with only a few

persisting to course end. Retrieved from https://www.gse.upenn.edu/pressroom/press-releases/2013/12/penn-gse-study-shows-moocs-have-relatively-few-active-users-only-few-persisti

Summerville, J. B. (2002). Taking ID online: Developing an online instructional design course. TechTrends, 46(4), 29-32.

Usability First. (n.d.) Introduction to user-centered design. Retrieved from http://www.usabilityfirst.com/about-usability/introduction-to-user-centered-design/

Waldrop, M. M. (2013). Campus 2.0: Massive open online courses are transforming higher education — and providing fodder for scientific research. Nature, 495(7440).

Wang, S., & Hsu, H. (2009). Using the ADDIE Model to Design Second Life Activities for Online Learners. TechTrends, 53(6), 76-81.

Westervelt, E. (2013, December 31). The online education revolution drifts off course. WFPL. Retrieved from http://wfpl.org/post/online-education-revolution-drifts-course

279