ASSESSMENT TOOLS FOR ONLINE COURSES AND PROGRAMS

K A B E L S T A N W I C K S , U N I V E R S I T Y A T A L B A N Y

S U N Y L A 2 0 1 4

QUICK POLLS

How many people have taken an online course?

How many people have taught, or are currently teaching, an online course?

How many people are planning on teaching, or would like to teach, an online course in the future?

ONLINE EDUCATION

Over the last decade, online education has continued to expand

In 2011, 32% of students in higher education were taking at least one online class (Allen & Seaman, 2013)

Involves more than just using technology to deliver instruction

ASSESSMENT NEEDS

Valid and reliable assessment tools are needed for online courses and programs

Quality assurance is a major challenge faced by higher education (Shelton, 2010)

Understanding and using e-learning assessment tools helps us understand students and maximize their potential for successful learning! (Black et al., 2008)

COURSE EVALUATION: VIRTUAL HIGH SCHOOLYamashiro & Zucker (1999)

Evaluates:

• curriculum/content

• pedagogy

• course design

• assessment

COURSE EVALUATION: CHICO STATE

Chico State (2003, 2009): http://www.csuchico.edu/roi/

Evaluates:

• learner support and resources

• online organization and design

• instructional design and delivery

• assessment and evaluation of student learning

• innovative teaching with technology

• faculty use of student feedback

Rubric provides examples for baseline, effective, and exemplary courses

COURSE EVALUATION: CHICO STATE

COURSE EVALUATION: CHICO STATE

COURSE EVALUATION: CHICO STATE

COURSE EVALUATION: CHICO STATE

COURSE EVALUATION: CHICO STATE

COURSE EVALUATION: CHICO STATE

COURSE EVALUATION: QUALITY MATTERS

Quality Matters (2011): https://www.qualitymatters.org

Evaluates (using point-based rubric):

• course overview

• learning objectives

• Assessment

• instructional materials

• learner interaction and engagement

• course technology

• learner support

• accessibility

COURSE EVALUATION: QUALITY MATTERS

COURSE EVALUATION: BLACKBOARD

Blackboard Exemplary Course Program Rubric (2013): http://www.blackboard.com/resources/catalyst-awards/BbExemplaryCourseRubric_Nov2013.pdf

Evaluates:

• course design

• interaction and collaboration

• Assessment

• learner support

Rubric uses weighted points with details on scoring courses as incomplete, promising, accomplished, and exemplary within each category.

COURSE EVALUATION: BLACKBOARD

COURSE EVALUATION: RUBRIC COMPARISONChico State ROI: Easiest rubric to use, assessment or course

design, provides detailed information for ranking

Quality Matters: Good for peer review, lacks examples detailed in other rubrics

Blackboard: Very thorough, intended for peer review but can be used for self assessment our course design, more complex to use, provides detailed information for ranking

PROGRAM EVALUATION TOOLS

Lockhart & Lacy (2002): institutional readiness and administration, faculty services, instructional design and course usability, student readiness, student services, learning outcomes, retention

Survey to Assess Student Opinions of Distance Education (Chaney, 2007): course-specific experiences and general distance education experiences

Khan & Smith (2007): institutional, management, ethical, technological, interface design, pedagogical, resource support, evaluation

Quality Scorecard for the Administration of Online Education Programs (Shelton, 2010): institutional support, course development, teaching and learning, course structure, student support, faculty support, evaluation and assessment

ADVANCED TOOLS

Template Project (Tricker et al., 2001): identify gaps between students’ requirements for course and course’s performance

Roblyer & Wiencke (2003): evaluates student interaction focusing on social-rapport, instructional design, technology resources, learner and instructor engagement

PDPP Evaluation Model (Zhang & Cheng, 2012): evaluates planning, development, process, product

QUESTIONS

Kabel Stanwicks, kstanwicks@albany.edu

This presentation is available on Slideshare: http://ow.ly/xX95Z

Speaking notes for this presentation available at: https://db.tt/voJNFajI

ADDITIONAL RESOURCESAllen, E., Seaman, J. (2013, January). Changing Course: Ten Years of Tracking Online Education in

the United States. Sloan Consortium. 

Bates, T. (2012, August 5). What’s right and what’s wrong about Coursera-style MOOCs [Web log post]. Retrieved from http://www.tonybates.ca/2012/08/05/whats-right-and-whats-wrong-about-coursera-style-moocs/

Belanger, Y., & Thornton, J. (2013). Bioelectricity: A Quantitative Approach Duke University’s First MOOC (Report). Retrieved from http://dukespace.lib.duke.edu/dspace/handle/10161/6216

Black, E. W., Ferdig, R. E., & DiPietro, M. (2008). An Overview of Evaluative Instrumentation for Virtual High Schools. American Journal of Distance Education, 22(1), 24-45.

Blackboard (2013). Blackboard Exemplary Course Program Rubric. Retrieved from http://www.blackboard.com/

California State University, Chico (2009). Rubric for Online Instruction. Retrieved from http://www.csuchico.edu/

Chaney, E. H. (2006). The Development of an Instrument to Assess Student Opinions of the Quality of Distance Education (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (304934887). 

Chaney, E. H., Eddy, J. M., Dorman, S. M., Glessner, L., Green, B. L., & Lara-Alecio, R. (2007). Development of an Instrument to Assess Student Opinions of the Quality of Distance Education Courses. American Journal of Distance Education, 21(3), 145–164.

ADDITIONAL RESOURCESCross, S. (2013). Evaluation of the OLDS MOOC curriculum design course: participant perspectives,

expectations and experiences. Retrieved from http://oro.open.ac.uk/37836/

Downes, S. (2013, March 18). Evaluating a MOOC [Web log post]. Retrieved from http://halfanhour.blogspot.ca/2013/03/evaluating-mooc.html

Khan, B., & Smith, H. (2007). A Program Satisfaction Survey Instrument for Online Students. In B. Kahn (Ed.), Flexible Learning in an Information Society (pp. 320-337). Hershey, PA: Information Science Publishing.

Legon, R. (2013, April 25). MOOCs do not represent the best of online learning (essay). Inside Higher Ed. Retrieved from http://www.insidehighered.com/views/2013/04/25/moocs-do-not-represent-best-online-learning-essay

Lewin, T. (2013a, April 29). Adapting to Blended Courses, and Finding Early Benefits. The New York Times. Retrieved from http://www.nytimes.com/2013/04/30/education/adapting-to-blended-courses-and-finding-early-benefits.html

Lewin, T. (2013b, December 10). After Setbacks, Online Courses Are Rethought. The New York Times. Retrieved from http://www.nytimes.com/2013/12/11/us/after-setbacks-online-courses-are-rethought.html

Lockhart, M., & Lacy, K. (2002). An assessment model and methods for evaluating distance education programmes. Perspectives: Policy & Practice in Higher Education, 6(4), 98–104.

McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice. Siemens, C. G., & Downes, S. MOOC.

ADDITIONAL RESOURCESMOOC pedagogy: the challenges of developing for Coursera. (2012, August 8). Association for

Learning Technology Online Newsletter. Retrieved from http://newsletter.alt.ac.uk/2012/08/mooc-pedagogy-the-challenges-of-developing-for-coursera/

Pappano, L. (2012, November 2). Massive Open Online Courses Are Multiplying at a Rapid Pace. The New York Times. Retrieved from http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html

Parr, C. (2013, April 18). How was it? The UK’s first Coursera Moocs assessed. Times Higher Education. Retrieved from http://www.timeshighereducation.co.uk/news/how-was-it-the-uks-first-coursera-moocs-assessed/2003218.article

Quality Matters (2011). Quality matters rubric standards 2011-2013 edition. Retrieved from https://www.qualitymatters.org

Ripley, A. (2012, October 18). College Is Dead. Long Live College! Time. Retrieved from http://nation.time.com/2012/10/18/college-is-dead-long-live-college/

 Roblyer, M., & Wiencke, W. (2003). Design and Use of a Rubric to Assess and Encourage Interactive Qualities in Distance Courses. The American Journal of Distance Education, 17(2), 77-98.

Shelton, K. (2010). A Quality Scorecard for the Administration of Online Education Programs: A Delphi Study. Journal of Asynchronous Learning Networks, 14(4), 36–62.

ADDITIONAL RESOURCES

Shelton, K. (2011). A Review of Paradigms for Evaluating the Quality of Online Education Programs. Online Journal of Distance Learning Administration, 14(1).

Siemens, G. (n.d.). Announcing: MOOC Research Initiative [Web log post]. Retrieved from http://www.moocresearch.com/blog

Tricker, T., Rangecroft, M., Long, P., & Gilroy, P. (2001). Evaluating Distance Education Courses: the student perception. Assessment & Evaluation in Higher Education, 26(2), 165–177.

Uvalić-Trumbić, S., & Daniel, J. (2013). Making sense of MOOCs: The evolution of online learning in higher education. In Scaling up Learning for Sustained Impact (pp. 1-4). Springer Berlin Heidelberg

Yamashiro, K. & Zucker, A. (1999). An Expert Panel Review of the Quality of Virtual High School Courses: Final Report. Arlington, VA: SRI International. Retrieved from http://thevhscollaborative.org/sites/default/files/public/vhsexprt.pdf

Zhang, W. & Cheng, Y. L. (2012). Quality Assurance in E-Learning: PDPP Evaluation Model and its Application. International Review of Research in Open & Distance Learning, 13(3), 66–82.