Katrice A. Hawthorne, PhD

Karen Vignare, PhD

Cristi Ford, PhD

A Framework for Decision-Making:

Developing an Outcomes-Based

Evaluation Plan for Adaptive Learning

dLRN 2015 Conference

October 16 - 17, 2015

Stanford, CA

◼Undergrad & Grad Enrollment: 87,000◼Demographics:

▪47% of the total enrollment comprised of minority students (34% African American)

▪20% of students are Pell eligible

◼Largest public online open access university ◼Primarily online, but serving 50% military

including on bases throughout world

University of Maryland University College

◼74% of undergraduate students enrolled in UMUC’s stateside programs work full-time.◼54% of students are working parents.◼Median age for students enrolled in UMUC’s

stateside undergraduate programs is 31 years old.

University of Maryland University College

“ An approach to instruction and remediation that uses technology and accumulated data to provide customized program adjustments based on an individual student’s level of demonstrated mastery.”

Waters, J. K. (2014, April 16). The great adaptive learning experience. Campus Technology. Retrieved from http://www.campustechnology.com

Defining Adaptive Learning

◼2010, Carnegie grant with CMU for Open Learning Initiative (cognitive tutors)

◼2014, launched three vendor tools for piloting (Cogbooks, Realizeit, and EdReady)

◼2014, Next Gen Courseware subgrantee to Stanford OLI and Lumen Learning

UMUC History with Adaptive Learning

6

UMUC Adaptive Research Scope

“The systematic acquisition and assessment of information to provide useful feedback about some object” (program, policy, technology, etc.).

Defining Evaluation

Trochim, W. M. K. (2006). Introduction to Evaluation. Retrieved from http://www.socialresearchmethods.net

Continuous Improvement Cycle

Evaluation Logic Model

What We THINK We Know

10

• Accurately predict persistence

• Influence persistence through interventions

• Predict which students need more help

• Non-cognitive factors are an important predictor of success/retention

Results - Course Completion

Results - Grade Distribution

Study Time and Student Achievement

◼Moderate correlation between study time and score gain in solving equations, r(26) = .458, p = .019.

◼Moderate correlation between study time and score gain in percents and real numbers, r(25) = .472, p = .017.

◼Strong correlation between study time and score gain in whole numbers, fractions, and mixed numbers, r(26) = .501, p = .009.

Score Gain

Student Perceptions of Adaptive

Student Comments

“I know so much more than I did before and this is an accomplishment for me in math. I have always stayed away from math until this class. This class helped me to move through math in a way I did not see me doing just weeks ago.”

“I really appreciated the A.I. It assesses my knowledge level as I complete questions and adjust accordingly allowing me to progress at my own pace.”

“It presented the lecture in bite size, easy to digest chunks of information and provided quizzes to assess comprehension with immediate feedback on answers.”

What We Have Learned

◼Learning is not linear

◼Combine learning science and data science

◼Effective, flexible faculty and rich feedback are integral

◼Not enough content in some areas

◼Mapping and course organization are important

What We Want to LEARN

Questions & Contact

Katrice A. Hawthorne, Associate Director, Evaluation and ResearchCenter for Innovation in Learning and Student SuccessUniversity of Maryland University College240-684-2875katrice.hawthorne@umuc.edu

Karen Vignare, Vice ProvostCenter for Innovation in Learning and Student SuccessUniversity of Maryland University College240-684-2610karen.vignare@umuc.edu