Download - Using Analytics to Inform "Tactical" Design Decisions

Using Analytics to Inform “Tactical” Design Decisions

Hae Okimoto, University of HawaiiCasey Sacks, Colorado Community College System

Karen Swan, University of Illinois SpringfieldKaren Vignare, University of Maryland University College

July 8, 2014

Predictive Analytics Reporting (PAR)

• National• Non-Profit• Multi-Institutional• Collaborative

Student Success & Institutional Effectiveness

About the PAR Framework

• established, growing non-profit collaborative focused on using existing institutional data to improve institutional effectiveness & student outcomes

• funded by Bill & Melinda Gates Foundation 2011-2014• managed at the Western Interstate Commission for

Higher Education (WICHE)• engagement with > 39 forward thinking US institutions• small, high functioning team with partner, subject and

domain expertise

• a “big data” analysis effort to identify drivers related to retention & progression, and to inform student loss prevention

• PAR member institutions voluntarily contribute de-identified student records to create a single federated database

• identifying scalable approaches to student success through common data definitions, common outcome measures, common definitions of interventions


reusable predictive

models

common definitions of terms

student level watch lists for targeted

interventions

multi-Institutional collaboration

measurable Intervention results

PREDICT

ACT

RESULTS

common definitions of interventions

scalable cross-institutional improvements


Institutional Partners Founding Partners (since 2011): American Public University SystemColorado Community College SystemRio Salado CollegeUniversity of Hawaii SystemUniversity of Illinois SpringfieldUniversity of Phoenix

New Members (as of Oct 2013): Northern Arizona University Kaplan University Excelsior College University of North Dakota

Implementation Partners (since 2012):Ashford UniversityBroward CollegeCapella UniversityLone Star College SystemPenn State World CampusSinclair Community CollegeTroy UniversityUniversity of Central FloridaUniversity of Maryland University CollegeWestern Governors University

predictives help focus on the highest need; but they don’t solve the problem

• Benchmarks understand how institutions’ student outcomes compare to their peers by scaling a multi-institutional database for benchmarking and research purposes.

• Models identify which students need assistance, by using in-depth, institutional specific predictive models which are unique to the needs and priorities of member institutions based on their specific data.

determine the best way to address areas of weakness identified in benchmarks and models by scaling and leveraging a member and literature validated framework for examining interventions within and across institutions (SSMx)

• Interventions

CCCS redesign of Reading, English & Math

• Loss points for students in the traditional sequence

• Charge committee to redesign developmental education

• Workgroup, policy, grant support, implementation

CCCS College Composition and Reading (CCR)

• From up to 6 courses to preparation after a single semester

• Integrated reading and English• Most students in the ALP type model• Extra support and structure for students with

lowest placement profiles

CCCS Math

• From 4 courses to selecting a pathway for preparation– STEM (transfer path), non-STEM (transfer path),

non-transfer math• Structure for corequisite instruction of college

and developmental math

University of Hawaii – Community College Developmental Math

• 75% of students from public school start at a community college

• 85% of entering community college students not prepared for college level math

• 61% enroll in remedial/developmental math• 56% successfully complete a course• 3-4 courses in developmental course sequence

The Issue - LCC

Students earning C or better in developmental math

Acceleration

Emporium Redesign

• Learning & practice time are individualized & optimized

• Software-based assistance is always available

• In-person assistance is available during scheduled class times & open lab

• High mastery standards are balanced with repeatable quizzes & tests

Success Rates

Redesign

Legacy

Theoretical Support

• Institutional Departure– Academic difficulties– Inability to integrate educational & occupational

goals– Failure to engage in social & intellectual life

• Tinto & Bean’s Model of Student Integration– Social– Academic

Using Data to Make Tactical Decision

University of Maryland University College• Investment in Office of Analytics (& IR)• Outside partnerships (PAR, Civitas Learning) &

grants (Kresge transfer data repository)• Executive level steering committee & Cross-

Functional Interventions team• Commencing a Rapid Action Team (yes, RAT!)

on student success

The Synergy of Student Analytics Projects

KresgeTransfer PAR

Civitas Learning

18

Student Success

Interventions Team

Advising

Academic Departments

Provost Office

Analytics

Inst. Research

Student Services

19

Benchmarking with PAR

• UMUC Denise Nadasen Co-chairing (with UIS) • Gain a better understanding of how to

integrate multi-institutional data• Learned how to manage a large-scale data

project• Developed data-sharing partnerships with

nationally known institutions

Interventions Accounting 220 & Accounting 221

Test ControlSuccessful Course Completion 72%* 58%*Term GPA 2.52* 2.10*Reenrollment 78% 72%Change in GPA 0.31 0.07

Key Findings

• Students participating in tutoring had a significantly higher rate of successful course completion.

• Students participating in tutoring had a significantly higher term GPA.

• The test group reenrollment rate was six percentage points higher but not statistically significant.

Interventions

Messaging• Designed by Cross-functional intervention

team• Using Civitas risk score for students in gateway

courses• Results are mixed—Fall term saw test

reenrollment rate increase but all other terms mixed

Jumpstart—New Onboarding

• Four Week, one credit—diagnostics, mission statement, career planning, academic plan

Key Findings (Fall & Spring, awaiting summer)• There is a significant difference between JumpStart

Completers and the Control Group for Semester GPA, Successful Course Completion, and Reenrollment.

• There is not a significant difference between All JumpStart Students and the Control Group for Semester GPA and Successful Course Completion.

• There is a significant difference between All JumpStart Students and the Control Group for Reenrollment.

Student Success Matrix (SSMX)

• an organizational structure that helps institutions inventory, organize & conceptualize interventions aimed at improving student outcomes

What is the Student Success Matrix?

• categorizes interventions along two major dimensions – known predictors of retention/progression & timing of intervention in context of its delivery

• other aspects addressed include functionality, return on investment (ROI), focus, and institutional level

©PAR Framework 2014

learner characteristics

learner behaviors

fit/feelings of belonging

other learner support

course/program characteristicsinstructor

behaviors

time connection entry progress completionpredictors

Why Use the Student Success Matrix?Provides a common structure for categorizing and quantifying student supports across an institution

• alignment of interventions to predictors• review of timing of interventions• Identifying gaps & redundancies in student support• prioritizing interventions• illuminating opportunities for impact measurement

Comparable categorization across institutions allows for intervention benchmarking and sharing what works with specific categories of ‘at risk students.

©PAR Framework 2014

Automated SSMX Tool

[email protected]