Deconstructing Disengagement:Analyzing Learner Subpopulations in

Massive Open Online Courses

René

Kizilcec

Chris

Piech

Emily

Schneider

MOOCs (in this paper) are

instructionist + individualised

• 6-10 weeks long

• 2-3 hours of video lectures/week

• autograded assessments with regular

deadlines

• discussion forum

Massive Open Online Courses

Heterogeneous population:

Learners join from anywhere in the

world, at any age, for any reason

Defining Success for Open-Access Learners

Assessment scores are problematic:

• not comparable across courses

• not available for all learners because

test-taking is not aligned with learner

goals

Defining Success for Open-Access Learners

Completion rates are highly problematic:

• numerator = certificate earners, i.e. learners who take assessments

• denominator = o total enrolled? overestimate; indicator of

interest and not participation

o total active? how defined?

• ignore plurality of learner intentions

• no nuance about subpopulations to help us design interventions or customized course features

Process measures hold promise:

• conceptualize learning as an ongoing

set of interactions with learning objects

and other humans

• allow early detection and prediction

• indicate points for intervention

Defining Success for Open-Access Learners

Defining Success for Open-Access Learners

Completion rates

Assessment scores

Process measures

How to classify learners into

meaningful subpopulations?

Classification Criteria

Classification methods for MOOC subpopulations:

Universal – valid across multiple courses

Theory-driven – reflect the processes of learning

Parsimonious – based on small, meaningful feature set

Predictive – suggest likely outcomes

Dynamic – account for new information over time

Lens for Analysis

• Compare subpopulations

• Compare courses

The Data

Analyzed Three Courses

Who took these MOOCs?

A lot of data!

Gender skew

Interesting age group

HDI skew

Clustering

Sub-populations basis?

Engaged

Not Engaged

Engagement Ideal

Time

Enga

gem

ent

Coarse Engagement Labels

(T) On Track: Did the weekly assignment on

time

(B) Behind: Did the weekly assignment, but

finished after the due date

(A) Auditing: Watched videos but did not do

the assignment

(O) Out: Did not interact with the course,

either through videos or assignments

We were able to predict who would take the final AUC = 0.96

The Aggregate Class A = AuditingO = OutT = On TrackB = Behind

In this picture Out Is not to scale!

The Aggregate Class A = AuditingO = OutT = On TrackB = Behind

5k

In this picture Out Is not to scale!

The Aggregate Class A = AuditingO = OutT = On TrackB = Behind

In this picture Out Is not to scale!

7k

Example Student 1

Example Student 2

Example Student 3

Clustering Methodology

There were 21,108 paths in the

GS class

Four Prototypical Trajectories

Cluster!

(k-means of L1 norm)

Four Prototypical Trajectories

And?

The Four Prototypical Trajectories

Prototypical Trajectory 1: Completing

Prototypical Trajectory 2: Auditing

Prototypical Trajectory 3: Disengaging

Prototypical Trajectory 4: Sampling

Four Prototypical Trajectories

Consistent across three courses:

Auditing learners watch lectures throughout course, but

attempt very few assessments

Completing learners attempt majority of assessments offered

in course

Disengaging learners attempt assessments at beginning of the

course, but then sparsely watch lectures or disappear entirely

Sampling learners briefly explore course by watching a few

videos

Four Prototypical Trajectories

The other courses?

Four Prototypical Trajectories

Four Prototypical Trajectories

<suspense>

Four Prototypical Trajectories

Four Prototypical Trajectories

Same pattern in all classes

HS Composition [46k]C

om

ple

tin

g

Completing

Sampling

Disengaging

Auditing

UG Composition [27k]C

om

ple

tin

g

Dis

en

ga

gin

g

Sampling

Auditing

MS Composition [21k]C

om

ple

tin

g

Dis

en

ga

gin

g

Auditing

Sampling

Validation

Cluster Validation

• Different values of k (split by time)

• Including “assignment pass” (95%

overlap)

• Excluding “behind” (94% overlap)

• Silhouette of 0.8 (that’s pretty good)

• Pass the common sense test

High Level

Clustering

Engagement in

MOOCs

Four

Prototypical

Patterns

Results &

Recommendations

Comparing Trajectories

between Courses

Sampling

Disengaging

Completing

Auditing

Sampling

Disengaging

Completing

Auditing

Sampling

Disengaging

Completing

Auditing

HS

UG

GS

3.0 3.5 4.0 4.5 5.0

Overall Experience

Completing (and Auditing)

have best experience

Overall Experience

Identify subpopulations early

to customize course features

Sampling

Disengaging

Completing

Auditing

Sampling

Disengaging

Completing

Auditing

Sampling

Disengaging

Completing

Auditing

HS

UG

GS

0.1 0.51.0 2.0 4.0 7.0 10.0

Average Forum Activity

Completing learners are

most active on the forum

Discussion Forum

Reputation systems &

Social features

Causal relationship?

Geographical Distribution

Trend confirmed by top four participating countries

United States, India, Russia, United Kingdom

Sampling

Disengaging

Completing

Auditing

Sampling

Disengaging

Completing

Auditing

Sampling

Disengaging

Completing

Auditing

HS

UG

GS

2 4 6 8 10 12 14 16

Odds Ratio (Male/Female)

Female Completing learners underrepresented in advanced courses

Gender

Frame assessments to

minimize stereotype threat

Stereotype threat? Spencer et al., 1999

Future Directions

Future Directions Experiments

Collaboration and Peer Effects

Interface Customization

Targeted Interventions

Nuanced Analytics

Auditing: MOOC-as-a-resource vs. MOOC-as-a-class

Disengaging: Early prediction for intervention

Reasons to enroll and trajectories

Engagement trajectories for real-time analytics in MOOCs

Dashboard visualizations

Thank you!

Stanford Lytics Lab lytics.stanford.edu

Office of the Vice Provost for Online Learning

Roy Pea, Clifford Nass, Daphne Koller

Our LAK reviewers

Reference

S. Spencer, C. Steele, and D. Quinn. Stereotype threat and women’s math

performance. Journal of Experimental Social Psychology, 35(1):4–28, 1999.

More info?

René Kizilcec kizilcec@stanford.edu

Chris Piech piech@cs.stanford.edu

Emily Schneider elfs@cs.stanford.edu

Stanford’s Learning Analytics Group:

Lytics Lab lytics.stanford.edu

Paper: http://goo.gl/OSX72