DECIPHERING THE MASSIVE OPEN ONLINE COURSES (MOOCS) ECOSYSTEM: WHAT
MAKES SUCCESSFUL MOOCS?
Ido Roll, Will Engle, Simon Bates!@hummus_monster @infology @simonpbates @CTLT_UBC!
Centre for Teaching, Learning, and Technology!University of British Columbia
1
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
Please raise your hand if you have pre-teen child(ren) at home.!
Please keep it up if you know where the following sentence is taken from:!
“The snow glows white on the mountain tonight”
2
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
LEARNING FROM VIDEO !IS POSSIBLE
3
If someone wants to learn, you cannot stop them.!@eric_mazur #STLHE2014
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
WE KNOW A LOT ABOUT UNIVERSITY TEACHING
What works?!
Small classrooms!
Interactivity!With material!With peers!With instructors!
Feedback!
…
4
MOOCs
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
THREE KEY QUESTIONS
5
Who are !our learners?
What defines !success?
What course design supports that?
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
CONTEXT
Four UBC courses (Spring/Summer 2013)!
Climate Literacy; Useful Genetics; Program Design; Game Theory!
330,000 learners registered. !
Five Edinburgh courses (Winter/Spring 2013)!
Introduction to Philosophy; Artificial Intelligence; Astrobiology; Nutrition; Critical Thinking.!
309,000 learners registered.!
https://www.era.lib.ed.ac.uk/handle/1842/6683!
6
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
VIDEO PRODUCTION
7
Aver
age
leng
th p
er v
ideo
(min
)
0
6
12
18
24
Average # of videos per week0 4 8 12 16
UBC 1UBC 2UBC 3UBC 4
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
THREE KEY QUESTIONS
8
Who are our learners?
What defines !success?
What course design supports that?
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
WHO ARE OUR LEARNERS?
General demographics consistent across courses and with Coursera.
9
Education level
% re
spon
ses
0
25
50
75
100
UBC 1 UBC 2 UBC 3 Coursera
High School DiplomaProfessional SchoolSome CollegeBachelor’s DegreeMaster’s DegreeDoctorate Degree
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
WHO ARE OUR LEARNERS?
General demographics consistent across courses and with Coursera.
10
Ethnicity
% re
spon
ses
0
25
50
75
100
UBC 1 UBC 2 UBC 3 Coursera
White or CaucasianSouth AsianEast AsianOther AsianBlack or African AmericanDecline to State
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
ROLE OF GEOGRAPHY
Does location matter? Are MOOCs “local”?
11
Country of Origin
% re
spon
ses
0
15
30
45
60
UBC overall Coursera Edinburgh
113.63.9
United StatesIndia CanadaUnited KingdomSpainBrazilGermany
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
ROLE OF GEOGRAPHY
Does location matter? Are MOOCs “local”?
12
Country of Origin
% re
spon
ses
0
15
30
45
60
UBC overall Coursera Edinburgh
43.95.2
United StatesIndia CanadaUnited KingdomSpainBrazilGermany
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
ROLE OF GEOGRAPHY
Break down by course. weighting by overall registration, population of country:!
How many more students are local, per 10,000 registrants, per capita?!
In Canada, !~150 additional local students!~500 Canadians overall
13
Extr
a lo
cal l
earn
ers
per 1
0,00
0 re
gist
rant
s, 1
,000
,000
resi
dent
s
0
2.5
5
7.5
10
UBC Edinburgh
2.0
1.0
55.7
3.3
5.54.8
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
THREE KEY QUESTIONS
14
Who are !our learners?
What defines success?
What course design support that?
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
WHAT DEFINES SUCCESS?
Definitions:!Visitors: log in at least once. Our denominator.
15
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
METRICS OF SUCCESS
Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.
16
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
METRICS OF SUCCESS
Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.!Learners: watch videos all the way through.
17
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
METRICS OF SUCCESS
Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.!Learners: watch videos all the way through. !Active learners: problem sets, discussion forums.
18
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
METRICS OF SUCCESS
Definitions:!Visitors: log in at least once. Our denominator.!Tasters: watch a video on Week 2.!Learners: watch videos all the way through. !Active learners: problem sets, discussion forums.
19
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
METRICS OF SUCCESS
Metrics:!
Attractiveness: convert Visitors to Tasters.!
Retention: convert Tasters to Learners!
Interactivity: convert Learners to Active Learners.
20
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
Visitors
Tasters Learners
Active Learners
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
METRICS OF SUCCESS
21
0
0.15
0.3
0.45
0.6
Attractiveness Retention Interactivity
48%54%
35%50%
31%29%
52%58%
36% 35%
57%
24%
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
Visitors
Tasters Learners
Active Learners
17k 13k 49k 85k 2k 3k 4k 16k
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
THREE KEY QUESTIONS
22
Who are !our learners?
What defines !success?
What course design support that?
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
WHAT AFFECTS CONVERSION RATES?
Correlate with:!
Average length per video!
Average length per week
23
Activities
Videos
Website
Week0 1 2 3 4 5 6 7 8
Visitors
Tasters Learners
Active Learners
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
ATTRACTIVENESS
Visitors -> Tasters: How many of those who visited the website watched Week 2 videos?!
By length per week: r = -0.72*, r² = 0.52!
24
% o
f Visi
tors
who
ar
e al
so Ta
ster
s
0%
20%
40%
60%
Overall legth of videos per week (min)0 45 90 135 180
UBCEdinburgh
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
ATTRACTIVENESS
Visitors -> Tasters: How many of those who visited the website watched Week 2 videos?!
By length per video: r = -0.33!
25
% o
f Visi
tors
who
ar
e al
so Ta
ster
s
0%
20%
40%
60%
Average length per video5 8 11 14 17
UBCEdinburgh
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
RETENTION
Tasters -> Learners: How many of those who watched @ Week 2 also watched @ Week 5?!
By length per week: r = -0.29!
26
% o
f Tas
ters
who
ar
e al
so L
earn
ers
0%
26.667%
53.333%
80%
Overall legth of videos per week (min)0 45 90 135 180
UBCEdinburgh
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
RETENTION
Tasters -> Learners: How many of those who watched @ Week 2 also watched @ Week 5?!
By length per video: r = 0.55, r² = 0.30
27
% o
f Tas
ters
who
ar
e al
so L
earn
ers
0%
20%
40%
60%
80%
Average length per video5 8 11 14 17
UBCEdinburgh
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
INTERACTIVITY
How many of the students who watch videos also submit problem sets?
28
% o
f vid
eo w
atch
ers
who
subm
itted
pr
oble
m s
ets
20%
30%
40%
50%
60%
70%
Week0 2 4 6 8
UBC 1UBC 2UBC 3UBC 4
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
SUMMARY
Local matters!
Courses have unique “signatures”!Structure, e.g., # and length of videos!
Engagement, e.g., % active learning.!
Overall length of videos is negatively correlated with retention!
But length of individual videos does not show such relationship. !
True for both universities
29
@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
DESIGN RECOMMENDATIONS
Long videos may be okay!
Especially later in the course!
Keep overall load reasonable.!
Active learning is not affected by week-by-week variations!
Two-stage courses: attractiveness and retention.
30
Top Related