Large-scale Learning Analytics at TU Delft
-
Upload
claudia-hauff -
Category
Education
-
view
339 -
download
2
Transcript of Large-scale Learning Analytics at TU Delft
![Page 1: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/1.jpg)
Claudia Hauff
Web Information Systems, TU Delft
Large-scale Learning Analytics
![Page 2: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/2.jpg)
It’s data that is on the Web … Web data … lets find the Web Information Systems people!
✤ 40+ MOOCs ✤ 1+ Million enrollments ✤ From primary school to PhD level ✤ Lots of user data (click logs)
![Page 3: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/3.jpg)
Our goals
Data
Knowledge
Application to learning
Gain actionable insights into learner behaviours at scale. a. Data Science b. Big data processing
Increase our knowledge about learners by looking beyond the learning platform a. Web data analytics
Design technology interventions that enable adaptive learning at scale. a. Web data analytics b. Human-centered design c. Learning technologies
![Page 4: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/4.jpg)
Learner profiling beyond the MOOC platform
ACM WebScience 2016
Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform: Gaining Insights about Learners from the Social Web, ACM WebScience, pp. 15-24, 2016.
![Page 5: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/5.jpg)
Whythis research?
Learner
Before the MOOC
NOTHING
Engagement, retention, …
During the MOOC
NOTHING
After the MOOC
![Page 6: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/6.jpg)
Howto solve the problem?
We propose:
a deeper understanding about learnerscan be gained by exploring their traces on the Social Web.
![Page 7: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/7.jpg)
Whatresearch questions?
1 On what Social Web platforms can a significant fraction of MOOC learners be identified?
Are learners who demonstrate specific traitson the Social Web drawn to certain types of MOOCs? 2
To what extent do Social Web platforms enable us to observe (specific) user attributes
that are relevant to the online learning experience? 3
![Page 8: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/8.jpg)
Learner identificationacross Social Web platforms
edX learners
Email Login name Full name+ +
1. Explicit Matching
Profile images & links
Identification via emails
2. Direct Matching
Identification via profile links from Step 1
3. Fuzzy Matching
Search learners by their login & full names
Compare: 1. profile link2. profile image3. login & full names
![Page 9: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/9.jpg)
Social Web platformsinvolved in our work
![Page 10: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/10.jpg)
![Page 11: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/11.jpg)
Matching resultsfor 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%
![Page 12: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/12.jpg)
Matching resultsfor 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%
On average, 5% of learners can be identified on globally popular Social Web platforms.
![Page 13: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/13.jpg)
Learners on
Linkedin- Using job titles & skills to characterise learners
Spreadsheet MOOC- Software Engineer- Business Analyst- …
Design Approach MOOC- Co founder- UX designer- …
![Page 14: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/14.jpg)
Learners onStackExchange
- Functional Programming learners in StackOverflow
- To what extent do learners change their question/answering behaviour during and after a MOOC?
![Page 15: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/15.jpg)
Take-homeMessages
On average, 5% of learners from 18 DelftX MOOCscan be identified on 5 globally popular Social Web platforms. 1
Learners with specific traits prefer different types of MOOCs.2
Learners’ post-course behaviour can be investigated by using their external Social Web traces.3
![Page 16: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/16.jpg)
Learning Transfer: does it take place?
Best Paper Nominee at ACM Learning At Scale 2016
An Investigation into the Uptake of Functional Programming in Practice
Guanliang Chen, Dan Davis, Claudia Hauff and Geert-Jan Houben, Learning Transfer: does it take place in MOOCs?, ACM Learning At Scale, pp. 409-418, 2016.
![Page 17: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/17.jpg)
Whatis learning transfer?
Learning transfer is the application of knowledge or skills gained in a learning environment to another context.
![Page 18: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/18.jpg)
Whydo we care?
Learning transfer is a more important measure of learning in MOOCs than retention, success or engagement.
![Page 19: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/19.jpg)
FP101x
@flickr:christiaan_008
Course programming language: Haskell
Run as a typical video-lecture based MOOC
Assessment: 288 Multiple Choice questions
Introduction to Functional Programming
37,485 learners registered.41% engaged with the course. 5% completed the course.33% were active on GitHub (1.1M events).
![Page 20: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/20.jpg)
Whatdid we do?
FP101xlogs surveys coding
activities
3 months 2.5 years + 0.5 years
+ +
email address
Are changes made in a functional language?
![Page 21: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/21.jpg)
GitHub
10+ million registered users
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007long-term
large-scale
detailed
![Page 22: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/22.jpg)
detailed logs
code changes
project meta-data
![Page 23: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/23.jpg)
A sanity check
![Page 24: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/24.jpg)
Are “GitHub learners” different? GitHub
learnersNon-GitHub
learners
#Learners 12,415 25,070
Completion rate 7.71% 4.03%
Avg. time watching videos 49.1 min 27.7 min
Avg. #questions attempted 31.3 17.5
Avg. accuracy of learners’ answers 23.4% 12.9%
GitHub learners are more engaged than non-GitHub learners and exhibit higher levels of knowledge.
![Page 25: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/25.jpg)
Are “Expert learners” different? Expert GitHub
learnersNovice GitHub
learners
#Learners 1,721 10,694
Completion rate 15% 6.5%
Avg. time watching videos 78.6 min 44.4 min
Avg. #questions attempted 57.9 27.0
Avg. accuracy of learners’ answers 38.0% 21.1%
Expert learners are more engaged than Novice learners and exhibit higher levels of knowledge.
![Page 26: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/26.jpg)
To what extent do engaged learners exhibit learning transfer?
5-10% >30%10-30%<5%
![Page 27: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/27.jpg)
To what extent do engaged learners exhibit learning transfer?
5-10%
![Page 28: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/28.jpg)
Which type of learner is more likely to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated Extrinsically motivated
![Page 29: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/29.jpg)
Which type of learner is more likely to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated
![Page 30: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/30.jpg)
Which type of learner is more likely to display learning transfer?
Experienced Inexperienced
![Page 31: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/31.jpg)
Which type of learner is more likely to display learning transfer?
Experienced
![Page 32: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/32.jpg)
Which type of learner is more likely to display learning transfer?
High-spacinglearning routine
Low-spacinglearning routine
![Page 33: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/33.jpg)
Which type of learner is more likely to display learning transfer?
High-spacinglearning routine
![Page 34: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/34.jpg)
Learners who transfer quickly move to Scala
FP101x
![Page 35: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/35.jpg)
Conclusions
Most transfer learning findings from the classroom hold.
The observed transfer rate is low: 8.5%.
Learners quickly moved on after the course to industrially-relevant functional languages.
@flickr:torsten-reuschling
![Page 36: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/36.jpg)
From Learners to Earners: Enabling MOOC Learners to Apply their Skills and Earn Money in an
Online Market Place
IEEE Transactions on Learning Technologies
Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff and Geert-Jan Houben. Can Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in
an Online Market Place, IEEE Transactions on Learning Technologies.
![Page 37: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/37.jpg)
WhatMOOCs aim to educate the world. Most successful learners are already highly educated. Learners from developing countries are underrepresented.
is the problem?
![Page 38: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/38.jpg)
Whatis the problem?
EX101x: Data Analysis to the MAX()
![Page 39: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/39.jpg)
HowPay learner at scale: recommend tasks from online market places to learners that are relevant to the course material.
can we tackle it?
![Page 40: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/40.jpg)
Howcan we tackle it?
![Page 41: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/41.jpg)
What1) To what extent do online market
places contain relevant tasks? 2) Are learners able to solve
real-world tasks with high quality?
do we need to look at?
![Page 42: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/42.jpg)
Setup1) Weekly spreadsheet “bonus
exercises” drawn from Upwork (manually checked) in EX101x
2) Accuracy check 3) Quality check (code smells)
![Page 43: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/43.jpg)
Howare learners doing?
Good accuracy & quality.
![Page 44: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/44.jpg)
Built a workingrecommender.
Deployed in a MOOC by the end of October.
![Page 45: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/45.jpg)
Our goals one more time…
Data
Knowledge
Application to learning
Gain actionable insights into learner behaviours at scale.
Increase our knowledge about learners by looking beyond the learning platform
Design technology interventions that enable adaptive learning at scale.
![Page 46: Large-scale Learning Analytics at TU Delft](https://reader033.fdocuments.in/reader033/viewer/2022042723/587adaeb1a28ab542b8b5291/html5/thumbnails/46.jpg)
MOOCs are vital to bring higher education to the world. Lots of unexplored potential. Plenty of data. Many users.http://bit.ly/lambda-lab
Overall …