Deploying Open Learning Analytics at a National Scale
-
Upload
michaeldwebb -
Category
Education
-
view
122 -
download
3
Transcript of Deploying Open Learning Analytics at a National Scale
Lessons from the Real World
Oct 2016 Deploying Open Learning Analytics at National Scale
Michael WebbDirector of Technology and Analytics, Jisc
Eitel J.M. Lauría, PhDProfessor of Information Technology & Systems, Marist College
Kate ValentiVice President of Operations, Unicon
Agenda
» Strategic View
› A brief introduction to Learning Analytics
› National issues in the UK
» Technical View
› Open architecture
› Predictive modeling
» Implementation View
› Trends and tactics from the field
» Discussion
A brief introduction to Learning Analytics
Our working definition...
What do we mean by Learning Analytics?
» The application of big data techniques such as machine based learning and data mining to help learners and institutions meet their goals:
› For our project:
– Improve retention (current project)
– Improve achievement (current project)
– Improve employability (current project)
– Improve learning design (later stage)
Learning Analytics stages get progressively “smarter”
Basic Analytics
What has happened
Automated Analytics
What is happening
Predictive Analytics
What might happen
A Strategic OverviewNational and institutional strategic issues
National issues in the UK: Retention
» 16-18 Education:
› 178,100 students aged 16-18 failed to finish (2012/13)
› costing UK £814 million a year
» Undergraduates:
› 8% of undergraduates drop out in their first year of study
› This costs universities up to £33,000 per student
National issues in the UK: Differential achievement
» Parental background and ethnicity impact achievement:
National issues in the UK: Differential achievement
2/03/2016 The case for Learning Analytics
» Which behaviours are associated with lower than expected academic achievement?
National issues in the UK: Teaching excellence framework
Technical Overview What does the architecture look like?
Jisc’s Learning Analytics project
Three core strands
Learning Analytics architecture and
serviceToolkit Community
Jisc Learning Analytics
Toolkit: Code of practice
2/03/2016 The case for Learning Analytics
Jisc Learning Analytics architecture
What
» Building a national architecture
» Defined standards and models
» Implementation with core services
Why?
» Standards mean models, visualisations and so on can be shared
» Lower cost per institutions through shared infrastructure
» Lower barrier to innovation – the underpinning work is already done
What do we mean by an open architecture?
» All APIs published, and process for engaging in their development
» Open standards and definitions
› Data Models and Definitions Creative Commons.
› Developed openly on Github
» All core elements open source or open specification (eg creative commons)
» Freedom to implement both commercial and open solutions as the non-core elements
Data Collection
DataStorageand Analysis
Presentation and Action
Jisc Learning Analytics open architecture: core
Alert and Intervention system
Staff Dashboards Consent Student App
LearningAnalytics Processor
LearningRecords Warehouse
Student Records VLE Library
DataExplorer
Self Declared Data
Meanwhile, in the US...Learning Analytics Processor: Predictive Modeling Framework
Motivation: Alarming Stats in 2010
36% 4-year completion rate across all four-year institutions in the US
21% for Black students
25% for Hispanic students
58% 6-year completion rate for four-year institutions
40% for Black students
49% for Hispanic Students
41% 25-to-34 Year-Olds with an Associate Degree or Higher (US ranked 12th among 36 developed nations)
Sources: U.S. Dept. of Education, Postsecondary Education Data System (2009) CollegeBoard, Advocacy & Policy Center, The Completion Agenda 2011 Progress Report
Open Academic Analytics Initiative @ Marist
EDUCAUSE Next Generation Learning Challenges (NGLC) grant
Funded by Bill and Melinda Gates Foundation
Use machine learning to find patterns in large datasets as means to predict student academic performance.
Create “early alert” framework:
• Predict academically at-risk students in initial weeks of a course
• Deploy intervention to improve chances of success
Based on Open ecosystem for academic analytics
• Sakai Collaboration and Learning Environment
• Pentaho Business Intelligence Suite (Kettle + Weka)
• Collaboration with commercial vendors (IBM SPSS Modeler)
Learning Analytics Processor @ Marist: Early Alert How does it actually work?
(binary classification problem)
Hardware Platform: IBM zEnterprise 114 with BladeCenter Extension (zBX) Virtualized Servers: 64 bit, 16/32 GB RAM Linux Red Hat
Extraction, Transformation &
Loading
Scoring(predictions on new student data using library of persisted learnt classifiers)
Predictive Model Building
(classifiers learnt from data)
New StudentData
(early in the Semester)
Prediction of at-risk studentsSingle node architecture
Relational Storage
Intervention
SATs, GPA,HS ranking, Course size,Course grade(target feature)
Age, gender,ethnicity,income level
SessionsResourcesLessonsAssignmentsForumsTests
Partialcontributionsto final grade
Logistic RegressionSVMsNaïve BayesJ48 Decision Trees
Student Academic
Data
Student Demographic
Data
LMS Event Log Data
LMS Gradebook Data
Learning Analytics Processor @ Marist: Early Alert New Iteration: Cluster Computing Architecture
New StudentData
(early in the Semester)
Prediction of At-risk students
Intervention
Scoring(predictions on
new student data using library of persisted learnt
classifiers)
Hardware Platform (Dev) Linux VMs (32GB RAM) running on IBM PureFlex System
Distributed Storage (HDFS)& Processing
Extraction, Transformation &
Loading
Predictive Model Building
(classifiers learnt from data)
Job
Sc
hed
ulin
g
Student Academic Data
Student Demographic Data
LMS Event Log Data
LMS Gradebook Data
Library Data
Student Engagement Data
Social Network Data
and more …
CURRENT
FUTURE
Scales well for Big Data use cases(more volume & variety)
Logistic RegressionRandom ForestsNaïve Bayes
Promising Outcomes
Phase II: Cluster Computing Accuracy Recall FP Rate
Marist
- 3 semesters, 25K records each 86% 87% 14%
North Carolina State University
- 3 semesters, 160K recs each 81% 77% 18%
- 3 semesters, online, 85K recs each 80% 82% 19%
Jisc Project:
• 260,000 records
• 4 institutions (Aberystwyth University, University of Gloucestershire, Cardiff Metropolitan University, University of Greenwich)
• Results due in December 2016
Implementation ViewTrends and tactics from the field
Jisc project in numbers
101 35 24 12 (+ 20)
Discovery activity assesses institutional readiness
– Goal: to assess institutional readiness (think organizational maturity)
» Measured on 26 factors crossing organizational and technical considerations
» Approximately 60% of the first 11 institutions are ready to implement Learning Analytics technology solutions
Source: Moving the Red Queen Forward, Educause Review September/October 2016, Dahlstrom
Varied activities show adoption flexibility
Profile Aim Activity Data sources
Russell Group Retention of widening participation + support for students to achieve 2.1 or better
Discovery + Tribal Insight + Learning Locker
Moodle + Student Records
Research led University Retention, improve teaching, empowering students
Discovery + OpenSource Suite + Student App
Moodle + Attendance+ Student Records
Teaching led University with WP mission
Retention - requirement to make identifying students more efficient so they can focus on interventions
Tribal Insight + Learning Locker
Blackboard + Attendance + Student Records
Research led University Student engagement Discovery + Student app + Learning Locker
Moodle + Student Records
Teaching Lead Understanding of how Learning Analytics can be used
Discovery + Technical Integration
Moodle
Organizational Trends
» Top level support is critical» Change culture makes things easier» Red tape is real (in policy management)» Academics looking for evidence-based results
It’s (almost) all about change management
& Tactics
» Convene a Learning Analytics committee (include students)» Identify champions and advocates» Adjust existing policies rather than creating new» Pilot the solution
It’s (almost) all about the data
& Tactics
» Perform data audits and quality checks (early and often)» Look for “all inclusive” offerings (predictions)» Look for integration options
Technical Trends
» Institutional infrastructure for data collection requires improvement» Unified data management desired but not realized» Data quality issues are common» Integration with existing infrastructure a challenge» Doing more with the same technical staff
Keep it simple, snowflakes
& Tactics
» Simplify for pilot; add complexity later» Overall, only add components you don’t already have» Flexibility (by institution and by vendors) is key
Pilot Trends
» Customizations are required to meet institutional needs» Ditto for integrations» Data gathering effort is considerable» Did we mention data quality?
Q&A
2/03/2016 The case for Learning Analytics
Interested in more detail?
» Data quality challenges» Predictive model research» Data collection, UDD, xAPI recipes, use of standards» Spark, ETL flows» ?
Michael [email protected]
Eitel J.M. Lauría, [email protected]
Kate [email protected]