JISC RSC London Workshop - Learner analytics
-
Upload
james-ballard -
Category
Education
-
view
486 -
download
1
description
Transcript of JISC RSC London Workshop - Learner analytics
Learning AnalyticsWhat? How? Why?
James Ballard
jameslballard
JamesBallard2
@jameslballard
Overview
What are Learning Analytics?
Learner Engagement - a metric for learning
Preparing institutions – tools and skills
Infinite Rooms
Open Discussion
What are learning analytics?Who are they for?
Activity 1 - Introduction
4
Learning AnalyticsWhat are they?
Not quite big dataIn 2012 we created 2,500,000,000,000,000,000 (2.5
quintillion) bytes of data every day
Annual Moodle log data 5Gb
Learning AnalyticsType of Analytics Level or Object of Analysis Who benefits
Learning Analytics Course-level: social networks, conceptual development, discourse analysis, intelligent curriculum
Learner, faculty
Departmental: predictive modelling, patterns of success/failure
Learners, faculty
Academic Analytics Institutional: learner profiles, performance of academics, knowledge flow
Administrators, funders, marketing
Regional (state/provincial): comparisons between systems
Funders, administrators
National and International National governments, education authorities
Siemens and Long (2011)
Common focus• Identifying learners at-risk of drop-out from
the course• Identifying momentum/crisis points
Retention• Predicting final exam success• Predicting future performance (e.g. school ->
university)Performance
• Quantitative views of activity• What are learners doingActivity• Usually linked to a bench-marking of staff
performance• Learning design patterns
Course• What types of things are learners doing• Learner engagement as a metric/proxyEngagement
Small groupsList some examples of what learning providers are measuring or might want to measure.
Activity 2 - Examples
Retention Performance Activity Course Engagem
ent
9
Learner EngagementA metric for learning
EngagementCommon activities such as checking announcements, viewing grades and uploading assignments represent little time investment from the user and may not be useful indicators of engagement.
MacFadyen and Dawson (2012)
Engagement
Engagement has emerged as an alternative view of the learner experience that can enrich the often reductionist language of performance, skills and competence.
HEA, Trowler and Trowler (2010)
Engagement ProcessEngagement is the new metric that supersedes previous linear metaphors, through a developmental process of discovery, evaluation, use, and affinity.
Haven (2007)
Small groupsTag previous examples within the engagement process.
Activity 2 - Examples
Involvement
• The presence of a learner within the institution including data such as physical or virtual visits
Interaction
• Provides a depth of understanding: where involvement measures touches, interaction measures actions.
Intimacy
• Helps understand sentiment and affection; the most common way to collect this type of data is through interviews or surveys.
Influence
• Determines the likelihood of the individual recommending learning to others and contributing to local culture(s).
InvolvementThe presence of a learner within the institution including data such as physical or virtual visits.
Overall Activity
Locations
Time of day
InvolvementThe presence of a learner within the institution including data such as physical or virtual visits.
Overall Activity
Locations
Time of day
InteractionProvides a depth of understanding: where involvement measures touches, interaction measures actions.
Activity types
Action analysis
Connectivity maps
Conole (2007)
InteractionProvides a depth of understanding: where involvement measures touches, interaction measures actions.
Activity types
Action analysis
Connectivity maps
IntimacyHelps understand sentiment and or affection; the most common way to collect this type of data is through interviews or surveys.
Learning Power
Self-theory
Motivated Strategies for Learning Questionnaire, MSLQ
Self-determination theory
Deakin Crick, Broadfoot, and Claxton (2004)
IntimacyHelps understand sentiment and or affection; the most common way to collect this type of data is through interviews or surveys.
Learning Power
Self-theory
Motivated Strategies for Learning Questionnaire, MSLQ
Self-determination theoryRehearsal Elaboration Organisation Self-Regulation Critical Thinking
0
1
2
3
4
5
6
MSLQ
Pre Post
Pintrich (1990)
InfluenceDetermines the likelihood of the individual recommending learning to others and contributing to local culture(s).
Social Network Analysis
Distributed Cognition
Collective Intelligence
Pathway of ParticipationDawson (2010)
InfluenceDetermines the likelihood of the individual recommending learning to others and contributing to local culture(s).
Social Network Analysis
Distributed Cognition
Collective Intelligence
Pathway of Participation
School Leader Network
Harré (1983)
22
Preparing InstitutionsEmpowering environments for learning
You are what you measure
Measure what you get
Metrics are based on the data that is easiest to extract/access, and what you don’t measure is lost.Get what you measure
Anything you measure will impel a person to optimize his score on that metric. Data is not neutral Don’t be surprised if people find ingenious and destructive ways in how they get there. For example, standardised assessment produce kids who perform well on these tests but can falter when asked to demonstrate their knowledge of the same material in a different way
‘Incremental change is not enough. You have to drive large-scale change by changing the environment in which people work’
– Kevin Bonnett, Deputy Vice Chancellor Student Experience
JISC Report
MMU Review
Open Discussion
What types of skills are required by e-learning teams?Do they already exist?
Activity 1 - Introduction
Analytics Process
Collection Storage Cleaning Integration Analysis Presentati
on
CETIS Analytics Series (2012)
Open Source Tools
Data Storage
MySQL / PostgreSQL Apache Hadoop HP VerticaData Mining
Pentaho Rapid Miner Social Network Analysis Gephi Visualisation Google Visualisation d3.js InfoVis Toolkit
Investing in staff experimentation with low cost components from a range of traditions may be a more prudent initial move, even if the most effective tool subsequently turns out to be a ready-made suite.
Data Mining
Algorithm Usage Purpose
Step regression Used for binary classification (0,1)• Select a
parameter• Assign a weight• Calculate value
Predicts simple binary results such as is a student at-risk?
Logistic regression Used for binary classification (0,1)
Same as above but more conservative
J48/C4.5 Decision trees (Quinlan, 1993)
Tries to find optimal split in variables
Good when data splits into groups
JRip Decision rules Find the “best” path and make this a rule until no sensible paths are left and set these to otherwise.
Good when multi-level interaction are common
K* Instance based classifiers
Predicts data based on neighbouring points.
Good when data is very divergent
Random Forest
Classification is used when one wants to predict something (label) which is categorical and not a number.
28
Infinite RoomsLearner enhanced technology
Research Project
ScopeWeb dashboards based on engagement process accessing a data warehouse model developed from Activity Theory. Utilises new and existing analytics and supports multiple learning design approaches.Objectives1. How can student activity help identify
and promote effective teaching practices?
2. Understand the role that analytics can play in learning design, feedback and assessment.
3. Explore how student contributions can provide dynamic indications of success.
If patterns of nonparticipation (disengagement) are to be disrupted an improved conceptual framework may be necessary.
Activity AnalysisEngeström’s (1987, 1999) approach allows us to overcome oppositions between activity and communication and highlight subject-community relations.
Modelling pedagogy with Activity Theory
Stevenson (2008)
http://goo.gl/vOuiqp
Exposing ActivityThe intention of this is to reveal the nature of the system, allowing designers (e.g. teachers) to evaluate the system in the wider context of their teaching and learning practice.
Data Model
Dimension
Fact
Action Post to forum
Tool Forum
Instance Discussion topic
User Oliver Twist
Role Student
Course Introduction to English
Date 02/10/2013
Time 9:45
System Moodle
Enables multi-dimensional tagging to explore data from different perspectives.
Data Capture
Things a learner doesActions• Submissions• Quiz attempts • Forum posts
Feedback to the learnerIntervention
s• Targets• Grades • Assignment feedback
Recognising learningAchieveme
nts• Course completions• Badges• Certificates
How learning is perceivedSurveys• Attitudes to learning/technology• Satisfaction survey
What types of things can we capture.
CodingOne can then begin to distinguish the possible actions that are generated through the use of tools from the operations needed to access them and code these via learning design theories.
Activity TypesApply learning design models to learner data.
Conole (2007)
VisualisationExplore different visualisations of the same data set for different insights.