Open-source LA and “what the student does”

Dr Phill Dawson, MonashDr Tom Apperley, Melbourne

Structure

• Until 3:30pm• You will get a break• I will talk a bit (background concepts)• I will show some tools• You will talk a lot• You write an algorithm• You will probably argue about ethics

Dr Phillip (Phill) Dawson

• Lecturer in Learning and Teaching at Monash

• Led a small grant in learning analytics

• Interested in how academics make decisions

Who are you?

• LA researchers?• Educational designers?• LMS administrators?• University managers?• Faculty-based academics?• Academic developers?• Non-university?

“the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (SoLAR)

What LA do you use currently in everyday teaching? (ie not research)

• Nothing?• Reports? (eg ‘who has logged in?’; ‘who has

submitted assignment one?’)• Dashboards?• Something else?

“Who is struggling or not engaging with my

course?”

Free,modular,configurableextendable,open-sourcelearning analytics block for teachersto identify students at risk

“the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and

optimizing learning and the environments in which it occurs” (SoLAR)

Learning happens because of…

1.What the student is?2.What the teacher does?3.What the student does?

Biggs 1999

“learning is what the student does”

Activity - pairs

• What do students do to learn in your context?– Long list– Specific– Verb stems– Online and offline– Effective and ineffective– Deep and strategic

Which of these can easily be captured by LA?

Which of these can’t possibly be captured by LA?

Sci-fi LA vs Real LA

Flickr user sndrv http://www.flickr.com/photos/sndrv/4519088620/ CC-BY

Flickr user dpape http://www.flickr.com/photos/dpape/2720632752/ CC-BY

Typical Open-Source LA Tools

• Gather data on student use of parts of LMS• No integration with Student Management

Systems• Teacher dashboards• Reports• Some synthesis• Inconsistent design and language– At code + UI levels

State of the Actual:Free/open/built-in LA

• Open-source tools– Engagement Analytics (documentation, demo vid)– Gismo– Analytics and Recommendations

• Vendor-supplied reports – (eg Desire2Learn)

Modular ‘indicator’ architecture(so you can make additional indicator plugins)

Attendance Downloads

CompletionFacebook

• If an assignment is very late it is riskier than if it is just a little late

• If an assignment is worth a greater percentage then it is riskier than if it is worth a small percentage

• If an assignment is past its due date and not submitted then it is riskier than if it was submitted late

How does the assessment indicator work?

for each assignment, quiz, lesson in the course whose due date has passed{

daysLateWeighting = ((number of days late) - overdueGraceDays) / (overdueMaximumDays - overdueGraceDays)

assessmentValueWeighting = (value of this task) / totalAssessmentValue

if (daysLateWeighting > 1){

daysLateWeighting = 1}else if (daysLateWeighting < 0){

daysLateWeighting = 0}

if (task was submitted late){

risk = risk + daysLateWeighting * assessmentValueWeighting * overdueSubmittedWeighting

}else if (task was not submitted){

risk = risk + daysLateWeighting * assessmentValueWeighting * overdueNotSubmittedWeighting

}}

Activity: specify a new ‘indicator’ of learning in your context

• What it does in one sentence• Procedure to give a number between 0% (no

risk) and 100% (high risk)– Words?– Pictures?– Flowchart?– Algorithm?

• What variables could we tweak?• How important is this indicator?

Ramsden 1992, p. 187

“From our students’ point of view, the

assessment always defines the actual

curriculum”

“Students can, with difficulty, escape

from the effects of poor teaching…”

Boud 1995, p. 35

“…they cannot (by definition if they

want to graduate) escape the effects of

poor assessment”

Boud 1995, p. 35

Teaching/Learning

AssessmentObjectives/Outcomes

LA are only as good as curriculum

• Training statistical LA on assessment or retention outcomes ≠ learning

• Teacher needs to be in control of LA use and specify learning

• LA makes it more difficult to “escape the effects of bad teaching”

“whether through denial, pride, or

ignorance, students who need help the most are least likely

to request it”

Martin & Arendale 1993 p. 2

It’s the end of week 2 and student X hasn’t ever logged in.

What do we do?

How can we make follow-up effective?

• Personal or robotic?• Paint a grim picture?• Refer on or see personally?• Specific guidance

It’s the week of the census and modeling suggests student X is 70% likely to fail.

What do we do?

How can we make follow-up ethical?

• Do students have a– Right to try (and fail?)– Right to give up– Right to be strategic

• Will draconian measures lead to LMS-farming?• Student-identified triggers

It’s week 10 and student X already has 60% of the course grade but hasn’t logged in for two weeks.

What do we do?

Discussion and close:the near-future for open-source

learning analytics

Extra time options

• Live demonstration of Engagement Analytics tool

• Further specifying an indicator into pseudocode for a developer

• Develop strategies for following up students at risk

• Discuss student views of analytics tools• Discuss open-source

References and sources• SoLAR definition of LA• Memes are from Quickmeme• NetSpot Innovation Fund logo courtesy of NetSpot. (You should apply for an open-

source development grant through them.)• Biggs, J. (1999). What the Student Does: teaching for enhanced learning. Higher

Education Research & Development, 18(1), 57-75. doi: 10.1080/0729436990180105

• Boud, D. (1995). Assessment and learning: contradictory or complementary. In P. Knight (Ed.), Assessment for Learning in Higher Education (pp. 35-48). London: Kogan Page.

• Ramsden, P. (1992). Learning to teach in higher education. London: Routledge.• Martin, D., & Arendale, D. (1993). Supplemental Instruction: Improving First-Year

Student Success in High-Risk Courses The Freshman Year Experience: Monograph Series (2nd ed., Vol. 7). Columbia, SC: National Resource Center for the First Year Experience and Students in Transition, University of South Carolina.