Moving Forward on Learning Analytics

A/Professor Deborah West,

Charles Darwin University

Ms Nicole Wall, Blackboard

Today’s session

• Brief background

• Review the information gathered on the business questions from last year

• Present an analysis of which of these questions can be answered via specific Analytics for Learn reports

• Review a set of reports to check their validity in relation to the use case

• Explore the development of ‘good’ questions which can be used to further refine reporting.

3

‘Learning Analytics is about collecting traces that learners

leave behind and using those to improve learning’ (Duval, 2012)

4

Our Focus

• collecting traces that students and staff leave behind and using those to improve the student experience

5

Different types of data and affordances

• Functionality of LMS

• Analytics packages for the LMS

• Data warehouses

6

Benefits of Learning Analytics

Able to provide data to:

• Create a more structured and effective intervention to improve student outcomes

• Understand workload planning

• Value add over time through student tracking

• Improve support, target training workshops

• Identify successful strategies and evaluate teacher effectiveness

• Strategy and planning

7

Question Themes

• Curriculum/course design and improvement

• Student behaviour

• Improve retention

• Resourcing/planning at institutional level:

• Academic actions/behaviour

8

Curriculum/Course Design Questions

• How active are students in course content overall and at an individual content level?

• What types of content are included in courses? How does this relate to what is most actively used and the results students achieve?

• Can I couple student’s ratings of content/activities with their use of that content / activities with their results?

• Is there a correlation between different course ‘styles’ (e.g. instructivist vs. constructivist) and student achievement within a given course or faculty?

9

Student Behaviour Questions

• When and how are students engaged with the LMS and associated tools?– what times/lengths of day– spread of access throughout academic calendar, particularly key

milestone dates?– what types of devices and where from?

• When did a student last access email? When did a student last access Wi-Fi?

• Is there correlation between messaging and test attempts?

10

Retention Questions

• What kind of pre-enrolment interactions lead to “better” outcomes for our students?

• Analytics for identifying, monitoring, managing and responding to students at risk.

• What are the indicators?• Who are the most at risk?• What interventions are more likely to be effective?• What trends can we see in terms of at risk students

11

Institutional level planning/resourcing

• Gauging impact of new initiatives/introduction of new technologies

• Where are students connecting with “the university” in their learning journey?

• Is the time and/or money/effect that has been put into something worthwhile? Are there particular things which are over or underrepresented?

12

Academic Behaviours

• Which academics and which “schools” need support to become better/fuller users of Bb Learn?

• What’s the frequency or % of academic attention per student within a subject?

• How engaged are academics with the full Bb Learn toolset?

• How much time instructors are spending and where the peaks of activity are?

• Usability and Ease of Access by lecturers.

13

Miscellaneous Questions

• Is this data: reliable, meaningful, and accurate?

• If we have the “answer” are we morally, ethically and legally required to act?

• How much inference can be drawn from the level of engagement a student has with an online environment?

14

Getting Answers from Analytics for Learn

Questions Themese #Qs Report Available

Report Can be Built

No Direct Match

Curriculum/Course design & improvement

19 47% 21% 31%

Student behaviour 7 42% 57%

Improve Retention 10 100%

Resourcing/planning at institutional level

8 38% 13% 50%

Academic actions/behaviour 6 50% 33% 17%

Questions from Different Perspectives 20 60% 40%

15

Writing good questions for your reports

• Focus on Questions That Are Aligned With Your Business Strategy

• Keep your questions specific and related to opportunities for applying the insights

• Be definitive - One of the most valuable questions asked is how can you engage your students better? But what is your definition of ‘engagement’.

16

Writing good questions for your reports

Grade & Outcome Results

CourseAttributes

Student Attributes

Final Grades

Student Information

System

Reports

Instructor Attributes

User Activity Data

CourseItem Data

17

Writing good questions for your reports

• What are you trying to measure?– Interactions, Accesses, Minutes, Submissions

• How are you trying to measure it?– Average, Percentage, Count

• To what?– Blackboard, Courses, Items

18

Writing good questions for your reports

• How do you want to filter or group the information– Course Attributes– Instructor Attributes– Student Attributes– Performance– Time Frame

19

Writing good questions for your reports

How many logins occurred in Semester 1 and 2 by Faculty

Which Faculty had the highest logins in the last 2 semester

How is engagement with Blackboard changing

✔

✗

✔

20

Activity

• Refine the question– Be specific; incorporate the measures and dimensions that are available– Consider the purpose of the question – what you want to know and for

what purpose– Might break down into more than one

• On the butchers paper draw what the report would look like– E.g. graph, table, pie chart

21

But…

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”

Albert Einstein