Learning design and learning analytics: building the links

EMMA Summer School, Ischia, July 2015

Rebecca Ferguson

• The Open University: largest in UK• Informal learning: iTunes, YouTube…• MOOCs on FutureLearn, OpenLearn

and elsewhere• Making use of big data for

more than 40 years• Learning analytics research / events• LACE project

2

Lead on MOOC evaluation at The Open University, UK

http://www.laceproject.eu/

Workshop overview

09.45 IntroductionLinking learning analytics, learning design and MOOCs

10.05 Group workFocusing on learning outcomes in MOOCs

10.25 Plenary

Discussing how these data might be used to support learning

10.40 Group work

Analytics, step by step

11.00 Plenary

What would MOOC learners and educators need to know to use these analytics?

11.15 Workshop end

3

You can view and download these slides athttp://www.slideshare.net/R3beccaF

What are learning analytics?

High-level figures

Brief overviews for internal and external reports

Academic analytics

Figures on retention and success, for the institution to assess performance

Learning analytics

Use of big data to provide actionable intelligence for learners and educators

4

Educators use analytics to

• Monitor the learning process• Explore student data• Identify problems• Discover patterns• Find early indicators for success• Find early indicators for poor marks or drop-out• Assess usefulness of learning materials• Increase awareness, reflect and self reflect• Increase understanding of learning environments• Intervene, supervise, advise and assist• Improve teaching, resources and the environment

5

Dyckhoff, A L, Lukarov, V, Muslim, A, Chatti, M A, & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at LAK13.

Learners use analytics to

• Monitor their own activities and interactions

• Monitor the learning process

• Compare their activity with that of others

• Increase awareness, reflect and self reflect

• Improve discussion participation

• Improve learning behaviour

• Improve performance

• Become better learners

• Learn!

6

Dyckhoff, A L, Lukarov, V, Muslim, A, Chatti, M A, & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at LAK13.

Analytics example: UK schools

7

• Aligned with clear aims• Huge and sustained effort• Agreed proxies for learning• Clear and standardised visualisation• Driving behaviour at every level

BUT• Stressed, unhappy learners• Analytics with little value for learners or educators• Omission of key areas, such as collaboration

Analytics example: Course Signals

Developed at Purdue University

8

Arnold, K E, & Pistilli, M (2012). Course Signals at Purdue: Using Learning Analytics To Increase Student Success. Paper presented at LAK12, Vancouver, Canada.

Analytics example: SNAPP

Network analysis

9

Analytics example: iSpot

Heading

10

Learning design in MOOCs

●Puts the learning journey at the heart of the design process●Provides a set of tools and information to support a learner-

activity based approach ●Helps to show the costs and performance outcomes of

design decisions●Enables the sharing of best practice●Helps MOOC designers to choose and integrate a coherent

range of media, technologies and pedagogies

●Enables a consistent and structured approach to review and analytics

11

Mor, Y, Ferguson, R, & Wasson, B. (2015). Editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action. British Journal of Educational Technology, 46(2), 221-229.

MOOC learning design tools

• MOOC design template

• MOOC planner

• MOOC map

• Journey planner

12

Design template analytics

13

Learning outcome How this is assessed

1. Be able to define an ecosystem.

2. Have joined the iSpot community and obtained identifications for animals, plants or fungi.

1. Multiple choice. Week 1, question 5

2. Self report.

Analytics

1. How many attempted that question? How many got it right 1st / 2nd / 3rd time? How many followed the link back to resources?

2. Access to iSpot data. Use of MOOC hashtag. Persistence over time. Ethical implications of tracking off-site.

Short description of course and learning outcomes

MOOC planner

• Delivered• Reflection• Collaboration• Conversation• Networking• Browsing• Assessment

14

Blocking out types of learning activity

Conole, Gráinne. (2010). Learning design – making practice explicit. Paper presented at ConnectEd, Sydney, Australia. http://cloudworks.ac.uk/cloud/view/4001

MOOC planner analytics

Delivered Content (reading, watching, listening and observing)Analytics: amount of content viewed, dwell time

Reflection (thinking, considering and reflecting)Analytics: returns to the same material, reflection exercises completed, quality of reflection

Collaboration (constructing, collaborating, defining and engaging)Analytics: collaboration exercises completed, quality of collaboration

Conversation (debating, arguing, questioning, discussing…)Analytics: number and length of contributions, quality of discussion

Browsing (exploring, searching, finding and discovering)Analytics: Number of click-throughs to external links, number of visits, number of resources

Assessment (answering, presenting, demonstrating, critiquing…)Analytics: Assessments completed, scores, dwell time on hints, persistence in answering questions

15

MOOC map analytics

●How long did you expect learners to spend on these key elements?

●How long did learners actually spend on the key elements

●How many missed out these elements?

●How many jumped ahead to these elements?

●Which types of element are consistently (un)popular?

●How many left the MOOC at these points?

16

The MOOC map identifies key elements of the course

Ass

imila

tive

Info

rmat

ion

Han

dlin

g

Pro

duct

ive

Exp

erie

ntia

l

Ada

ptiv

e

Com

mun

icat

ive

Ass

essm

ent

Org

anis

atio

n

0

100

200

300

400

500

600

Min

ute

s

MOOC journey planner analytics

17

Relationships between tools, resources, activities & narrative

A framework for data collection

Analytics to solve problems

Analytics could filter discussions or group learners

18

You have been actively engaged in the discussions, which is excellent, thank you, but with more than 23,000 participants it

means that our responses and comments risk getting lost.

This will be primary school material for some of you and exactly the opposite for others. It is just not possible to tailor the

material to each of you […]Introduction to Forensic Science:University of Strathclyde

Start with the pedagogy

• How do people learn?

• How can we use data to facilitate that process in our MOOC?

• Which elements are learners struggling with?

• Which sections engage them the most?

• What prompts them to ask questions?

• Are they finding assessment challenging?

• What misconceptions have learners shown?

• Are there any accessibility issues?

• How can analytics be used to obtain desired learning outcomes?

19

Learning analytics and design

Learning design – helping to identify useful analytics●What do learners need to know in order to

network, collaborate, browse or reflect?●What do educators need to know to support them?

Learning design – helping to identify gaps in the data●What data do we need to collect? Learning design – helping to identify gaps in our toolkit●Which design elements can we look at easily?●Which ones still pose problems?

Learning design – helping to frame & focus analytics questions●What did they learn?… in relation to learning outcomes●Were they social?... when they were collaborating●Did they share links?... when encouraged to browse●Did they return to steps?... when encouraged to reflect

20

Making the links

Group activity20 minutes

Visit FutureLearn and register if necessaryhttps://www.futurelearn.com/

Select a MOOC currently open for registration.

Look at the introductory material and the first week – What are the expected

learning outcomes?

How will learners know they have achieved these?

21

Plenary15 minutes

What types of learning outcome are specified?

In what ways are learners assessed or could they be assessed?

What sorts of data and analytics could be used to support learners?

What sorts of data and analytics could be used to support

educators? 22

Group activity20 minutes

Take the first week of your chosen MOOC

Classify each step in terms of learning activity:(delivered content, reflection, collaboration,

conversation, browsing or assessment)

Note how long the step would be likely to take

Discuss which types of analytic would be useful to learners

and to educators

23

Plenary15 minutes

Share your findings

Which types of data and analytic could be used to support these types of

learning activity?

What would the educators need to know?

What would the learners need to know?

24