Supporting Higher Education to Incorporate Learning Analytics

OEB‘16SHEILAWorkshop30thNovember2016

JeffHaywood,DraganGašević,AdolfoRuizCalleja,MarenScheffel

Supporting Higher Education to Incorporate Learning Analytics

ProgrammeforOEB‘16SHEILAWorkshop,30thNovember20161000–1005 Welcome&introduc3ons–ProfJeffHaywood(UniversityofEdinburgh)

1005–1110 Threepresenta3ons&discussion“Overview–LearningAnaly3cs&SHEILAProject”ProfDraganGasevic(universityofEdinburgh)“SHEILAProject–interviewfindings”ProfAdolfoRuizCalleja(UniversityofTallinn)“SHEILAProject–conceptmapping”MarenScheffel(OpenUniversityoftheNetherlands)

1110–1130 Coffeebreak

1130–1300GroupworkonapplyingtheRapidOutcomeMappingApproach(ROMA)toassessingins3tu3onalreadinessfordeployingLA&plenarydiscussion

1300 Close

LearningAnaly3cs&SHEILAProject-Overview-

DraganGašević@dgasevic

November30,2016OEBSHEILAWorkshopBerlin,Germany

hap://sheilaproject.eu/

Educa3onalLandscapeToday

Growingneedforeduca3onAc3velearning

Educa3ondefunding

Educa3onalLandscapeToday

Growingneedforeduca3onAc3velearning

Educa3ondefunding

Feedbackloopsbetweenstudentsandinstructors

aremissing/weak!

LEARNINGANALYTICS

Learningenvironment

Educators

LearnersStudent

Informa3onSystems

Blogs

Videos/slides

Mobile

Search

Educators

Learners

Networks

StudentInforma3onSystems

Learningenvironment

Blogs

Mobile

Search

Networks

Educators

LearnersStudent

Informa3onSystems

Learningenvironment

Videos/slides

Learninganaly3cstopics

Learninganaly3cstopics

Veryfewins3tu3on-wideexamplesofadop3on

Adop3onchallenge

Policiesfor

learninganaly3csprac3ce

Adop3onchallenge

Equalengagementwithdifferentstakeholders

Adop3onchallenge

Trainingopportuni3estouselearninganaly3cs

Adop3onchallenge

Monitoringimpactandlearningfromtheuse

hap://sheilaproject.eu/

Policydevelopmentframework

Usedbydifferenthighereduca3onins3tu3onsRecognizesins3tu3onspecificdrivers

Promoteslearninganaly3csforforma3veassessment

hap://sheilaproject.eu/

hap://sheilaproject.eu/

Macfadyen,L.,Dawson,S.,Pardo,A.,Gašević,D.,(2014).Thelearninganaly3csimpera3veandthesociotechnicalchallenge:Policyforcomplexsystems.Research&Prac+ceinAssessment,9(Winter2014),17-28.

hap://sheilaproject.eu/

Projectac3vi3es

Literaturereview

hap://sheilaproject.eu/

Tsai,Y-S.,Gašević,D.(2016).Adop+onofLearningAnaly+csinHigherEduca+on:State,Challenges,andPolicies(Execu+vesummary).SHEILAProjectReport,hap://bit.ly/sheila_lr_es

Projectac3vi3es

LiteraturereviewInterviewswithdecisionmakers

hap://sheilaproject.eu/

Projectac3vi3es

LiteraturereviewInterviewswithdecisionmakers

Groupconceptmappingwithexperts

hap://sheilaproject.eu/

Projectac3vi3es

LiteraturereviewInterviewswithdecisionmakers

GroupconceptmappingwithexpertsSurveysandfocusgroupswithstudentsandteachingstaff

hap://sheilaproject.eu/

Expected3meline

Policydevelopmentframework(v1)–June2017Fourins3tu3onalpolicies(v1)–June2017

hap://sheilaproject.eu/

Expected3meline

Policydevelopmentframework(v1)–June2017Fourins3tu3onalpolicies(v1)–June2017

Policydevelopmentframework(v2)–March2018Fourins3tu3onalpolicies(v2)–March2018

hap://sheilaproject.eu/

Interestedinjoining?

Associatepartnersasearlyadoptersofthepolicydevelopmentframeworkarewelcome

hap://sheilaproject.eu/

Manythanks!

hap://sheilaproject.eu/

Interviews in SHEILA project

Adolfo Ruiz-Calleja, Kairit Tammets, Yi-Shan Tsai, Jeff Haywood, Maren Scheffel, Pedro Muñoz Merino and Dragan Gasevic

30th, November 2016

Index

32

■ Methodological aspects

■  Topics of the interviews

■ Results

■  Conclusions

Index

33

■ Methodological aspects

■  Topics of the interviews

■ Results

■  Conclusions

Methodology: ROMA

34 Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analytics imperative and the sociotechnical challenge: Policy for complex systems. Research & Practice in Assessment, 9(Winter 2014), 17-28.

Methodology

35

■  Purpose:

■  Sampling:

To collect opinions from decision makers in higher education institutions regarding the adoption of Learning Analytics.

•  Policy makers •  Vice-rectors •  Study departments •  Academic offices •  IT departments

Methodology

36

■  65 interviews

■  52 institutions

■  12 countries

Index

37

■ Methodological aspects

■  Topics of the interviews

■ Results

■  Conclusions

Topics

38

■  LA projects and their goals

■  LA strategy

■  Progress and achieved goals

■  Challenges

■  Ethics and privacy

Index

39

■ Methodological aspects

■  Topics of the interviews

■ Results

■  Conclusions

Results: LA projects and goals

40

■  Exploratory objectives

■  Very often LA projects are related to research

■ Diversity of interests

•  Sense of uncertainty •  What can LA do for my institution? •  What can our data tell us?

•  UK: correlation between behavioural patterns and learning outcomes

•  Spain: support teachers and improve their performance •  Baltics: improve students retention

Results: LA strategy

41

■  For some institutions (UK, Latvia, Finland) LA is part of their wider strategy

■ Others follow a bottom-up approach: some projects without connecting backbone structure

■ Managers become more interested on LA

Results: Achieved goals

42

■  Too early phase in many institutions

■  LA is considered useful

■ Awareness of LA, better understanding of challenges and improvement of data culture

■  In-house software is typically used •  Vendors offer Dashboards or systems to detect drop-out

Results: Challenges

43

■  Cultural barriers

■  Little technical experience

■ Human resources

•  Data-driven and not question-driven •  Traditional institutional culture •  Lack of awareness •  Lack of data culture

•  Many institutions do not feel ready for maintaining a data management infrastructure

•  Little experience on how to deal with online environments

•  Lack of time •  Priorities

Results: Ethics and privacy

44

■  Ethical use of data remains a big challenge

■  Privacy laws are very diverse

■  Ethical issues open technical problems

•  Although not all of them understand it as a problem (Estonia) •  Clear need of ethical guidelines and solutions

•  e.g. Germany and Austria are much more restrictive than UK or Estonia

•  It is very difficult to avoid processing the data of one specific student if he does not allow it

Index

45

■ Methodological aspects

■  Topics of the interviews

■ Results

■  Conclusions

Conclusions

46

Conclusions

47

■  European universities are in an early stage of adoption of LA

■ Bottom-up approach from researchers, and awareness of policy makers

■  Several barriers have been detected, including ethics

■  There are important differences between countries

Interviews in SHEILA project

Adolfo Ruiz-Calleja, Kairit Tammets, Yi-Shan Tsai, Jeff Haywood, Maren Scheffel, Pedro Muñoz Merino and Dragan Gasevic

30th, November 2016

GroupConceptMapping

MarenScheffelOpenUniversiteitNederland

maren.scheffel@ou.nl@m_a_s_c

GroupConceptMappingPhase1:

Brainstorming

Phase2: Sor3ng

Phase3: Ra3ng

Anessen3alfeatureofahighereduca3onins3tu3on'slearninganaly3cspolicyshouldbe...

PointMap1 2

3

4 5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

2930

31

32

33

34

35

36

37

38

39 40

41

42

43

44

45

46

47

48

49

50

51

5253

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

7879 80

81

82

83

84

85

8687

88

89

90

91

92

93

94

95

96

97

9899

ClusterReplayMap

ClusterReplayMap

ClusterReplayMap

ClusterMap

1.privacy&transparency

2.roles&responsibiliZes(ofallstakeholders)

3.objecZvesofLA(learnerandteachersupport)

4.risks&challenges

5.datamanagement

6.research&dataanalysis

Ra3ngMap–Importance

1.privacy&transparency

2.roles&responsibiliZes(ofallstakeholders)

3.objecZvesofLA(learnerandteachersupport)

4.risks&challenges

5.datamanagement

6.research&dataanalysis

ClusterLegendLayerValue15.08to5.2725.27to5.4635.46to5.6545.65to5.8455.84to6.03

Ra3ngMap–Ease

1.privacy&transparency

2.roles&responsibiliZes(ofallstakeholders)

3.objecZvesofLA(learnerandteachersupport)

4.risks&challenges

5.datamanagement

6.research&dataanalysis

ClusterLegendLayerValue13.79to4.1224.12to4.4534.45to4.7844.78to5.1155.11to5.44

Ra3ngLadderGraph

importance ease

privacy&transparency

privacy&transparency

risks&challenges

risks&challenges

roles&responsibili3es(ofallstakeholders)

roles&responsibili3es(ofallstakeholders)

objec3vesofLA(learnerandteachersupport)

objec3vesofLA(learnerandteachersupport)

datamanagement

datamanagement

research&dataanalysis

research&dataanalysis

3.79 3.79

6.03 6.03

r=0.66

GoZone–Privacy&Transparency

1 720 31

43862

10172445

64 65

8892

91556

6074

87

34

69

96

6.08

5.44

3.12

ease

3.83 6.03 6.59

importance

r=0.45

GoZone–Privacy&Transparency

1 720 31

43862

10172445

64 65

8892

91556

6074

87

34

69

96

6.08

5.44

3.12

ease

3.83 6.03 6.59

importance

r=0.45

2.transparency,i.e.clearlyinformingstudentsofhowtheirdataiscollected,usedandprotected88.acleardescrip3onofdataprotec3onmeasurestaken10.acleardescrip3onofdatausage17.beingclearaboutthepurposeforcollec3oncertaintypesofdata24.alignedwithdataprotec3onregula3ons(ins3tu3onal,na3onal,interna3onal)34. toassurethatthecollecteddataisusedonlyforthepurposeofimprovinglearningandinstruc3on96.anagreementbetweenlearners,teachersandpolicymakersonregula3ngaproperuseofdata

GoZone–Roles&Responsibili3es

5

38

62

11

1922

33

39 48

7091

25

28

37

40

55

61

6627

47 49

6.08

4.72

3.12

ease

3.83 5.48 6.59

importance

r=0.26

GoZone–Roles&Responsibili3es

5

38

62

11

1922

33

39 48

7091

25

28

37

40

55

61

6627

47 49

6.08

4.72

3.12

ease

3.83 5.48 6.59

importance

r=0.26

55.beingclearaboutthepurposeoflearninganaly3cs61.aclearar3cula3onofrolesandresponsibili3eswhenitcomestotheuseofins3tu3onaldata39.topromotebroadadop3onoflearninganaly3csbyspecifyingsuppor3veregula3onsandcaselaw

GoZone–Objec3vesofLA

3

80

84

99

12

7898

3550

93

4244

586376

79

94

6.08

3.79

3.12

ease

3.83 5.44 6.59

importance

r=-0.12

GoZone–Objec3vesofLA

3

80

84

99

12

7898

3550

93

4244

586376

79

94

6.08

3.79

3.12

ease

3.83 5.44 6.59

importance

r=-0.12

63.touselearninganaly3csforimprovingthequalityofteaching58.toallowforproac3vebehaviorofstudentstowardstheireduca3on,withsetngpersonalgoals44.tosupportlearnerslearningtolearnandimproveskills98.toencouragemeaningfulrela3onshipsbetweenstudents

GoZone–Risks&Challenges

18

41

8326

46

97

53

59

90

95

6.08

4.16

3.12

ease

3.83 5.52 6.59

importance

r=0.52

GoZone–Risks&Challenges

18

41

8326

46

97

53

59

90

95

6.08

4.16

3.12

ease

3.83 5.52 6.59

importance

r=0.52

90.todrawaaen3ontowhatthe"bigpicture"endgoalofthedatauseis59.todemarcateclearlybetweendifferentusesofanaly3csinins3tu3onalsetngs95.toensurethebenefitstostudentsoutweightherisks53.toensureadequaterepresenta3onduringhigh-stakesdecisionprocesses18.integratedwitharewardsystemfortheuseofevidence-basedimprovementstoteachingandlearning41.todiscouragestudentsandfacultyfromgamingthesystem

GoZone–DataManagement

4

14

89

23

67

71

85

6.08

4.5

3.12

ease

3.83 5.16 6.59

importance

r=-0.54

GoZone–DataManagement

4

14

89

23

67

71

85

6.08

4.5

3.12

ease

3.83 5.16 6.59

importance

r=-0.54

23.todefineclearrulesforcollabora3onwithotherresearchers

GoZone–Research&DataAnalysis

6

30

57

6872

75

77

81

8

32

527382

1351

1621

29

36

54

6.08

4.24

3.12

ease

3.83 5.08 6.59

importance

r=0.27

GoZone–Research&DataAnalysis

6

30

57

6872

75

77

81

8

32

527382

1351

1621

29

36

54

6.08

4.24

3.12

ease

3.83 5.08 6.59

importance

r=0.27

57.toassurethatinstruc3onalinterven3onsarebasedonwell-studiedandempiricallyvalidatedanaly3calmethodsandalgorithms75.toencouragethedevelopmentofdashboardswithmeaningfulandunderstandableoutcomes68.torecognisethelimita3onsoflearninganaly3cs72.toprovideclearexamplesofhowtointerpretdata,especiallywhentheremaybeinterac3ons81.to(re)evaluatecon3nuouslytheanaly3csbeingapplied54.toenhancetheconnec3onbetweentheacademicandthesocietyandindustry,showingwhatisbeingdoneattheins3tu3onsisrelatedtotheeduca3onandtraining21.tomakeexplicitmathsfrominnova3vedevelopment/useofanaly3csinindividualclassroomstoscalingup

Thank You!MarenScheffelOpenUniversiteitNederlandmaren.scheffel@ou.nl@m_a_s_c