SHEILA Project - Workshop Slides Online Educa Berlin 2016
-
Upload
lace-project -
Category
Education
-
view
213 -
download
0
Transcript of SHEILA Project - Workshop Slides Online Educa Berlin 2016
Supporting Higher Education to Incorporate Learning Analytics
OEB‘16SHEILAWorkshop30thNovember2016
JeffHaywood,DraganGašević,AdolfoRuizCalleja,MarenScheffel
Supporting Higher Education to Incorporate Learning Analytics
ProgrammeforOEB‘16SHEILAWorkshop,30thNovember20161000–1005 Welcome&introduc3ons–ProfJeffHaywood(UniversityofEdinburgh)
1005–1110 Threepresenta3ons&discussion“Overview–LearningAnaly3cs&SHEILAProject”ProfDraganGasevic(universityofEdinburgh)“SHEILAProject–interviewfindings”ProfAdolfoRuizCalleja(UniversityofTallinn)“SHEILAProject–conceptmapping”MarenScheffel(OpenUniversityoftheNetherlands)
1110–1130 Coffeebreak
1130–1300GroupworkonapplyingtheRapidOutcomeMappingApproach(ROMA)toassessingins3tu3onalreadinessfordeployingLA&plenarydiscussion
1300 Close
LearningAnaly3cs&SHEILAProject-Overview-
DraganGašević@dgasevic
November30,2016OEBSHEILAWorkshopBerlin,Germany
hap://sheilaproject.eu/
Educa3onalLandscapeToday
Growingneedforeduca3onAc3velearning
Educa3ondefunding
Educa3onalLandscapeToday
Growingneedforeduca3onAc3velearning
Educa3ondefunding
Feedbackloopsbetweenstudentsandinstructors
aremissing/weak!
LEARNINGANALYTICS
Learningenvironment
Educators
LearnersStudent
Informa3onSystems
Blogs
Videos/slides
Mobile
Search
Educators
Learners
Networks
StudentInforma3onSystems
Learningenvironment
Blogs
Mobile
Search
Networks
Educators
LearnersStudent
Informa3onSystems
Learningenvironment
Videos/slides
Learninganaly3cstopics
Learninganaly3cstopics
Veryfewins3tu3on-wideexamplesofadop3on
Adop3onchallenge
Policiesfor
learninganaly3csprac3ce
Adop3onchallenge
Equalengagementwithdifferentstakeholders
Adop3onchallenge
Trainingopportuni3estouselearninganaly3cs
Adop3onchallenge
Monitoringimpactandlearningfromtheuse
hap://sheilaproject.eu/
Policydevelopmentframework
Usedbydifferenthighereduca3onins3tu3onsRecognizesins3tu3onspecificdrivers
Promoteslearninganaly3csforforma3veassessment
hap://sheilaproject.eu/
hap://sheilaproject.eu/
Macfadyen,L.,Dawson,S.,Pardo,A.,Gašević,D.,(2014).Thelearninganaly3csimpera3veandthesociotechnicalchallenge:Policyforcomplexsystems.Research&Prac+ceinAssessment,9(Winter2014),17-28.
hap://sheilaproject.eu/
Projectac3vi3es
Literaturereview
hap://sheilaproject.eu/
Tsai,Y-S.,Gašević,D.(2016).Adop+onofLearningAnaly+csinHigherEduca+on:State,Challenges,andPolicies(Execu+vesummary).SHEILAProjectReport,hap://bit.ly/sheila_lr_es
Projectac3vi3es
LiteraturereviewInterviewswithdecisionmakers
hap://sheilaproject.eu/
Projectac3vi3es
LiteraturereviewInterviewswithdecisionmakers
Groupconceptmappingwithexperts
hap://sheilaproject.eu/
Projectac3vi3es
LiteraturereviewInterviewswithdecisionmakers
GroupconceptmappingwithexpertsSurveysandfocusgroupswithstudentsandteachingstaff
hap://sheilaproject.eu/
Expected3meline
Policydevelopmentframework(v1)–June2017Fourins3tu3onalpolicies(v1)–June2017
hap://sheilaproject.eu/
Expected3meline
Policydevelopmentframework(v1)–June2017Fourins3tu3onalpolicies(v1)–June2017
Policydevelopmentframework(v2)–March2018Fourins3tu3onalpolicies(v2)–March2018
hap://sheilaproject.eu/
Interestedinjoining?
Associatepartnersasearlyadoptersofthepolicydevelopmentframeworkarewelcome
hap://sheilaproject.eu/
Manythanks!
hap://sheilaproject.eu/
Interviews in SHEILA project
Adolfo Ruiz-Calleja, Kairit Tammets, Yi-Shan Tsai, Jeff Haywood, Maren Scheffel, Pedro Muñoz Merino and Dragan Gasevic
30th, November 2016
Index
32
■ Methodological aspects
■ Topics of the interviews
■ Results
■ Conclusions
Index
33
■ Methodological aspects
■ Topics of the interviews
■ Results
■ Conclusions
Methodology: ROMA
34 Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analytics imperative and the sociotechnical challenge: Policy for complex systems. Research & Practice in Assessment, 9(Winter 2014), 17-28.
Methodology
35
■ Purpose:
■ Sampling:
To collect opinions from decision makers in higher education institutions regarding the adoption of Learning Analytics.
• Policy makers • Vice-rectors • Study departments • Academic offices • IT departments
Methodology
36
■ 65 interviews
■ 52 institutions
■ 12 countries
Index
37
■ Methodological aspects
■ Topics of the interviews
■ Results
■ Conclusions
Topics
38
■ LA projects and their goals
■ LA strategy
■ Progress and achieved goals
■ Challenges
■ Ethics and privacy
Index
39
■ Methodological aspects
■ Topics of the interviews
■ Results
■ Conclusions
Results: LA projects and goals
40
■ Exploratory objectives
■ Very often LA projects are related to research
■ Diversity of interests
• Sense of uncertainty • What can LA do for my institution? • What can our data tell us?
• UK: correlation between behavioural patterns and learning outcomes
• Spain: support teachers and improve their performance • Baltics: improve students retention
Results: LA strategy
41
■ For some institutions (UK, Latvia, Finland) LA is part of their wider strategy
■ Others follow a bottom-up approach: some projects without connecting backbone structure
■ Managers become more interested on LA
Results: Achieved goals
42
■ Too early phase in many institutions
■ LA is considered useful
■ Awareness of LA, better understanding of challenges and improvement of data culture
■ In-house software is typically used • Vendors offer Dashboards or systems to detect drop-out
Results: Challenges
43
■ Cultural barriers
■ Little technical experience
■ Human resources
• Data-driven and not question-driven • Traditional institutional culture • Lack of awareness • Lack of data culture
• Many institutions do not feel ready for maintaining a data management infrastructure
• Little experience on how to deal with online environments
• Lack of time • Priorities
Results: Ethics and privacy
44
■ Ethical use of data remains a big challenge
■ Privacy laws are very diverse
■ Ethical issues open technical problems
• Although not all of them understand it as a problem (Estonia) • Clear need of ethical guidelines and solutions
• e.g. Germany and Austria are much more restrictive than UK or Estonia
• It is very difficult to avoid processing the data of one specific student if he does not allow it
Index
45
■ Methodological aspects
■ Topics of the interviews
■ Results
■ Conclusions
Conclusions
46
Conclusions
47
■ European universities are in an early stage of adoption of LA
■ Bottom-up approach from researchers, and awareness of policy makers
■ Several barriers have been detected, including ethics
■ There are important differences between countries
Interviews in SHEILA project
Adolfo Ruiz-Calleja, Kairit Tammets, Yi-Shan Tsai, Jeff Haywood, Maren Scheffel, Pedro Muñoz Merino and Dragan Gasevic
30th, November 2016
GroupConceptMappingPhase1:
Brainstorming
Phase2: Sor3ng
Phase3: Ra3ng
Anessen3alfeatureofahighereduca3onins3tu3on'slearninganaly3cspolicyshouldbe...
PointMap1 2
3
4 5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
2930
31
32
33
34
35
36
37
38
39 40
41
42
43
44
45
46
47
48
49
50
51
5253
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
7879 80
81
82
83
84
85
8687
88
89
90
91
92
93
94
95
96
97
9899
ClusterReplayMap
ClusterReplayMap
ClusterReplayMap
ClusterMap
1.privacy&transparency
2.roles&responsibiliZes(ofallstakeholders)
3.objecZvesofLA(learnerandteachersupport)
4.risks&challenges
5.datamanagement
6.research&dataanalysis
Ra3ngMap–Importance
1.privacy&transparency
2.roles&responsibiliZes(ofallstakeholders)
3.objecZvesofLA(learnerandteachersupport)
4.risks&challenges
5.datamanagement
6.research&dataanalysis
ClusterLegendLayerValue15.08to5.2725.27to5.4635.46to5.6545.65to5.8455.84to6.03
Ra3ngMap–Ease
1.privacy&transparency
2.roles&responsibiliZes(ofallstakeholders)
3.objecZvesofLA(learnerandteachersupport)
4.risks&challenges
5.datamanagement
6.research&dataanalysis
ClusterLegendLayerValue13.79to4.1224.12to4.4534.45to4.7844.78to5.1155.11to5.44
Ra3ngLadderGraph
importance ease
privacy&transparency
privacy&transparency
risks&challenges
risks&challenges
roles&responsibili3es(ofallstakeholders)
roles&responsibili3es(ofallstakeholders)
objec3vesofLA(learnerandteachersupport)
objec3vesofLA(learnerandteachersupport)
datamanagement
datamanagement
research&dataanalysis
research&dataanalysis
3.79 3.79
6.03 6.03
r=0.66
GoZone–Privacy&Transparency
1 720 31
43862
10172445
64 65
8892
91556
6074
87
34
69
96
6.08
5.44
3.12
ease
3.83 6.03 6.59
importance
r=0.45
GoZone–Privacy&Transparency
1 720 31
43862
10172445
64 65
8892
91556
6074
87
34
69
96
6.08
5.44
3.12
ease
3.83 6.03 6.59
importance
r=0.45
2.transparency,i.e.clearlyinformingstudentsofhowtheirdataiscollected,usedandprotected88.acleardescrip3onofdataprotec3onmeasurestaken10.acleardescrip3onofdatausage17.beingclearaboutthepurposeforcollec3oncertaintypesofdata24.alignedwithdataprotec3onregula3ons(ins3tu3onal,na3onal,interna3onal)34. toassurethatthecollecteddataisusedonlyforthepurposeofimprovinglearningandinstruc3on96.anagreementbetweenlearners,teachersandpolicymakersonregula3ngaproperuseofdata
GoZone–Roles&Responsibili3es
5
38
62
11
1922
33
39 48
7091
25
28
37
40
55
61
6627
47 49
6.08
4.72
3.12
ease
3.83 5.48 6.59
importance
r=0.26
GoZone–Roles&Responsibili3es
5
38
62
11
1922
33
39 48
7091
25
28
37
40
55
61
6627
47 49
6.08
4.72
3.12
ease
3.83 5.48 6.59
importance
r=0.26
55.beingclearaboutthepurposeoflearninganaly3cs61.aclearar3cula3onofrolesandresponsibili3eswhenitcomestotheuseofins3tu3onaldata39.topromotebroadadop3onoflearninganaly3csbyspecifyingsuppor3veregula3onsandcaselaw
GoZone–Objec3vesofLA
3
80
84
99
12
7898
3550
93
4244
586376
79
94
6.08
3.79
3.12
ease
3.83 5.44 6.59
importance
r=-0.12
GoZone–Objec3vesofLA
3
80
84
99
12
7898
3550
93
4244
586376
79
94
6.08
3.79
3.12
ease
3.83 5.44 6.59
importance
r=-0.12
63.touselearninganaly3csforimprovingthequalityofteaching58.toallowforproac3vebehaviorofstudentstowardstheireduca3on,withsetngpersonalgoals44.tosupportlearnerslearningtolearnandimproveskills98.toencouragemeaningfulrela3onshipsbetweenstudents
GoZone–Risks&Challenges
18
41
8326
46
97
53
59
90
95
6.08
4.16
3.12
ease
3.83 5.52 6.59
importance
r=0.52
GoZone–Risks&Challenges
18
41
8326
46
97
53
59
90
95
6.08
4.16
3.12
ease
3.83 5.52 6.59
importance
r=0.52
90.todrawaaen3ontowhatthe"bigpicture"endgoalofthedatauseis59.todemarcateclearlybetweendifferentusesofanaly3csinins3tu3onalsetngs95.toensurethebenefitstostudentsoutweightherisks53.toensureadequaterepresenta3onduringhigh-stakesdecisionprocesses18.integratedwitharewardsystemfortheuseofevidence-basedimprovementstoteachingandlearning41.todiscouragestudentsandfacultyfromgamingthesystem
GoZone–DataManagement
4
14
89
23
67
71
85
6.08
4.5
3.12
ease
3.83 5.16 6.59
importance
r=-0.54
GoZone–DataManagement
4
14
89
23
67
71
85
6.08
4.5
3.12
ease
3.83 5.16 6.59
importance
r=-0.54
23.todefineclearrulesforcollabora3onwithotherresearchers
GoZone–Research&DataAnalysis
6
30
57
6872
75
77
81
8
32
527382
1351
1621
29
36
54
6.08
4.24
3.12
ease
3.83 5.08 6.59
importance
r=0.27
GoZone–Research&DataAnalysis
6
30
57
6872
75
77
81
8
32
527382
1351
1621
29
36
54
6.08
4.24
3.12
ease
3.83 5.08 6.59
importance
r=0.27
57.toassurethatinstruc3onalinterven3onsarebasedonwell-studiedandempiricallyvalidatedanaly3calmethodsandalgorithms75.toencouragethedevelopmentofdashboardswithmeaningfulandunderstandableoutcomes68.torecognisethelimita3onsoflearninganaly3cs72.toprovideclearexamplesofhowtointerpretdata,especiallywhentheremaybeinterac3ons81.to(re)evaluatecon3nuouslytheanaly3csbeingapplied54.toenhancetheconnec3onbetweentheacademicandthesocietyandindustry,showingwhatisbeingdoneattheins3tu3onsisrelatedtotheeduca3onandtraining21.tomakeexplicitmathsfrominnova3vedevelopment/useofanaly3csinindividualclassroomstoscalingup
Thank [email protected]@m_a_s_c