Learning Analytics for Holistic Improvement ALASI 2014
-
Upload
ruth-deakin-crick -
Category
Education
-
view
295 -
download
0
description
Transcript of Learning Analytics for Holistic Improvement ALASI 2014
![Page 1: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/1.jpg)
UTS CRICOS PROVIDER CODE: 00099F
ANALYTICS FOR HOLISTIC EDUCATIONAL IMPROVEMENT:
cic.uts.edu.au
![Page 2: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/2.jpg)
THE STORY IN A SLIDE
• Organisations are complex and ‘messy’
• There are different ‘ways of knowing’ through which humans seek knowledge and represent that knowledge through data.
• Using data to derive actionable insights for leadership decisioning must take account of such complexity and be aligned to purpose & timely.
• Case study – hierarchical process modelling is a way of structuring data for leadership decisioning
• The perimeta dashboard – what it looks and feels like
• Actionable insghts for leadership decisioning
• What next? Key issues?
cic.uts.edu.au
![Page 3: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/3.jpg)
http://www.youtube.com/watch?v=jBl6ALNh18Q
Jurgen Habermas b 1929
Knowledge seeking human interests Empirical analytical Hermeneutical Emancipatory
DIFFERENT FORMS OF RATIONALITY
![Page 4: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/4.jpg)
cic.uts.edu.au
Learning Analytics UNESCO Policy Briefing — Simon Buckingham Shum
Empancipatory Rationality individual learner progress
Hermeneutical Rationality institution-wide contextual
Strategic Rationality region/state/national/international
4
DATA FOR LEARNING ANALYTICS
Learners
Leaders/Teachers
Policy Makers Researchers
![Page 5: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/5.jpg)
Measurement models for learning journeys: using different types of data and different quality criteria
Narrative; self-report; interpretation. Self report questionnaires, observation, self-assessment. Tests of knowledge, depths of understanding, MCQs, visualisations. Community of Peers, public presentations, evaluations Hierarchical Process modelling
Identity purpose
motivation
Personal qualities
Knowledge structuring
Performing
Authenticity Trustworthiness Credibility Coherence Reliability Validity Replicability Usefulness
![Page 6: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/6.jpg)
BACKGROUND
The Education Charter
The Echo Project
Three Oasis Academies
![Page 7: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/7.jpg)
WHAT TO DO?
"Here is Edward Bear, coming downstairs, bump, bump, bump, on the back of his head, behind Christopher Robin. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it"
Alan Alexander Milne The opening paragraph of “Winnie-the-Pooh”, 1926
![Page 8: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/8.jpg)
Systems thinking – managing complexity Developing resilience – becoming learning organisations Responding to emergence – unpredictable outcomes Taking responsibility for our own development Engaging all stakeholders in a networked improvement community Using rich data for actionable insights
![Page 9: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/9.jpg)
KEY QUESTIONS
How can we measure the full range of what we value in schools? How can we collect and manage evidence to understand the big picture as well as the detail? How can we lead and manage in conditions of complexity? How can we used data for real-time leadership decisioning as well as for evaluating performance
![Page 10: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/10.jpg)
Macro Layers and feedback loops……
CommunityLearning
Leaders Learning
Teachers Learning
Students Learning
Learning is a core process at multi-levels in learning organisations
Blockley, D. (2010) The Importance of Being Process, Civil Engineering and Environmental Systems, 27(3)189-199
![Page 11: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/11.jpg)
cic.uts.edu.au
THE COMPLEX PROBLEM OF CREATING LEARNING COMMUNITIES
![Page 12: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/12.jpg)
IMAGINE ……
![Page 13: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/13.jpg)
13
Define purpose
Analyze system
Design Model
Design measurement parameters for
evaluating success
Collect data according to
measurement model over time
Enter into HPM model and analyse
performance
Feedback to system leaders
Explore solutions as prototypes
Implement prototypes and
evaluate
Hierarchical Process Modeling
for a Holistic Approach to
System Improvement
P
A SD
![Page 14: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/14.jpg)
What sort of design should we use to model our learning communities?
![Page 15: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/15.jpg)
HIERARCHICAL PROCESS MODELLING
![Page 16: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/16.jpg)
cic.uts.edu.au
Purpose = How (who what where when)
![Page 17: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/17.jpg)
Outcome 1.1 Establishing and sustaining a group of high achieving learning communities that enables everyone to realise their full potential and refuses to put limits on achievement
2. Outputs 2.1 Developing the learning of students so that they realise their full potential
2.2 Developing the learning of teachers so that they realise their full potential
2.3 Developing the learning of leaders so that they lead the learning of teachers and students effectively
2.4 Engaging parents/carers effectively in the learning activities of the Academy and in supporting the learning of their children
3. Inputs
3.1
Feel
ing
valu
ed a
nd in
clud
ed in
our
com
mun
ity
3.2
Feel
ing
part
of a
com
mun
ity th
at fo
cuse
s its
ac
tiviti
es o
n le
arni
ng a
nd a
chie
vem
ent
3.3
Kno
win
g th
at I
am b
ecom
ing
an e
ffect
ive
lear
ner
3.4
Kno
win
g th
at w
e ar
e be
ing
help
ed to
reac
h ou
r ful
l pot
entia
l 3.
5 M
aint
aini
ng a
goo
d ra
te o
f pro
gres
s at
key
tra
nsiti
ons,
for e
xam
ple,
prim
ary
to s
econ
dary
sc
hool
3.6
Ach
ievi
ng re
sults
that
mee
t my
aspi
ratio
ns
and
expe
ctat
ions
3.7
Feel
ing
that
my
succ
esse
s ar
e re
cogn
ised
3.
8 C
ontri
butin
g ac
tivel
y to
a c
omm
unity
whi
ch
focu
ses
on s
ervi
ce to
oth
ers
3.9
Con
tribu
ting
activ
ely
to a
com
mun
ity w
here
sh
ared
lead
ersh
ip is
pro
mot
ed
3.10
Pro
vidi
ng e
vide
nce
that
trai
ning
and
de
velo
pmen
t opp
ortu
nitie
s ha
ve h
elpe
d m
e to
im
prov
e m
y cl
assr
oom
pra
ctic
e an
d/or
my
effe
ctiv
enes
s as
a le
ader
and
man
ager
3.11
Con
tribu
ting
activ
ely
to a
com
mun
ity th
at
lear
ns to
geth
er, s
hare
s w
hat w
orks
bes
t and
kn
ows
wha
t it n
eeds
to d
o to
impr
ove
3.12
Fee
ling
incl
uded
and
invo
lved
with
my
child
ren’
s ed
ucat
ion
at th
e A
cade
my
and
know
ing
that
ther
e is
an
open
doo
r for
con
tact
whe
n I
need
it
3.13
Bei
ng e
ffect
ive
role
mod
els
as le
ader
s of
le
arni
ng
Sources of
evidence Students Teachers Leaders Parents/ carers
PURPOSE H
ow
data
![Page 18: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/18.jpg)
A Hierarchical Model: Defined by Purpose
![Page 19: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/19.jpg)
What type of data do we need to collect to measure the core processes which fulfil our purpose?
![Page 20: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/20.jpg)
20
Define purpose
Analyze system
Design Model
Design measurement parameters for
evaluating success
Collect data according to
measurement model over time
Enter into HPM model and analyse
performance
Feedback to system leaders
Explore solutions as prototypes
Implement prototypes and
evaluate
Hierarchical Process Modeling
for a Holistic Approach to
System Improvement
P
A SD
![Page 21: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/21.jpg)
CAPTURING DATA FOR CONTINUAL IMPROVEMENT
cic.uts.edu.au
Surveys for Open Learning Analytics (SOLA ) Platform hosting apps to support self-directed change
An integrated platform for systemic improvement
DATA WAREHOUSING FOR SYSTEM WIDE RESEARCH
LEGACY SYSTEMS
BLACKBOARD
HIERARCHICAL PROCESS MODELLING
KNOWLEDGE STRUCTURING TOOLS
SOCIAL LEARNING ENVIRONMENTS
![Page 22: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/22.jpg)
Stakeholders Type of data Focus 1 Y7 students Numerical scores per dimension Learning power 2 Y8 & 10
Students Numerical values for each question students as learners and
values & citizenship 3 Y9 students Stories rated by researchers Personal transformation 4 Y11 leavers Numerical values for each question Aspirations, trajectories
5 Parents/carers Numerical values for each question Participation
6 Teachers Numerical values for each question Impact of professional learning on student outcomes
7 Leaders Numerical values for each question Leaders as learners
8 KPI s Numerical values of performance Behavioural indicators
9 Y11 Numerical values of performance Attainment
![Page 23: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/23.jpg)
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Taking responsibility for my own learning over time through defining my purposes, understanding and managing my feelings, knowing how I go about learning & planning my learning journey carefully.
Making connections between what I already know & new information & experience. Making meaning by linking my story, my new learning & my purpose.
Wanting to get beneath the surface & find out more. Always wondering why and how.
1
Dependent Closed Openness to learning
An emotional orientation of being open & ready to invest in learning, having flexible self-belief, willing to persist & manage any self-doubt. A necessary pre-requisite for developing resilience in learning
Using my intuition & imagination to generate new ideas & knowledge. Taking risks & playing with ideas and artefacts to arrive at new solutions.
Being part of a learning community at work, at home, in education & in my social networks. Knowing I have social resources to draw on when I need them
Having the optimism & hope that I can learn & achieve over time. Having a growth mindset; believing I can generate my own new knowledge for what I need to achieve
Being able to work with others, to collaborate and co-generate new ideas and artefacts. Being able to listen and contribute productively to a team.
![Page 24: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/24.jpg)
THE NATURE OF UNCERTAINTY
Fuzziness Incompleteness Randomness
![Page 25: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/25.jpg)
.. COMBINED WITH PERFORMANCE MEASURES THAT COPE EXPLICITLY WITH UNCERTAIN EVIDENCE
1 0
Evidence that A is successful Lack of Evidence
Evidence that A is not successful
We don’t really know what is going on!
Performing well with little uncertainty
Performing poorly with some uncertainty
![Page 26: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/26.jpg)
From Likert type scales to measures of uncertainty
![Page 27: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/27.jpg)
Hall et al’s (2004) mapping from linguistic descriptions of ‘performance’ and ‘confidence in judgement of performance’ to interval values of performance
![Page 28: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/28.jpg)
So what does our Perimeta model tell us about Parents/carers ..…an example
![Page 29: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/29.jpg)
29
Define purpose
Analyze system
Design Model
Design measurement parameters for
evaluating success
Collect data according to
measurement model over time
Enter into HPM model and analyse
performance
Feedback to system leaders
Explore solutions as prototypes
Implement prototypes and
evaluate
Hierarchical Process Modeling
for a Holistic Approach to
System Improvement
P
A SD
![Page 30: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/30.jpg)
cic.uts.edu.au
PERIMETA DASHBOARD FEATURES
Input data – one node for each Respondent and each Question
Results compiled for each Question
Results compiled for each Gender
Results overall for specified area of performance
Results compiled for each Academy
![Page 31: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/31.jpg)
cic.uts.edu.au
PERIMETA MODELS FROM DIFFERENT EVIDENCE STRANDS
S7: Learning of leaders (SLT)
S4: Learning of students in Year 11
S6: Learning of teachers
S5: Quality of relationship w. parents/ carers
![Page 32: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/32.jpg)
![Page 33: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/33.jpg)
cic.uts.edu.au
PERIMETA MODEL RESULTS
![Page 34: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/34.jpg)
cic.uts.edu.au
DRILLING DOWN INTO THE EVIDENCE TO UNDERSTAND KEY ISSUES
Statement POS I was helped and supported when I struggled 0.98 My teachers taught me well 0.90 The Academy made good use of technologies 0.90 I was valued and included 0.89 I was often challenged to do better 0.89 There were great facilities 0.89 I felt part of a learning community 0.89 I developed responsibility towards others 0.89 I had a wide range of opportunities for learning 0.89 I felt safe 0.89
Outcome Establishing and sustaining learning communities Outputs Developing the learning of students so that they realise their full potential Inputs Feeling valued and
included in our community
Feeling part of a community focused on learning & achievement
Knowing that I am becoming an effective learner
Knowing that we are being helped to reach our full potential
Statement UNC I supported activities that would improve things for other people
0.38
I led activities 0.34 I am looking forward to the move to Sixth Form, FE, training or work
0.33
My parents/carers felt involved 0.15 Everyone worked very hard to make it a great place to learn and to do well
0.15
I enjoyed my time 0.15 I could not have gone to a better school 0.14 I was offered opportunities for leadership 0.14 I never felt that anyone was putting limits on what I could achieve
0.13
My teachers accurately predicted my GCSEs 0.13
Statement NEG I am looking forward to the move to Sixth Form, Further Education, training or work
0.18
I led activities during my time at the Academy 0.17 No others above 10%
![Page 35: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/35.jpg)
THE PERIMETA TOOLKIT SUPPORTS COLLABORATIVE ACTION TO IMPROVE LEARNING
Focusing on PERFORMANCE vs broad outcomes
Understanding COMPLEX PROCESSES
Weighing up uncertainty of EVIDENCE
Informing robust CHOICES
Enabling robust & collaborative ACTION
KEY FEATURE: “Italian Flag” evidence model & rich uncertainty calculus
KEY FEATURE: Excel interface for linked data and predictive models
KEY FEATURE: Hierarchy of interactions focused on outcomes
![Page 36: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/36.jpg)
So how do we respond to what Perimeta is telling us as a learning community?
![Page 37: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/37.jpg)
37
Define purpose
Analyze system
Design Model
Design measurement parameters for
evaluating success
Collect data according to
measurement model over time
Enter into HPM model and analyse
performance
Feedback to system leaders
Explore solutions as prototypes
Implement prototypes and
evaluate
Hierarchical Process Modeling
for a Holistic Approach to
System Improvement
P
A SD
![Page 38: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/38.jpg)
Professional Enquiry through Rapid Prototyping to address key challenges highlighted by Perimeta….
Plan
Act Study
Do
…..within a disciplined evaluation framework
![Page 39: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/39.jpg)
Experiential knowing, hunches, ideas
P D
SA
P D
SA
P D
SA P D
SA
P D
S A
P D
S A
P D
S A P D
S A
P D
S A
P D
S A
P D
S A P D
S A
Measurable System Improvement: New Knowledge and Know How
Bryk, A. Gomez L. Gunrow A. (2010) Getting Ideas into Action: Building Networked Improvement Communities in Education, Carnegie Foundation for the Advancement of Teacher Education, Carnegie, Stanford.
![Page 40: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/40.jpg)
How can we use technology to enhance our ability to develop organisational learning through Perimeta
![Page 41: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/41.jpg)
cic.uts.edu.au
AN ACADEMY WITH SYSTEMIC ANALYTICS
Strategic Progress
Learner Dispositions
Teacher Dispositions
Stories of Change & Evidence
Reflective Enquiry Blogs
Academy X
![Page 42: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/42.jpg)
cic.uts.edu.au
AN ACADEMY NETWORK WITH SYSTEMIC ANALYTICS
42
Analytics Integration
Evidence Hub
Data Data Data Data
![Page 43: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/43.jpg)
cic.uts.edu.au
Share and debate evidence…
About…
SYSTEMS LEARNING & LEADERSHIP EVIDENCE HUB HTTP://SYSLL.EVIDENCE-HUB.NET
![Page 44: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/44.jpg)
cic.uts.edu.au
SYSTEMS LEARNING & LEADERSHIP EVIDENCE HUB HTTP://SYSLL.EVIDENCE-HUB.NET
44
Replay Oasis/Univ Bristol/Open Uni event: http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
![Page 45: Learning Analytics for Holistic Improvement ALASI 2014](https://reader033.fdocuments.in/reader033/viewer/2022052622/5595d2221a28abf72b8b4627/html5/thumbnails/45.jpg)
cic.uts.edu.au
Data Sets + Data Streams
LEARNING ANALYTICS HUB (3.0)
Learning Analytics System Simulations
Recommendation Engines Social Learning Environments
Online Tools designed to measure core processes and to
stimulate self-directed change
Buckingham Shum, S. & Deakin Crick, R. (2012) Learning Dispositions and Transferable Competencies: Pedagogy, Modelling, and Learning Analytics, Proceedings of LAK12: 2nd International Conference on Learning Analytics & Knowledge, April 29 - May 2, 2012, Vancouver, BC. ACM: New York. ISBN 978-1-4503-1111-3.