June presentations org_adoption_learning_analytics
53
A tale of two constructs: Understanding learning analytics adoption Shane Dawson [email protected] Twitter: @shaned07
Transcript of June presentations org_adoption_learning_analytics
A tale of two constructs: Understanding learning
analytics adoption
Shane [email protected]: @shaned07
Introduction
Your Phone knows your GPA
http://studentlife.cs.dartmouth.edu/smartgpa.pdf
Introduction
SmartGPA study• Using smart phones to track student study
behaviour – lead indicators for academic performance• Class attendance• Conversational interactions• Mobility• Studying time and• Social (partying behaviour)
http://studentlife.cs.dartmouth.edu/smartgpa.pdf
Introduction
Massive interest in data and analytics.Examples in education - diverse and rapidly growing
• Academic performance• Student retention• Pastoral care• Academic literacies• Social networks – collaborations
Introduction
Yet in terms of wide-scale institutional adoption there are few examples
Why?
• Brief overview of learning analytics• Findings from a national study • Discussion
• 2 trajectories• Systems model
• An idealised model?• Complexity of LA
Introduction
Algorithms – predictions
Introduction
Education is no different• Huge investment in analytics • Ease of access to learner data - LMS• Growth in technical devices• Growth in blended/ online learning models
Introduction
Get answers to your most important questions like:• How can I easily find students who are at-risk?
Introduction
Definition
…is the collection, collation, analysis and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning
Learning Analytics • “game changer” for education
Large data sets – trends/ patterns or anomalies.
Learning Analytics
LA/EDM research:• Predictions of learning success (early
alerts)/Performance and retention• Indicators of on/off task attention (Baker)• Carelessness and gaming (Baker)• SRL proficiency (Gasevic; Winne)• NLP (Rose; Gasevic)• Learning dispositions (Deakin Crick)• Graduate qualities/ 21st C literacies (Siemens)
LA Research
Great research BUT:
• Ignores the complexity of university wide practices• Small scale and technology specific• Tends to be institutional specific• Lacks guidance to aid further adoption• Frequently requires high level skills and capacities
LA Research
Innovative research BUT:
• Very few University wide examples of LA adoption• Obviously an area of increasing need and
importance
LA Research
• Huge investments in LA• Consultancy and services• Technologies• Research funding• University priorities
Where is the broad scale impact?What are the “ideal” processes and practices for LA adoption?
National project to benchmark LA status, policy and practices for Australian Universities
Many thanks to:Cassandra Colvin, Alex Wade and Tim Rogers
National Project
• understand current LA practice in Australia• unpack the challenges to institutional
adoption • Identify practices that can aid the
implementation of LA
Aims
2 complementary but separate studies
• Study 1 – interviews with senior institutional leaders• Study 2 – concept mapping with LA expert panel
Approach
First study:Interviews with 32 Universities:
• Identification of current practice, methods and approaches
• Identification of key drivers for institutions, stage of development, process for implementation, project leads
Study 1
First study:Coding:
• Development and application of coding protocol • Cluster analyses performed (PAM clusters)
Study 1
• Much interest in LA• Stated organisational priority
• LA projects were in the early phases of implementation and small scale (at time of interview July 2014)
• 2 distinct clusters across variables such as: implementation, conceptualisation, readiness
Cluster 1 (n=15) – Solutions focusedCluster 2 (n=17) – Process focused
Study 1
High interest – slow uptake - predominantly at the stage of basic reporting
• Goldstein & Katz (2005) reviewed US universities and noted vast majority were in Stage 1 or 2 (of 5 stages in maturity)
• Yanosky (2009)
• Bichsel (2012)
Benchmarking
Clearly there remain challenges with implementing LA at scale
Benchmarking
2 Distinct trajectories for implementation
• Differences in cluster variables such as:• Conceptualisation of LA• Readiness• Implementation approach
Study 1
• Cluster 1• focused on retention outcomes• Limited mention of LA as a means to
improve learning• Main driver is budget (cost savings)
• Cluster 2• Broader view of learning analytics and its
application into learning and teaching practice
Conceptualisation
• Cluster 1• Limited to no articulated strategy• Minimal capacity building activities• Success is seen as staff access to information• Technology infrastructure sound and
developing• Cluster 2
• Defined strategy• Developing capacity building activities• Technology – less emphasis on development.
Readiness
• Cluster 1• Minimal stakeholder engagement • Leadership top down and siloed• Vendor tools heavily integrated
• Cluster 2• Extensive stakeholder engagements• Leadership top down – but wide-spread.• Multiple engaged units (IT, teaching, faculty)• Where vendor tools/ processes adopted
maintained critical perspective
Implementation
Strategic capability
Context
Student demographics
Student retention and performance
Govn accreditation
University group / ATN/ Rural/ Go8
Strategic capability
1. Solutions focused• LA to address a pressing need• Time sensitive
2. Process focused• Networked and integrated model• Minimal time pressures• Innovation and experimentation
Strategic capability
What are the ideal dimensions for long term sustainable uptake of LA?
• Invite to Australian and international LA experts• 28 completed the entire concept mapping phases• 3 phases – brainstorming; sorting and ranking of
statements• Following the final ranking phase – a 7 cluster
solution emerged.
Study 2
Study 2
Bringing it together
Study 1 – 2 clusters Study 2 – 7 clusters
Essentially – how an organisation approaches its conceptualisation of LA underpins (2 clusters) the method for deployment and adoption (7 clusters)
Systems Model
Strategic capability
Interested Implementing
Implementation capability
Tool/Data Quality
Research/ Learning
Educator uptake
Systems Model
Strategic capability
Implementing
Tool/Data Quality
Educator uptake
Systems Model
Strategic capability
Interested Implementing
Implementation capability
Tool/Data Quality
Research/ Learning
Educator uptake
LA Maturity
LA Maturity
LA Maturity
Solutions focusedAddress an immediate concern – e.g. retention
LA Maturity
Process focusedBroad view of LA
Merging Models
Merged model – responsive and agile
Challenges to be addressed:
• Leadership awareness• Teams are seldom interdisciplinary• IT driven and system focused• Scale versus understanding• Capabilities and skills deficit. • Over reliance on current research – requires
further validation across different contexts to demonstrate transportability of models
Bringing it together
Leveraging the outcomes of short term goals for long term gain
• How do we merge both models to gain both short and long term impact?
Complexity
Complexity
Law of requisite variety (Ashby 1958)
• To control a system – the number of problems need to be matched by at least an equal number of responses
• The more complex the system the more problems
Ashby W.R. (1958) Requisite variety and its implications for the control of complex systems, Cybernetica 1:2, p. 83-99. (available at http://pcp.vub.ac.be/Books/AshbyReqVar.pdf, republished on the web by F. Heylighen—Principia Cybernetica Project)
Complexity
Law of requisite complexity (Boisot & McKelvey 2011)
• It take complexity to defeat complexity• A system must possess a level of complexity that
at least matches that of its environment to function effectively
Boisot, M., & McKelvey, B. (2011). Complexity and organization--Environment Relations: Revisiting Ashby's Law of Requisite Variety. The Sage handbook of complexity and management, 279-298.
Education is a complex system• Resilient to change• Adaptive and self-organising clusters• Change is non-linear and often unexpected
LA is complex – tendency towards simplification for implementation
Complexity
LA often reduced to independent components• Data• Analysis• Technology• Dashboards/ visualisations• Staff training
Does not adequately deal with the inter-relationships nor the overarching complexity of the system
Complexity
• The view to simplify can lead to fixed boundaries and organisational silos
• IT example
Complexity
Conclusion
• LA requires new models for implementation and leadership
• Enabling leadership• Whole of organisation• Models that are agile and research informed
• Working in complexity creates friction• Embrace the friction – generates innovation
Conclusion
• A solutions based model can drive change – but need to be mindful of responding to changing organsational needs
• Process based model can drive innovation and interest – but need to be mindful of how to scale
Conclusion
• Combined model framed in the organisational context
• Small, diffuse pockets of innovation to build capacity and build interest
• View to scale adoption – demonstration of impact (technical, pedagogical)
• Distributed enabling leadership (complexity leadership)
Conclusion
• Any “successful” adoption of LA will be dependent on an institution’s ability to rapidly recognise and respond to the organisational culture and the concerns of all stakeholders.
