Black, Adam Dr - Efficacy and how to improve learner outcomes
Transcript of Black, Adam Dr - Efficacy and how to improve learner outcomes
Efficacy and how to improve learner outcomes
EAQUALS Conference
April 25, 2014, Budapest
Dr. Adam Black
Efficacy and Research, Pearson (Professional)
Overview
• Defining Efficacy
• Building a path to Efficacy
• Efficacy Reviews: a framework and lessons learned
• Efficacy Studies: holistic measures of impact
• Efficacy Analytics: global trends
• Do an Efficacy Review yourself
• Q&A
Efficacy and improving learner outcomes, EAQUALS April 2014 2
Context
―No program can be evaluated properly without a common
understanding of what it’s supposed to achieve.
An unfortunate consequence of treating purposes casually is
a tendency to accept goals that seem important in theory
without pausing to consider whether it is possible to achieve
them within the time available.‖
Our Underachieving Colleges
Derek Bok (former President, Harvard University)
Efficacy and improving learner outcomes, EAQUALS April 2014 4
―I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop.‖
Bill Gates, Jan 2013
Context
Why now?
• There is a shared understanding that high-quality education drives personal, economic and societal growth
• Governments, individuals, employers and institutions recognise the need to deliver high-quality learning
• New technology makes it increasingly possible to see what works and what doesn’t in helping learners to achieve their goals
Efficacy and improving learner outcomes, EAQUALS April 2014 6
What do we mean by efficacy?
A measurable impact on learner outcomes
Efficacy and improving learner outcomes, EAQUALS April 2014 7
efficacy (dictionary definition)
• ability to produce the intended result
Efficacy (Pearson’s definition)
• make a measureable impact on learner outcomes
efficiency (dictionary definition)
• achieve maximum productivity with minimum wasted effort
Pearson’s path to efficacy: three integrated activities for improving learner outcomes
research design and
development piloting
full deployment
customer use
Efficacy Analytics: mine data from products to gain insights into iterative improvements, learner behaviours and future innovation
Efficacy Reviews: predict likelihood of impacting learner outcomes and plan improvements
Efficacy Studies: learn from long-term holistic studies of outcomes
An Efficacy Framework: how likely is it that your project will successfully
improve learner outcomes?
Criteria area Rating Rationale summary
• Action plan
• Governance
• Monitoring and reporting
• Internal capacity and culture
• User capacity and culture
• Stakeholder relationships
Outcomes
• Intended outcomes
• Overall design
• Value for money
• Comprehensiveness of evidence
• Quality of evidence
• Application of evidence
Evidence
Planning and implementation
Capacity to deliver
Efficacy
An Efficacy Framework: likelihood of impact
An Efficacy Framework: an explanation of ratings
Good – requires slight refinement, but on track
Mixed – some aspects require attention, some solid
Problematic – requires substantial attention, some require urgent rectification
Off-track – requires urgent action and problem solving
Ratings are not grades on performance
Ratings prompt discussions that lead to actions
Ratings prioritise and suggest timeline
Efficacy and improving learner outcomes, EAQUALS April 2014 12
Criteria area Rating Rationale summary
• Action plan
• Governance
• Monitoring and reporting
• Internal capacity and culture
• User capacity and culture
• Stakeholder relationships
Outcomes
• Intended outcomes
• Overall design
• Value for money
• Comprehensiveness of evidence
• Quality of evidence
• Application of evidence
Evidence
Planning and implementation
Capacity to deliver
Efficacy
An Efficacy Framework: likelihood of impact
An Efficacy Framework: a deep-dive on outcomes
Overall design
• Is the product designed in a way that will most effectively help your target group reach their goals?
• Does the design allow you to automatically collect evidence of your progress?
• Have you adapted the design based on feedback from users?
• Could the design by used by others?
Value for money
• Do you understand the benefits of your product/service to your target group? Relative to other options?
• Is the cost of the product/service competitive, considering the benefits it would deliver?
Intended outcomes
• Have you identified specific outcomes for your target group?
• Do you have a way to measure the intended outcomes?
• Do you have ambitious and measurable targets in place, and deadlines for achieving them?
• Are your intended outcomes clearly documented and understood by your team and customers?
Example of green rating Example of red rating
• All outcomes are specific and clearly documented.
• People within and outside my organisation understand the intended outcomes and can communicate them clearly.
• Future targets are ambitious and achievable.
• Outcomes can be regularly measured against set targets.
• Design is superior to other options/competitors with features focused on delivering outcomes.
• Real-time evidence is generated.
• The design can be adapted and developed.
• Others could use this design, and it has been shared with them.
• Feedback/research has identified what benefits the product/service needs to deliver to users.
• Feedback and return-on-investment research shows that the cost of the product/service reflects the benefits.
• Outcomes are not documented or specific.
• People within and outside my organisation do not understand the intended outcomes or communicate them in the same way.
• Targets do not exist to measure outcomes against.
• Outcomes are only defined at a high level.
• No feedback from users (formal or informal), and benefits of using the product/service are unclear to our team and users.
• Perceptions of value for money and user experience are poor.
• The design does not meet target group expectations and is difficult to use.
• The design does not reflect intended outcomes.
• The design does not allow for the collection of feedback.
• The design is specific to a local situation and cannot be replicated.
An Efficacy Framework: in action
Review of evidence
• Strategy papers • Customer feedback • Audits • Progression research • Policy briefs
Internal interviews
• Sales • Strategy • Marketing • Planning • Executive leadership
Customer and stakeholder interviews
• Government bodies • Universities • Potential employers • Associations
Efficacy workshop
Outputs
Assessment of current efficacy
Actions needed to enhance efficacy
Highly collaborative and focused on improvement opportunities
Framework area Initial review
3- month estimate
6-month estimate
Comment
Outcomes
Intended outcomes After 6 months, outcomes and metrics will be clear and will influence design. Value for money will be tested from pilots.
Overall design
Value for money
Evidence
Comprehensiveness of evidence After 6 months, the plan to develop the forward evidence base will be finalised and initiated. Quality of evidence
Application of evidence
Planning and implementation
Action plan After 6 months, long-term plans and reporting structures will be in place and governance agreed. Reporting will be at an early stage.
Governance
Monitoring and reporting
Capacity to deliver
Pearson capacity and culture After 6 months, Capacity issues will be clear, pilots delivered and lessons learned and applied. Stakeholder relationships plans will be launched and gathering feedback.
Customer capacity and culture
Stakeholder relationships
An Efficacy Framework: driving improvement
Efficacy and improving learner outcomes, EAQUALS April 2014 16
An Efficacy framework: lessons learned
1. You won’t improve a learner outcome you can’t define clearly!
2. You can’t demonstrate you’re improving a learner outcome if you’re not measuring it!
3. Appropriate learner outcomes vary by age, stage, and situation (school, college, private language school, corporation)
4. To improve learner outcomes, stakeholders must be aligned to the same goals (tutors, administrators, education authorities, etc.)
Efficacy and improving learner outcomes, EAQUALS April 2014 17
Efficacy Studies: holistic, long-term studies with specific learners, teachers, and institutions
Efficacy and improving learner outcomes, EAQUALS April 2014 19
Identify common learner difficulties Personalise learning Optimise learning by L1
Research learner behaviours that lead to
success (machine learning)
Improve learner engagement
(activity design)
Predict learners who will fail for early intervention
(predictive algorithms)
Efficacy Analytics: insights into learning behaviours
Efficacy and improving learner outcomes, EAQUALS April 2014 21
both students have the same net score
0 100 200 300 400 500 600
-25
-20
-15
-10
-5
0
5
10
Ne
t S
core
Responses Submitted by Student
0 100 200 300 400 500 600
0
50
100
150
200
250
300
Student 57
Fractal D = 1.60
Ne
t S
co
re
Responses Submitted by Student
fractal alert: alert teacher and learner to intervene
responses submitted over course
student who will succeed - smooth Fractal = 1.60
student who will fail or not complete – noisey Fractal = 1.94
Efficacy Analytics: identifying learners at risk
Patent awarded 2013
Efficacy and improving learner outcomes, EAQUALS April 2014 22
http://efficacy.pearson.com
Efficacy framework: try it yourself
Efficacy and improving learner outcomes, EAQUALS April 2014 24