X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASUREMENT SYSTEM FOR CHAPLAINCY SERVICES...
-
Upload
clifton-anderson -
Category
Documents
-
view
231 -
download
0
Transcript of X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASUREMENT SYSTEM FOR CHAPLAINCY SERVICES...
vMEASUREMENT SYSTEM FOR CHAPLAINCY SERVICESJames Noble
London, May 2015
2
CONTENTS
1Context 01
2Recap on theory of change (principles)
02
3Review of the chaplaincy theory of change
03
4Measurement principles 04
5Developing a measurement framework for chaplaincy projects
05
6Iizuka! 06
7Questions / discussion 07
3
WHY IS MEASURING IMPACT IMPORTANT?
Saves staff time
Influences the debate on
“what works”Improved services
Raises profile Motivates staff /
volunteers
Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater Impact. New Philanthropy Capital.
Helps secure funding
4
WHAT ARE THE CHALLENGES OF IMPACT MEASUREMENT?
Lack of skills
CostComplexity
Risk of weak results
Lack of confidence in value of soft outcomes
Risk of negative impact
on service delivery
Lack of technology
Wary of losing sight of ethos
CONTEXT
NPC: TRANSFORMING THE CHARITY SECTOR
6
NPC works at the nexus between charities and
funders
Charity
SectorFunder
Increasing the impact of charities
eg, impact-focused theories of change
Strengthening the partnership
Eg, collaboration towards shared
goals
Increasing the impact of funders
eg, effective commissioning
ConsultancyThink tank
7
“IMPROVING YOUR EVIDENCE”
Develop the evaluation capability and expertise of providers who offer services and
interventions to offenders.
http://www.clinks.org/support/evaluation-and-effectiveness
@imp_evidence
RESEARCH WITH TIER 1S
RECAP ON THEORY OF CHANGE
10
COMPONENTS OF NPC’S APPROACH TO MEASURING SOCIAL IMPACT
Map your theory of change
Prioritise what you measure
Choose your level of
evidence
Select your sources and
tools
Effective measurement framework developed
Strategic visionLeadership
Case for impact measurement
a) articulate how you think a service or intervention will have an impact;
b) and test your thinking against the
evidence you have available.
11
The essence of impact measurement:
Teaching assistants (TAs) trained to provide 1-1 support
Process for identifying children & families who will benefit
Pupils eat the breakfasts
Workplans better tailored to individual pupils needs
Engagement and enjoyment in maths lessons generally
More confident in Maths
Maths results amongst targeted pupils show improvement
Minimum of one hour of 1-1 tuition with TA per week
Inputs
Activities
Intermediate outcomes
Final outcomes
Description of resources deployed and targeting process
- Attendance records- Feedback from pupils,
teachers & parents
- Pupil behaviour measures (hard and soft)- Teachers perceptions- Mock test scores
Improved Key Stage test scores
LOGIC MODEL FOR A SCHOOL BREAKFAST CLUB
- Description of activities- Log of contact time- Example work plans
Healthy breakfasts available
Focus on activities/games to enthuse pupils
Pupils engage & enjoy a) tailored learning; b) group activities
Outputs
Targeted pupils attend and continue to attend
Behaviour improves
Accountability line
13
VALUE OF THE THEORY OF CHANGE APPROACH
• Give an account of impact. By this we mean providing a logical reason, or set of reasons, for why your products/service could have impact on outcomes.
NESTA: Standards for Impact Investing
• There should be a clear description of the service, including (i) the outcomes it aims to deliver, and (ii) who it is for
NOMS Commissioning Intentions
• Organisations applying for Level 1 need to demonstrate emerging evidence, clearly explaining the logic or the theory behind their work.
Project Oracle: What are the Standards of Evidence?
Which, if any, of the following techniques have you used in the last year?
Developed a theory of change
17% 49%YesNo - but would have liked to
14
NESTA / SRU / PROJECT ORACLE STANDARDS OF EVIDENCE
• Level 1: You can describe what you do and why it matters logically, coherently, and convincingly
• Level 2: You capture data that shows positive change but you cannot confirm you caused this
• Level 3: You can demonstrate causality using a control or comparison group
• Level 4: You have one + independent replication evaluations that confirm these conclusions
• Level 5: You have manuals, systems and procedures to ensure consistent replication and positive impact (fidelity)
http://www.nesta.org.uk/publications/standards-evidence-impact-investing
REVIEW OF THE CHAPLAINCY THEORY OF CHANGE
http://www.clinks.org/support-evaluation-and-effectiveness/demonstrating-outcomes
MEASUREMENT PRINCIPLES
http://www.clinks.org/support/evaluation-and-effectiveness
Specific Aim(middle of triangle)
Outcome(s) Outcome indicator(s)
Data collection method(s)
When and by whom
How and for whom will the data be used
RUN THE PROGRAMME AND COLLECT THE ASSOCIATED EVIDENCE
17http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Monitoring%20and%20evaluation%20framework.pdf
18
IMPACT MEASUREMENT TIPS
1. Use theory of change or logic model to articulate how your project or programme works
2. Make sure it’s informed by existing evidence/research
3. Focus on the immediate, short-term effects you have
4. Collect good quality qualitative evidence. Don’t cherry-pick
5. What administrative data is available? Only collect new quantitative data if you have to.
6. Try to measure the progress of service users longitudinally. Consider sampling rather than measuring everyone.
7. Consider your opportunities to measure the counterfactual
8. Publish and promote what you have learned
http://www.clinks.org/sites/default/files/Introduction%20to%20evaluation_final_0.pdf
INTRODUCTION TO THE CURRENT PROJECT
20
IMPACT READINESS FUND
CONTENT
FORMAT
23
REQUIRED FEATURES
• Monitor changes in outcomes between sessions: Internal and external
• Intuitive and easy to use
• Visual and numeric outputs. Exportable to Excel.
• Different views and levels of access
• High-level access to sign-off entries from low level users
• Personal data needs to be stored securely
• Flexible, adaptable
• Possible to quickly register an individual and add detailed information later
• Software should be usable on tablet and desktop platforms
• Unlimited users
• Link mentors, mentees and area administrators
• Off-line data entry
24
25
26
THOUGHTS?