v
WORKING TOGETHER TO MEASURE IMPACTANNE KAZIMIRSKI, NPC
SIAA conference, Cergy-Pontoise, 10th December 2013
TRANSFORMING THE UK CHARITY SECTOR
2
NPC works at the nexus between charities and
funders
Charity
SectorFunder
Increasing the impact of charities
eg, impact-focused theories of change
Strengthening the partnership
Eg, collaboration towards shared
goals
Increasing the impact of funders
eg, effective commissioning
ConsultancyThink tank
3
• Charities and social enterprises under pressure to demonstrate impact
• New funding mechanisms require robust and comparable measurement
• But:
• Impact information is often of low quality, not embedded, not easily comparable, and rarely standardised
• Charities are spending resources developing custom measurement frameworks and tools
• Shared measurement is a potential solution
BACKGROUND
4
WHAT IS SHARED MEASUREMENT?
Shared measurement is using common tools to track outcomes across similar organisations and settings:
• a process: understanding a sector’s shared outcomes
• a product: any tool used by more than one organisation to measure impact.
5
WHO HAS TRIED TO IMPLEMENT SHARED MEASUREMENT?
Reviews 13 attempts at implementing shared measurement
6
HOW DO YOU SUCCESSFULLY IMPLEMENT SHARED MEASUREMENT WITHIN A SECTOR?
7
IS SHARED MEASUREMENT A GOOD IDEA?
Disadvantages
• Takes time to agree on shared values and goals
• Doesn’t reflect nuances of different organisations
• May not meet funder / commissioner requirements
• Risk comparing apples: oranges
• Reputational risk of comparability
• Data protection prevents sharing
Advantages
• Saves time and resources
• Reduces duplication in reporting
• Improves standards of impact measurement
• Externally validates impact data
• Promotes system thinking
• Allows comparing what works best for whom
8
• Funded by the UK government to develop a toolkit that will measure the outcomes of rehabilitation services
• Partnership with an academic institute and several charities
• Toolkit is for prison and post-prison programmes focusing on offenders’ relationships with their family and with their peers
OFFENDERS’ FAMILY RELATIONSHIPS
9
10
PROJECT PROCESS
Evidence review – identifying outcomes and toolkits
Consultation with providers & commissioners to select outcomes to focus on
Development and piloting of toolkit
Consultation with providers & commissioners to inform guidance
Analysis and reporting
12
DEVELOPMENT OF RELATIONSHIPS TOOLKIT
• Desistance theory:
• Link between creation/ investment in family relationships / reduction in peer pressure with reduced offending
• Consultation:
• Outcomes differed depending on the type of intervention & beneficiary
• Needed to reach a balance between being prescriptive and allowing for flexibility
• We researched existing scales and evaluations in each outcome area and prioritised:
• good validity and reliability; recently developed; brevity.
Toolkit: 25 standardised scales. For each outcome area: short general scales & longer specialised scales.
13
RELATIONSHIPS TOOLKIT STRUCTURE
We have identified 5 areas of outcomes shown below, supplemented by a personal development module developed by a parallel project:
Partner relationships
Child relationshipsWider family relationships
ParentingPeer
relationships
Personal development
14
OUTCOME AREAS
Partner relationships
Increased satisfaction with partner relationship, reduced conflict and improved commitment.
Child relationships
Quality of relationships with children, children’s well-being and behaviour.
Wider family relationships
Family functioning, resilience, conflict and communication, satisfaction with relationships in the family and quality of relationships
Parenting
Improved satisfaction with the parent-child relationship and parenting skills
Peer relationships
Satisfaction with peer relationships and reduction of negative peer relationships.
15
CHARACTERISTICS OF TOOLKIT
• The questions can be used as monitoring or as part of an evaluation with a ‘before and after’ (pre/post) research design, with a comparison group where possible
Before measure
After measure
Programme / service
Before measure
After measure
“Intervention” group
“Comparison” group
16
PILOTING DESIGN
Agreeing outcome areas and scales to include
Agreeing demographics/ risk factor questions
Tailoring to client population and service
Agreeing timing of post-questionnaires
Materials include:• Tailored pre and post questionnaires• Data collection sheets• Staff and client information sheets• Consent procedure
The timing of the post-questionnaires varies by provider, from one month to two.
Initial pilot feedback:• Suitability of peer relationship questions for older offenders• Capturing the end of negative partner relationships for women offenders
17
DISCUSSION
Working together across your sector to develop a shared measurement approach:
• What are/ would be the main challenges?
• What are the solutions to these challenges?
Top Related