Mountains or Molehills?
Reducing the perceived burden of monitoring and
evaluating outcomes
Rebecca
Colclough
Dr Mark Llewellyn
Workshop Aim
To provide participants with useful
background information and a practical step
-by-step guide to planning your evaluation
approach and developing a simple and
effective outcome monitoring questionnaire
for your project
STEP 1: Digging Around
• Use simple Google searches to do some research into your area of interest
• What evaluation tools and techniques are similar organisations using?
• Some common systems are available including Outcome Stars and Quality of Life Indicators: check if these could be appropriate
• Take stock each time you start a new project
• Talk to other people doing similar work
STEP 2: Set your Outcomes
“Outcomes are the difference your project
can make to your community or
beneficiaries”
Big Lottery
SMART Outcomes
• Specific• Measurable• Achievable• Realistic• Time-bound
– Significant– Meaningful– Attainable– Relevant– Resourced– Results-based– Timely
Outcomes need to be SMART before you can begin to work out how to measure whether you have achieved them or not!
Example 1
OUTCOME• Increased confidence
SMART OUTCOME• Older people who participate in the programme
will evidence an increase in their confidence to speak up for themselves, which will be shown immediately following their participation, and will still be evident 6 months later
Example 2
OUTCOME• Increased use of recreational facilities
SMART OUTCOME• The number of people using the recreational facilities
will be evidenced to have increased by 10% in the first year of the project and by 20% by the end of the project
Or• The number of people who report using more than one
of the recreational facilities at the Centre will have increased from 15% to 20% in the first year of the project and will reach 25% by the end of the project
STEP 3: Evaluation Questions
• Turn your SMART outcomes into evaluation questions
• Remember, evaluation is COMPARATIVE by naturepast with presentproject 1 with project 2method a) with method b)before and after
GOOD evaluation questions
• Easy to understand• Written in plain language (no jargon or
abbreviations)• Not too long• Only ask one thing per question • Don’t ask ‘leading’ questions • Offer mutually exclusive and exhaustive
categories• Can be closed or open-ended• Can use recognised ‘scales’
Ordinal Scales (LIKERT)
Excellent Very good Fair Poor
Strongly agree
Agree Neither agree nor disagree
Disagree Strongly disagree
Always Often Sometimes Rarely Never
None Very mild Mild Moderate Severe
Completely satisfied
Very satisfied
Somewhat satisfied
Somewhat dissatisfied
Very dissatisfied
Completely dissatisfied
Developing Evaluation Questions
SMART Outcome:
Older people who participate in the
programme will evidence an increase in
their confidence to speak up for
themselves, which will be shown
immediately following their participation, and
will still be evident 6 months later
…continued
Evaluation Metric:
What % of beneficiaries report an
improvement in their confidence to speak up
for themselves?
…continuedEvaluation Question:
To make this evaluation question comparative you must ask it at least twice: BEFORE and AFTER your project intervention (in this case the question must also be asked 6 months later as well)
STEP 4: Develop your Evaluation Plan
Develop an evaluation plan which details all
of your intended project outcomes and
evaluation questions, and sets out how you
will collect the data to monitor your
achievement of these, alongside wider
service monitoring and evaluation
Evaluation Plan: Contents (i)• Data reporting deadlines
• Service monitoring and evaluation questions (RBA ‘How much are we doing’ and ‘How well are we doing it’?)
• SMART outcomes (RBA ‘Is anyone better off’?)
• Outcome evaluation questions
Evaluation Plan: Contents (ii)• Use of monitoring and evaluation data
To meet funding requirements?To influence social policy?To gain further project funding? To scope a particular issue or problem?
• Evaluation participants (who you will collect the data from)?
Beneficiaries?Volunteers?Stakeholders?
Evaluation Plan: Contents (iii)
• The type of data you will collect
• The data collection methods you will use
Evaluation Plan: Contents (iv)
AnalysisHow you will analyse your results
ReportingWhat kind of reports you will need to produce from yourresults
DisseminationHow your findings will be shared to have thegreatest impact
Unfortunately there won’t be time today to go into these areas inmore detail
Quantitative / Qualitative
Quantitative data is essentially
NUMBERS
Qualitative data is essentially
WORDS
Quantitative
• Learning little from many: can make generalisations
• Standardised responses
• Non-deliberative• Numerical• Based on sample
sizes
• Learning a lot from a few: hard to make generalisations
• Open responses; conversational and flexible
• Deliberative• Non-numerical; ideas;
beliefs; meaning• Based on experience
Qualitative
What kinds of data can be collected?
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”
Albert Einstein
Data Collection Methods
Different types of data require different
data collection methods to be used
Quantitative• Survey• Routine monitoring
data collection (stats)
Qualitative• Interview• Focus group• Digital story
Ensuring Quality Data
• Making sure you collect the right data at the right time and in a standardised way is essential to making sure the data you collect, analyse and share is good quality
• Different ‘methods’ use different ‘tools’ to do this, such as a questionnaire, a structured interview schedule or a focus group discussion guide
What is an Outcome TOOL?
An outcome tool is something designed toensure you can assess or describe change in aconsistent and standardised way. Thiswill produce data which is of a better quality
Good quality data makes it possible to drawtogether information on a number of beneficiariesand give an overview of the change achieved for awhole project or group of projects
Developing an Outcome Tool for your Project
Measuring outcomes for beneficiaries
Simple and Effective
• Make the TOOL as useful as possible
• Simple design which is easy to follow
• Gather data which answers your evaluation questions
• Use recognised SCALES where appropriate to ensure quality
• Think about when and how the TOOL is used
Service Quality
Take the opportunity to
assess the QUALITY of
your service along with
beneficiary outcomes
Free Text Space
• It is always useful to include free text space for beneficiaries to say more about something
• Whilst not everyone will complete free text space, those who do will often have something constructive or positive to say
• You can get great quotes for future use!
Capture project ‘champions’
• Make sure there is space for beneficiaries to fill in their contact details if they wish
• Take the opportunity to ask if they would mind being interviewed or contributing further to your project evaluation
• User voice is exceptionally powerful so don’t risk losing it!
Simple Structure
• Order the questions well according to any natural timelines
• Move from familiar to unfamiliar
• Don’t ask sensitive questions near the beginning
• Limit the number of questions
• End with easy questions
• Make the completion instructions clear
Data Collection Points
“Timing has a lot to do with the outcome of a
rain dance”
Cowboy Proverb
Data Collection Points
• Baseline• Progress• End of Intervention• Impact
You must ask exactly the same questions eachtime when trying to evaluate outcomes. Servicemonitoring and quality questions can be one-off
Completion Method
• Face to face• By post• By the client alone or with support• By a proxy (for people with dementia)• By a volunteer
Standardising your data collection method
helps ensure the data gathered is high quality
Useful Resourceswww.ces-vol.org.uk
http://www.biglotteryfund.org.uk/er_eval_explaining_the_difference.pdf Link to a document called ‘Explaining the difference your project makes’
http://www.evaluationtrust.org/evaluation/evaluate
http://www.worldbank.org/ieg/ipdet/presentation/M_05-Pr.pdf Power point module on writing good evaluation questions
http://www.keene.edu/crc/forms/designingsurveysthatcount.pdfSlides on good practice in writing evaluation questions
http://www.homelessoutcomes.org.uk/resources/1/Guide%20to%20Outcomes%20Tools%20Second%20Edition.pdfReally useful publication describing outcome tools (written for the homelessness sectorbut more widely applicable)
Top Related