• Share some of what we learnt from our evaluation – from areas
with plans in place
• Introduce an approach to “measuring what matters” and some
materials (all optional!) that you might find helpful
• Do a group exercise looking at how you could start to measure the
difference you’re making and tell a stronger story about your
successes
• Try to answer some of your evaluation questions or signpost you to
where you can get additional support
In today’s workshop we will …
Using information to make a judgement about how you’re doing;
assessing the value or worth of what you’re doing.
… and why does it matter?
It can help you in three main ways
• Accountability
• Learning and improvement
• Motivation and engagement
What IS evaluation?
What did we find in our
own recent evaluation?
Ten messages from our work with
areas that have plans in place
1. Plans vary but many areas share similar goals and priorities. Local
context is key to understanding what areas plan to do.
2. Some great stories about changes happening – some small, some
very significant – areas themselves are best placed to say what’s
small and what’s significant.
3. Partnerships worry about things being slower than anticipated
(though there is no national pressure on deadlines). Causes include
sometimes having over-ambitious goals or being a bit let down by
partners.
Learning about plans and progress
and reviewing …
4. For many areas in the first year delivering on their plan, “behind the
scenes” work had taken more energy and time than anticipated and
a lot of achievements were in setting up/maintaining partnerships
rather than “out front” (making changes in the community).
5. Following review activities, a common realisation has been that the
Big Local journey is up and down, even round and round in circles,
rather than in a straight line of growth and improvement – and that’s
OK.
6. Some anxiety about evaluation … It’s too soon? Where do we
start? Don’t we need special skills and knowledge? We need to
focus on doing the work not measuring the results?
…
7. Some things seem too hard to measure – eg, ambitious or long-
term changes, or “fluffy” changes.
8. A pause to reflect and review was helpful … it can be hard to find
the time but worth it
9. Areas are realising that not everything works – it isn’t always
comfortable, but it’s helpful to acknowledge this and Local Trust is
not expecting everything to work or run smoothly all the time.
10. Because things feel slow, there’s a danger that reviewing can
leave areas feeling demoralised so it’s good to think more
broadly about success and to value more than just how many
things have been ticked off a ‘to do’ list.
…
*
PLAN - Planning and identifying what matters most
- what you need to assess or measure and how you
might do it
DO - Collecting the information you need and
analysing it
USE - Using your findings – to make judgements,
to learn/improve, to report to others
What does evaluation involve?
1. PLAN
2. DO
3. USE
A common view is that evaluation is simply about collecting
information (eg, doing a survey) and then writing a report about what
you find. In fact it involves three main stages:
Stage 1:
Evaluation planning
Five tips to help you plan how you’ll assess your
progress – help you be clearer about what you want to
measure and how you might measure it.
1. PLAN
2. DO
3. USE
Example – an area’s success criteria
Our success criteria …
When considering how successful we have been, our Big Local will also consider
how true we have been to these values:
• we will have worked in an inclusive and respectful way
• we will have kept talking and listening (ongoing community engagement)
• we will have been bold and imaginative, not afraid to try new things
• we will have linked projects and activities together where it makes good sense
• we will have done the best we can
• we will leave a legacy behind
Residents will report an increased confidence in accessing job
readiness and employment support services and a more positive
outlook on improving their financial situation through work, having
been assisted through a new local high-street service that is friendly,
informal and community-led
What would you measure? It helps to get to the heart of the
change, and to express changes as single, simple statements.
2. Describe clear changes
Members of the community
will help us run the job club,
sharing their time and skills
with others
Out-of-work residents who
use the club will feel more
confident about seeking
support to find
training/employment
Out-of-work residents who
use the club will feel more
positive about their chances
of finding work.
We will set up an informal,
friendly job club in our Big
Local Hub to support long-
term unemployed residents
Activity Activity Change Change
We will offer youth nights at the community
centre
Young people will be more employable
We will offer group work
sessions during youth nights at the community
centre
Young people will improve
their social and communication
skills
Young people will be more employable
3. Have stories that work
Example – part of an area’s story map
The things we do or fund
to make a difference
(activities)
The differences we want to
make (changes/results) End result
1. Activities that preserve or
improve the appearance
of our green spaces and
open spaces
2. Activities that promote
community access/
enjoyment of our open
spaces e.g. growing
projects, allotments
Our open spaces are (and
look) better cared for
More people are involved in
activities that protect or
improve our local environment
Our open spaces are more
accessible and welcoming
More use of our parks and
open spaces (that is, they are
better used by all parts of the
community)
An
improved
environment
that is cared
for and
enjoyed by
our
community
*
Pause
Suggestions so far. You might want to take time out to …
(1) consider what success means to your partnership – what else,
besides what you’ve done and the difference you make, would you like
to assess yourself by …
(2) make sure you are clear about the changes you want to make and
how you think you’re going to make them, is it clear what the changes
are AND who they are for …
(3) check you are clear on how what you do will help bring about the
difference you want to make - checking your ‘story’ works … is it
worth trying to map/summarise your ‘change stories’?
4. Be specific
(break ‘big’ changes down)
• Breaking big changes down and being more specific about them
can make them easier to measure – that is, spend some time just
clarifying what you mean by ‘big’ changes like “community
cohesion” or “social inclusion”, “quality of life”, or “improved
environment”
• Some changes are long-term and lots of other factors come into
play … try to focus firmly on what’s realistically in your control
Example
Our big change
Our specific changes
(what an improved local environment
means to us)
Improved local
environment for
the whole
community
Cleaner open spaces
Safer open spaces
Our open spaces are better used by local
people
The community take better care of the space
What does the change look
like?
How will we know? What
signs or indicators would be
good evidence?
(things you can count,
things people tell you, things
you can see are all valid)
5. Ask ‘how will we know’ questions
Improved local environment for the whole community
Our specific changes
(what an improved local environment
means to us)
What we will count, measure or
observe
(signs of the change)
Cleaner open spaces Level of littering
Level of graffiti
Level of complaints
Level of use of dog litter bins
How clean people feel the space is
Safer open spaces ?
Spaces better used by local people ?
The community take better care of the
space?
*
Exercise - how will you know? (what evidence?)
Our specific changes
(what an improved local
environment means to us)
What we will count, measure or observe
(signs of the change)
Cleaner open spaces • Level of littering
• Level of graffiti
• Level of complaints
• Level of use of dog litter bins
• How clean people feel space is
Safer open spaces • Level of reported crime
• Number of accidents in play area
• People’s perceptions of crime
• How safe people feel spaces are
More visited/used by local
people
• Level of use by families
• Proportions/profile of those using (eg, age, families)
• Level of bookings for community events and attendance
at these
The community take better
care of the space
• Number of volunteers helping with litter picks and events
• Level of littering/graffiti (see above)
Our specific changes
(what better environment
means to us)
What we will count, measure
or observe
(signs of the change)
How we will
count it
(methods)
Cleaner open spaces • Level of littering
• Level of graffiti
• Level of complaints
• Level of use of dog litter bins
• How clean people feel space is
?
Safer open spaces • Level of reported crime
• Number of accidents in play area
• People’s perceptions of crime
• How safe people feel spaces are?
More visited/used by local
people
• Level of use by families
• Proportions/profile of those using
• Level of bookings for community
events and attendance at these
?
The community take better
care of the space
• Number of volunteers helping with
litter picks and events
• Level of littering/graffiti (see above) ?
The last part of the plan (collecting evidence)
Our specific
changes
(what an improved
local environment
means to us)
What we will count, measure
or observe
(signs of the change)
How we will
count it
(methods)
Cleaner open spaces • Level of littering
• Level of graffiti
• Level of complaints
• Level of use of dog litter bins
• How clean people feel space is
Annual litter pick
Council records
Survey
Safer open spaces • Level of reported crime
• Number of accidents in play area
• People’s perceptions of crime
• How safe people feel spaces are
Police records
Council records
Survey
More visited/used by
local people
• Level of use by families
• Proportions/profile of those using
(eg, age, families)
• Level of bookings for community
events and attendance at these
Question in schools
parent survey
Council records
The community take
better care of the space
• Number of volunteers helping with
litter picks and events
• Level of littering/graffiti (see above)
Monitoring records of
volunteers
(as above)
Stage 2:
collecting and analysing
information
1. PLAN
2. DO
3. USE
Some of the options open to you and some
good practice tips
Depending on your questions and the type of information you need (eg, numbers
or stories), you could use:
1. Survey questionnaires and feedback forms (your own)
2. Standardised questionnaires or scales (others)
3. Available data (info that others have collected)
4. Interviews
5. Focus groups
6. Participatory methods
7. Creative and arts
8. Case studies (often based on interviews)
9. Documents and records - your own (minutes, event reports) or others’
(news articles, correspondence)
10. Visual records (photographs, video)
Other?
Options for collecting information
1. Prioritise – you can’t measure everything or collect information
from everyone. What steps or changes matter most to telling
your story?
2. Be realistic – and proportionate – what you do to measure
your success shouldn’t get in the way of actually doing stuff.
3. Be lawyers not scientists? - think about ‘the best possible
evidence’ not ‘scientific proof’
4. Be a recycler! – what else is already being measured/counted,
what info is already out there that you could use for your
purposes?
5. Be flexible and open - this kind of work isn’t ‘neat’ and we are
all trying something new, taking risks and trying to learn more
about what works. Be open to using different approaches and
methods – counting numbers and capturing stories are both
valid.
Top tips for collecting information
…
6. Consider different perspectives – it can help to include
different groups and different viewpoints
7. Try to capture learning as well as success – try and create
the kind of environment where you can learn from what
doesn’t work as well as what does!
8. Think about how you’ll use information BEFORE you collect
it
9. Don’t collect information you won’t really use
10. ALWAYS do a test run first of any questionnaire or form you
use
…
Remember that “the purpose of evaluation is to improve, not
prove.” (J Stufflebeam)
There are lots of good reasons for reviewing regularly, building
in regular review time as you go along, not least because it
helps you see if something’s not working and means you can
change direction more quickly if you need to.*
Your report to Local Trust will ask you to reflect on your
learning … should be a group activity, try to avoid one person
writing on behalf of the group as you can lose useful insights
and assumptions can be made.
Aim for honesty! “All my successes have been built on my
failures.” (Benjamin Disraeli)
Reflect, learn, improve
Don’t stop with your report to Local Trust – consider using: meetings,
community forums, websites, summaries, photo galleries, online case
studies, blogs, twitter, or Facebook timelines as ways to share your
results – eg:
• visuals to quickly share progress with milestones (Westwood and
Ravensthorpe)
• Facebook timeline for reporting and eye-catching posters instead of
formal wordy reports to update residents (Ramsey)
• Using story-telling and posting impact stories online (Allenton)
Share and celebrate
Top Related