“measuring what matters”

Deb James and Tim Vanson, Early Years Evaluation Team

• Share some of what we learnt from our evaluation – from areas

with plans in place

• Introduce an approach to “measuring what matters” and some

materials (all optional!) that you might find helpful

• Do a group exercise looking at how you could start to measure the

difference you’re making and tell a stronger story about your

successes

• Try to answer some of your evaluation questions or signpost you to

where you can get additional support

In today’s workshop we will …

Using information to make a judgement about how you’re doing;

assessing the value or worth of what you’re doing.

… and why does it matter?

It can help you in three main ways

• Accountability

• Learning and improvement

• Motivation and engagement

What IS evaluation?

What did we find in our

own recent evaluation?

Ten messages from our work with

areas that have plans in place

1. Plans vary but many areas share similar goals and priorities. Local

context is key to understanding what areas plan to do.

2. Some great stories about changes happening – some small, some

very significant – areas themselves are best placed to say what’s

small and what’s significant.

3. Partnerships worry about things being slower than anticipated

(though there is no national pressure on deadlines). Causes include

sometimes having over-ambitious goals or being a bit let down by

partners.

Learning about plans and progress

and reviewing …

4. For many areas in the first year delivering on their plan, “behind the

scenes” work had taken more energy and time than anticipated and

a lot of achievements were in setting up/maintaining partnerships

rather than “out front” (making changes in the community).

5. Following review activities, a common realisation has been that the

Big Local journey is up and down, even round and round in circles,

rather than in a straight line of growth and improvement – and that’s

OK.

6. Some anxiety about evaluation … It’s too soon? Where do we

start? Don’t we need special skills and knowledge? We need to

focus on doing the work not measuring the results?

…

7. Some things seem too hard to measure – eg, ambitious or long-

term changes, or “fluffy” changes.

8. A pause to reflect and review was helpful … it can be hard to find

the time but worth it

9. Areas are realising that not everything works – it isn’t always

comfortable, but it’s helpful to acknowledge this and Local Trust is

not expecting everything to work or run smoothly all the time.

10. Because things feel slow, there’s a danger that reviewing can

leave areas feeling demoralised so it’s good to think more

broadly about success and to value more than just how many

things have been ticked off a ‘to do’ list.

…

*

PLAN - Planning and identifying what matters most

- what you need to assess or measure and how you

might do it

DO - Collecting the information you need and

analysing it

USE - Using your findings – to make judgements,

to learn/improve, to report to others

What does evaluation involve?

1. PLAN

2. DO

3. USE

A common view is that evaluation is simply about collecting

information (eg, doing a survey) and then writing a report about what

you find. In fact it involves three main stages:

Stage 1:

Evaluation planning

Five tips to help you plan how you’ll assess your

progress – help you be clearer about what you want to

measure and how you might measure it.

1. PLAN

2. DO

3. USE

1. Define success

Example – an area’s success criteria

Our success criteria …

When considering how successful we have been, our Big Local will also consider

how true we have been to these values:

• we will have worked in an inclusive and respectful way

• we will have kept talking and listening (ongoing community engagement)

• we will have been bold and imaginative, not afraid to try new things

• we will have linked projects and activities together where it makes good sense

• we will have done the best we can

• we will leave a legacy behind

Residents will report an increased confidence in accessing job

readiness and employment support services and a more positive

outlook on improving their financial situation through work, having

been assisted through a new local high-street service that is friendly,

informal and community-led

What would you measure? It helps to get to the heart of the

change, and to express changes as single, simple statements.

2. Describe clear changes

Members of the community

will help us run the job club,

sharing their time and skills

with others

Out-of-work residents who

use the club will feel more

confident about seeking

support to find

training/employment

Out-of-work residents who

use the club will feel more

positive about their chances

of finding work.

We will set up an informal,

friendly job club in our Big

Local Hub to support long-

term unemployed residents

Activity Activity Change Change

We will offer youth nights at the community

centre

Young people will be more employable

We will offer group work

sessions during youth nights at the community

centre

Young people will improve

their social and communication

skills

Young people will be more employable

3. Have stories that work

Example – part of an area’s story map

The things we do or fund

to make a difference

(activities)

The differences we want to

make (changes/results) End result

1. Activities that preserve or

improve the appearance

of our green spaces and

open spaces

2. Activities that promote

community access/

enjoyment of our open

spaces e.g. growing

projects, allotments

Our open spaces are (and

look) better cared for

More people are involved in

activities that protect or

improve our local environment

Our open spaces are more

accessible and welcoming

More use of our parks and

open spaces (that is, they are

better used by all parts of the

community)

An

improved

environment

that is cared

for and

enjoyed by

our

community

*

Pause

Suggestions so far. You might want to take time out to …

(1) consider what success means to your partnership – what else,

besides what you’ve done and the difference you make, would you like

to assess yourself by …

(2) make sure you are clear about the changes you want to make and

how you think you’re going to make them, is it clear what the changes

are AND who they are for …

(3) check you are clear on how what you do will help bring about the

difference you want to make - checking your ‘story’ works … is it

worth trying to map/summarise your ‘change stories’?

4. Be specific

(break ‘big’ changes down)

• Breaking big changes down and being more specific about them

can make them easier to measure – that is, spend some time just

clarifying what you mean by ‘big’ changes like “community

cohesion” or “social inclusion”, “quality of life”, or “improved

environment”

• Some changes are long-term and lots of other factors come into

play … try to focus firmly on what’s realistically in your control

Example

Our big change

Our specific changes

(what an improved local environment

means to us)

Improved local

environment for

the whole

community

Cleaner open spaces

Safer open spaces

Our open spaces are better used by local

people

The community take better care of the space

What does the change look

like?

How will we know? What

signs or indicators would be

good evidence?

(things you can count,

things people tell you, things

you can see are all valid)

5. Ask ‘how will we know’ questions

Improved local environment for the whole community

Our specific changes

(what an improved local environment

means to us)

What we will count, measure or

observe

(signs of the change)

Cleaner open spaces Level of littering

Level of graffiti

Level of complaints

Level of use of dog litter bins

How clean people feel the space is

Safer open spaces ?

Spaces better used by local people ?

The community take better care of the

space?

*

Exercise - how will you know? (what evidence?)

Our specific changes

(what an improved local

environment means to us)

What we will count, measure or observe

(signs of the change)

Cleaner open spaces • Level of littering

• Level of graffiti

• Level of complaints

• Level of use of dog litter bins

• How clean people feel space is

Safer open spaces • Level of reported crime

• Number of accidents in play area

• People’s perceptions of crime

• How safe people feel spaces are

More visited/used by local

people

• Level of use by families

• Proportions/profile of those using (eg, age, families)

• Level of bookings for community events and attendance

at these

The community take better

care of the space

• Number of volunteers helping with litter picks and events

• Level of littering/graffiti (see above)

Our specific changes

(what better environment

means to us)

What we will count, measure

or observe

(signs of the change)

How we will

count it

(methods)

Cleaner open spaces • Level of littering

• Level of graffiti

• Level of complaints

• Level of use of dog litter bins

• How clean people feel space is

?

Safer open spaces • Level of reported crime

• Number of accidents in play area

• People’s perceptions of crime

• How safe people feel spaces are?

More visited/used by local

people

• Level of use by families

• Proportions/profile of those using

• Level of bookings for community

events and attendance at these

?

The community take better

care of the space

• Number of volunteers helping with

litter picks and events

• Level of littering/graffiti (see above) ?

The last part of the plan (collecting evidence)

Our specific

changes

(what an improved

local environment

means to us)

What we will count, measure

or observe

(signs of the change)

How we will

count it

(methods)

Cleaner open spaces • Level of littering

• Level of graffiti

• Level of complaints

• Level of use of dog litter bins

• How clean people feel space is

Annual litter pick

Council records

Survey

Safer open spaces • Level of reported crime

• Number of accidents in play area

• People’s perceptions of crime

• How safe people feel spaces are

Police records

Council records

Survey

More visited/used by

local people

• Level of use by families

• Proportions/profile of those using

(eg, age, families)

• Level of bookings for community

events and attendance at these

Question in schools

parent survey

Council records

The community take

better care of the space

• Number of volunteers helping with

litter picks and events

• Level of littering/graffiti (see above)

Monitoring records of

volunteers

(as above)

Stage 2:

collecting and analysing

information

1. PLAN

2. DO

3. USE

Some of the options open to you and some

good practice tips

Depending on your questions and the type of information you need (eg, numbers

or stories), you could use:

1. Survey questionnaires and feedback forms (your own)

2. Standardised questionnaires or scales (others)

3. Available data (info that others have collected)

4. Interviews

5. Focus groups

6. Participatory methods

7. Creative and arts

8. Case studies (often based on interviews)

9. Documents and records - your own (minutes, event reports) or others’

(news articles, correspondence)

10. Visual records (photographs, video)

Other?

Options for collecting information

1. Prioritise – you can’t measure everything or collect information

from everyone. What steps or changes matter most to telling

your story?

2. Be realistic – and proportionate – what you do to measure

your success shouldn’t get in the way of actually doing stuff.

3. Be lawyers not scientists? - think about ‘the best possible

evidence’ not ‘scientific proof’

4. Be a recycler! – what else is already being measured/counted,

what info is already out there that you could use for your

purposes?

5. Be flexible and open - this kind of work isn’t ‘neat’ and we are

all trying something new, taking risks and trying to learn more

about what works. Be open to using different approaches and

methods – counting numbers and capturing stories are both

valid.

Top tips for collecting information

…

6. Consider different perspectives – it can help to include

different groups and different viewpoints

7. Try to capture learning as well as success – try and create

the kind of environment where you can learn from what

doesn’t work as well as what does!

8. Think about how you’ll use information BEFORE you collect

it

9. Don’t collect information you won’t really use

10. ALWAYS do a test run first of any questionnaire or form you

use

…

Stage 3:

Using your findings

Reflect. Learn. Improve. Share. Celebrate.

1. PLAN

2. DO

3. USE

Remember that “the purpose of evaluation is to improve, not

prove.” (J Stufflebeam)

There are lots of good reasons for reviewing regularly, building

in regular review time as you go along, not least because it

helps you see if something’s not working and means you can

change direction more quickly if you need to.*

Your report to Local Trust will ask you to reflect on your

learning … should be a group activity, try to avoid one person

writing on behalf of the group as you can lose useful insights

and assumptions can be made.

Aim for honesty! “All my successes have been built on my

failures.” (Benjamin Disraeli)

Reflect, learn, improve

Don’t stop with your report to Local Trust – consider using: meetings,

community forums, websites, summaries, photo galleries, online case

studies, blogs, twitter, or Facebook timelines as ways to share your

results – eg:

• visuals to quickly share progress with milestones (Westwood and

Ravensthorpe)

• Facebook timeline for reporting and eye-catching posters instead of

formal wordy reports to update residents (Ramsey)

• Using story-telling and posting impact stories online (Allenton)

Share and celebrate