Measuring your impact:a reference for students’ unions

2013

This reference guide has been produced specifically for students’ unions, based on the principles of social return on investment (SROI).

The key characteristics of SROI are:

- Stakeholders –the people who matter – are always at the heart of the process.

- Outcomes – this method shows the real benefits created for stakeholders, whether individuals, communities, society or the environment.

This reference guide is designed to be used alongside the NUS workbook, and sets out the methodology for a complete evaluation of students’ union impact.

ContentsImpact Assessment Basics 4

Stage 1: Identify purpose, scope and stakeholders 8

Stage 2: Understand what changes 11

Stage 3: Measure the change 15

Stage 4: Determine Attribution 18

Stage 5: Report 21

Introduction

What is impact measurement?Impact measurement is a process of understanding, measuring and reporting on the value created by an organisation. This value can be social, environmental and/or economic. As not all value can be captured by financial statements, this method focuses on the stakeholders who experience real change and lets them explain the value in a way that is meaningful to them.

A brief history of this guideThe National Union of Students (NUS) approached the New Economics Foundation (nef) to help them develop a method to allow student’s unions to measure and demonstrate their impact. This guide arose out of collaborative work between nef, the NUS and ten pilot students’ unions.

AcknowledgmentsThe authors would like to thank everyone who gave their time and expertise to support the production of this guide, especially Ben Ward and James Robertson (NUS), Jo Thomas (University of Birmingham Guild of Students) and the students’ unions whose work and feedback shaped the entire process.

Impact Assessment Basics

5

Impact Assessment Basics

Impact Assessment Basics

The Key PrinciplesThe impact assessment process essentially involves:

• Talking to stakeholders to identify what outcomes they experience

• Understanding how your work contributes to those outcomes

• Finding a way to measure your impact

• Demonstrating your value, and learning how to produce more of it

The method outlined here is based on three fundamental features:

1. Stakeholders are central

This analysis focuses on the people who are important to an organisation – its stakeholders. Each stakeholder group, such as students, the university or local community, is involved in identifying their own objectives, and reporting on how well those objectives are met. This guide will help you engage with your stakeholders and best represent their interests and opinions.

2. Theory of change – Know how your work creates impact

The things you do are designed to create change. Creating an impact map – a flowchart showing the journey from the inputs to the outcomes that emerge – helps you think about how your actions affect your stakeholders, and also gives you a clear way of telling your story to others.

3. Transparency – the process is open to scrutiny

Being open about your assumptions and decisions will make your findings more credible with your audience. It will also provide a good opportunity to test your assumptions and think about where your decision-making could be improved.

The BenefitsOne of the benefits of impact assessment is that you take a step back from day to day operations and get a broader view of your work and organisation. This helps to gain new understanding and refocus on the bigger picture. In addition, there are specific advantages to using the method outlined in this reference guide:

• Accountability – recording and reporting on your impact allows funders to see how their money is being used for real change.

• Better information for future decision making – this process highlights which activities are working well, where there are areas of weakness, and where resources could be reallocated to create more impact.

• Cost and time effectiveness – by focusing on the impacts that your stakeholders care about, you can learn the most crucial information in a relatively short space of time.

6

Impact Assessment Basics

The 5 stages of an impact assessmentThis guide will walk you through the 5 key stages of impact assessment, which will help you make the most of the workbook.

Stage 1: Define the scope

• Determine the main objective of this process

• Identify the most important activities or outcomes for assessment

• Select the stakeholders that will need to be consulted

• Determine materiality

Stage 2: Understand what changes

• Understand the difference between outputs and outcomes

• Engage stakeholders

• Create an impact map

Stage 3: Measure the change

• Select indicators

• Collect data

• Engage stakeholders (again)

Stage 4: Determine attribution

• Think about other factors

• Establish benchmarks

• Check for displacement

Stage 5: Analyse findings – report and embed learning

• Identify audiences and interests

• Communicate the findings they care about

• Embed the learning

7

Impact Assessment Basics

Before you startIt is important to be realistic about your resources before embarking on an impact evaluation exercise. It can be time consuming and iterative, so to avoid frustration and stress, here are some points to consider before you get started:

Time:

• How soon can you start, and when will you need to deliver the final report?

• Will the timeline of this project overlap with busy periods for the union, the university or other stakeholders?

• Will you be able to reach your stakeholders at critical points in your work plan? (e.g. will students still be on campus when you want to contact them?)

Try to draw up a timeline with key activities and milestones so you can plan what happens at what time of year.

People:

• Do you have enough people to carry out the work within the timeline?

• Do you have people with the right skills?

• Is there someone who can shepherd the project from beginning to end, to ensure continuity?

• Do you have someone you can ask for help if you get stuck?

Establishing the answers to these questions will help you start and finish the project.

Using this guideThis guide is designed to help you complete the workbook as you prepare an impact assessment. It will guide you step by step, from start to finish.

Each stage includes, where applicable:

• a description of the stage

• tips for completing the workbook tasks

• examples to provide more clarity

• a checklist to make sure you’re on top of things

Stage 1: Identify purpose, scope and stakeholdersIdentify your main aims, which activities and outcomes you want to measure, and who experiences a change as a result of your work.Focus only on what is key to your analysis.

9

Stage 1: Identify purpose, scope and stakeholders

IntroductionThis is the starting point for every impact assessment. Even though you already know what you do and for whom, it is important to work through this stage to ensure you haven’t missed anything, that everyone understands the scope, and to be explicit about your starting point.

Step 1: Determine your main objectiveWhy have you decided to embark on an impact assessment? Having a firm answer to this question will determine the scope of your work, your starting point, and shape everything you do along the way. You may want to bring a group of colleagues together for this discussion in order to get people involved from the start.

Be clear about your purpose and refer back to it at each stage to ensure you stay focused and on-track.

See workbook exercise 1.1 for some useful things to think about to get you started.

Step 2: Define the scopeOnce you know what you are trying to achieve, the next step is to decide what should be included, and what should be excluded from your analysis.

• Be realistic about your expectations. You cannot necessarily capture all your activities and impacts in one go.

• Are you trying to demonstrate the value of an aspect of your work, or of a specific union activity? In this case, start defining your scope in terms of the activities you want to showcase.

• Are you trying to find out which programmes are generating the most impact? In this case, identify the outcomes you want to measure and set your scope around them.

• How deep or broad does your measurement need to be? If you want to get a general idea of your impact, don’t include unnecessary details.

• Be realistic about your timing. Map out your project against other deadlines you will have to meet, the university calendar and scheduled events such as annual surveys. Try to build in extra time if you’re clashing with a particularly busy period.

• Be realistic about your resources. How many people are available to work on this project? What other work will they have on the go?

See workbook exercise 1.2 for some useful things to think about during the scoping phase.

Step 3: Select stakeholders One of the main principles of impact measurement is placing stakeholders at the centre of the process. This means that stakeholders should have the opportunity to provide input along the way, so you can ensure you are really representing their interests. This can also help you distinguish between what is truly important, and what is an optional extra.

Stakeholders can be any people or organisations who have an interest in your work. They may be the funders, beneficiaries, participants or staff. Each stakeholder group has its own objectives, and each interacts with your work in a different way. Some stakeholders influence your work by shaping it, enabling it or obstructing it. Equally, some activities might have an effect on some of your stakeholder groups and not others. It is therefore important to look at each group individually when measuring outcomes.

See workbook exercise 1.3 for some useful things to think about in selecting your stakeholders.

10

Stage 1: Identify purpose, scope and stakeholders

Stage 1 was really beneficial in thinking everything through. It was a big time investment but it paid off – University of Birmingham Guild of Students.

Step 4: Determine materiality

Materiality is an important concept in outcomes analysis, and applies to both activities/outcomes and stakeholders. Although students’ unions do many things and achieve many different outcomes, some are central to the organisation and others are supplementary. For example, if not for the students, the union would not exist. They are therefore material stakeholders. While managers at the halls of residence may appreciate the work of the union, the union could still function without them – they are likely not material in most cases. Determining materiality really helps to focus on why the union exists, and does the work it does. As you determine materiality, it is likely that you will decide to exclude some non-material factors from your analysis.

However, sometimes even material factors will need to be excluded from the analysis because of insufficient resources. For example, alumni might be material stakeholders, but tracking them down might be prohibitively complicated. Whenever something material has been excluded, be sure to provide an explanation in your final report so it is clear what your analysis applies to.

See workbook exercise 1.4 for prompts for thinking about materiality.

Stage 2: Understand what changesFind out which outcomes are most important to stakeholders then map them to your activities and outputs.

12

Stage 2: Understand what changes

IntroductionIn Stage 1, we created the broad-brush strokes for the outcomes analysis. We asked basic questions to determine some initial parameters for the project. In Stage 2, you start to link the different strands to find out, or demonstrate, how stakeholders, activities, outcomes and objectives all fit together into a single story. You will also find out which parts of this story are most important to your stakeholders, and how this might differ by stakeholder group or subgroup.

Step 1: Understand the difference between outputs and outcomesOutputs and outcomes are different but related ideas. Many organisations measure outputs as they are more concrete and easy to quantify. However, as you engage stakeholders you will see more and more that it is the eventual outcome that really matters.

Outputs

Outputs are the direct and tangible products from the activity; for example the number of people who received advice, or the number of students who were members of clubs.

Outcomes

Outcomes are changes that take place for stakeholders as a result of the activity, and are the most important things to measure. For example, a new job, increased income or improved quality of life that students’ union activities contribute to. This is more in line with the objectives of the stakeholder.

Outcomes can take place directly as a result of an output, or indirectly over time as the result of other outcomes being achieved.

Example – Student representative electionsEvery year the union runs elections for student representatives. When the union advertises for candidates to step forward, how many sign up? When the union has a drive to promote engagement in the process, how many people vote? These (generally) countable results are outputs.

So what are the outcomes then? Think about the objectives of your primary stakeholders – for example the candidates and the student body. The candidates may want some leadership experience, or a new social circle, or a reputational gain. The student body may want changes in university policy that make their educational experience more fulfilling. These can then be described as the desired outcomes.

See workbook exercise 2.1 for some tips for separating outputs and outcomes.

13

Stage 2: Understand what changes

Step 2: Engage stakeholdersNow you know how to define outcomes, it is time to find out what they are. The best way to determine what is important is to ask those who are most affected.

There are 3 main stages to understanding stakeholders’ outcomes:

1. Make assumptions/best guesses based on research and/or your experience of the stakeholders (ask front-line workers or volunteers for their impressions)

2. Collect information from existing documents or databases, such as past survey results, funding contracts etc.

3. Test assumptions and learning directly with stakeholders

You probably already have a fair idea of what stakeholders are interested in. You may also have several assumptions about their needs. It may be that some stakeholder types already explicitly state goals and objectives, for example in funding agreements. You may already do surveys that cover off this information. However, it is still a best practice to test these assumptions with the stakeholders themselves.

The more people you can get to participate, the better your results will be. So think about how to reach the most people at the minimum inconvenience. Different stakeholder groups respond to different approaches, and it is important to give some thought to your methodology as well as logistics before you start. For example, some students might prefer email contact so they respond when they have some free time. If you hold a focus group, think about people’s schedules and transport issues before setting the time and place.

It is also important to be sensitive about the amount of time and resources stakeholders can give to this process, whether they be staff, funders, or participants. So you should ensure that their time is used as efficiently as possible. Also, be explicit about your goals with the stakeholders. Introduce the research you are doing, explain why you want their participation, how the information will be used, and warn them that you may

be in touch down the line for additional information at the data collection stage (Stage 3). And always provide them with the opportunity to ask questions.

Some best practices1 for stakeholder engagement include the following:

• Ensure that a representative range of stakeholder groups is included in the analysis

• Ensure the independence and objectivity of people conducting the research – the more removed the interviewer is, the more likely people are to answer questions honestly

• Involve stakeholders in the design of the engagement process, and encourage feedback

• Acknowledge differences among stakeholders

• Ensure confidentiality

• Document the rationale and processes of stakeholder engagement methods chosen

Note: In an impact assessment you may need to engage stakeholders on two separate occasions. The first contact is to find out which outcomes they value most. It helps to validate the impact map and determine which outcomes to focus on. The second contact with stakeholders will be during the data collection phase in Stage 3, where evidence is gathered to demonstrate that change has taken place.

See workbook exercise 2.2 for some considerations for arranging and structuring stakeholder engagement sessions.

1 AA1000 Assurance Standard developed by AccountAbility, an international institute promoting accountability for sustainable development.

14

Stage 2: Understand what changes

Step 3: Create an impact mapOnce you have gathered information from your stakeholders, you are ready to compile the information into a coherent narrative. The impact map lays out the cause and effect chain that links the work you do with the value you create. It demonstrates the way that each activity leads to an output, the output leads to a short term outcome and ultimately how you reach your longer term outcomes.

By examining the sample impact map on the next page, you will see how each of the elements link together. You will also notice that there are not necessarily one to one relationships. For example, several activities may need to work together to produce a single outcome. Conversely, one activity on its own may have mutiple outcomes.

In reality, you will probably draft this impact map before engaging stakeholders, but will modify it several times after each engagement session. The first draft can be helpful in structuring the engagement sessions, or deciding what questions to ask. However, you need to be flexible so that stakeholer feedback can be included in your analysis in a meaningful way.

Note, there is no single way of representing an impact map. Use whatever method works best for you. Have a look at the examples in the workbook.

See workbook exercise 2.3 for some templates to help you complete your own impact map.

Impact Map Template

Stakeholder Activities Outputs Short-term outcomes

Long-term outcomes

Person or group of people who have an interest in – or are affected by – the activity and/or outcome

The things you do The direct result of the activity

The change that occurs as a result of the activity

Longer term changes due to the activity, or as a result of short term outcomes

e.g. student body e.g. run classes rep elections

e.g. class reps are elected to represent students

e.g. students’ concerns are represented to faculty

e.g. courses are improved in line with students’ needs

The impact map really helped us to visualise what we do and it made it easier to convey our story to students and other stakeholders” – University of Northumbria Students’ Union

Stage 3: Measure the changeChoose indicators that will show the change has happened, gather objective information to provide evidence and engage stakeholders for subjective corroborating data.

16

Stage 3: Measure the changes

IntroductionYou have now identified the outcomes you want to measure. You will need to decide how best to demonstrate that the outcomes have occurred, and provide evidence for your conclusions. The following steps will guide you through this process.

Step 1: Select indicatorsOften, outcomes are very difficult to observe or measure, especially things like confidence, wellbeing or self-esteem. An indicator is a way of knowing or estimating that the outcome has been achieved. Indicators are signs or signals that can be observed and measured. Some outcomes may require more than one indicator – for example, knowing whether someone has a good job might require an indicator of employment income and an indicator of job satisfaction.

It is preferable, where possible, to use a combination of subjective (or self-reported) and objective indicators that complement each other. In the example above, employment income is an objective measure whereas feelings of job satisfaction are subjective.

What to consider when choosing indicatorsEnsure your indicator really says what you think it does. For example, counting the number of election flyers distributed by candidates reflects more about their financial or time budgets than it does about their level of engagement or suitability. Be careful of taking logical leaps to suit the data you have available.

It may be necessary to reference data held by other organisations, such as the university, government departments or umbrella organisations (e.g. NUS). It is worthwhile thinking through possible sources of information with colleagues and undertaking some web research. If possible, reflect the indicators back to stakeholders or people who work directly with them, to see if they resonate.

See workbook exercise 3.1 to get started on your indicators.

Step 2: Collect dataAt this point, it is time to review your indicators and consider which data needs to be collected, both quantitative and qualitative, objective and subjective.

The data can come from the following sources:

• Primary stakeholders

People directly involved in the creation of value, for example participants or employees

• Organisations

Membership organisations, think tanks, government departments, market research firms, consulting companies, partner organisations

• Previous research

Universities, government departments, research organisations

Data can come from any number of sources, but ensure that all sources are credible and clearly documented, even if information was gathered verbally through interviews or through email information requests.

Establishing data availabilityHaving the right data is crucial to impact assessment. Consider the following questions.

• Does the organisation have systems in place for tracking its activities and its stakeholders?

• What systems could be put in place?

• Does existing data measure outcomes or outputs?

• Does it include measures of success?

17

Stage 3: Measure the changes

It’s important to remember that we are here to measure what needs to be measured, not what’s easy and nice to measure – University of Northumbria Students’ Union

Although data collection can be one of the most challenging aspects of this process, it is important to measure the most important outcomes, even if you can only do it for a subgroup or on a small scale. Where areas are excluded because of data limitations, it is important to be transparent about this and document it in your report.

See workbook exercise 3.2 to start planning the data gathering phase.

Step 3: Engage Stakeholders (again)Now you have started looking at available data for measuring outcomes, you have probably run up against some roadblocks. This is where it is particularly useful to go back to your stakeholders to get information and advice.

At this stage, the more data you can collect, the more robust your analysis will be, so think about the best way to reach the most people. Surveys are generally an efficient tool at this stage, but make sure the questions will give you the answers you want.

See workbook exercise 3.3 for top tips and best practices for creating and administering surveys.

Stage 4: Determine AttributionDetermine how much of the outcome is because of the work you do.

19

Stage 4: Determine Attribution

Introduction to attributionOne of the key principles of impact assessment is transparency. Transparency gives your findings added credibility in the eyes of the reader. So it is important not to overclaim – claim that all the benefits that stakeholders receive are from your work alone. For example, providing good job search support to a student certainly contributes to them finding a job, but their skills and qualifications are likely to be the more influential factors in that outcome. Taking this into account is referred to as attribution.

Step 1: Think about other factorsThere are two main ways to determine the proportion of the outcome that the organisation can take credit for. Neither is perfect, so choose the one you feel is most appropriate, and remember to document your decision and your reasons for making it.

• The first method is to simply ask stakeholders for their opinion. In Stage 3 when you are conducting surveys or other stakeholder engagement, consider including a question on attribution.

• The second method is to ask stakeholders who else has been involved in the outcome, and then make your own estimate of how much each has contributed.

See workbook exercise 4.1 for things to think about when assigning attribution proportions.

Step 2: Establish benchmarksBenchmarking is a critical concept when doing an impact assessment and helps avoid overclaiming. It estimates the number of people who would have experienced an outcome, even if they had not

participated in your activity. For example, of students who receive advice and then go on to improved academic performance, some percentage probably would have shown an improvement even without the advice service.

While it is impossible to know for sure what would have happened in an alternate reality, there are ways of estimating this. One way is to simply ask stakeholders for their own guess at how likely the outcome would have been without you. Another way of estimating a benchmark is to get data on a virtual control group, or benchmark group.

The benchmark group should resemble your stakeholder group as closely as possible. For example, if you compare employment outcomes of union members in your region with those of another region, you will not be able to account for differences in the job markets between those two places.

Some dimensions to think about when constructing a benchmark group are:

• Geographic area (different regions have different opportunities and underlying determinants of success)

• Socioeconomic background, if available (some groups face bigger challenges or have different aspirations)

• Time period (don’t compare recession job markets with boom markets)

• Other population characteristics (are there a high number of foreign students, or mature students with families?)

See workbook exercise 4.2 for some considerations and hints for benchmarking.

20

Stage 4: Determine Attribution

This part is really tricky. But we can see that it’s a really important part of the work” – Glasgow Caledonian University Students’ Association

Step 3: Check for displacementDisplacement is when the benefit for one person comes at the expense of another. For example, if the union helps a student find off campus housing, does that mean someone else in the city will not find housing? If so, this would lead to a net change of nil. While displacement is difficult to measure and may not be relevant to the particular project under impact review, it is still useful to think it through.

See workbook exercise 4.3 for some considerations for displacement.

Stage 5: ReportThink about your target audience(s), identify their key interests and communicate your findings effectively. Embed lessons learned within the organisation.

22

Stage 5: Report

IntroductionNow you have completed your assessment, all you need to do is tally up your results and think about what to do with them. This is the most important phase of the process – the analysis is meaningless unless it is used to communicate with stakeholders and improve the work of the organisation.

In presenting the results of your analysis, consider your audience, tailoring the discussion to each stakeholder. Stakeholders will have different objectives and relationships to your organisation. You may consider producing more high-level summaries for some readers, with more in-depth reports for others. For example, the university may be most interested in student retention and external reputation; whereas students might care more about social networks and academic achievement.

Be prepared to draft more than one version to get the most out of your reporting.

Step 1: Identify audiences and interests This is a good time to revisit the very first step you took in your impact assessment: defining your overall purpose. Who did you do this work for, and what did they want to know? What did you ultimately hope to achieve by undertaking the impact assessment? This will largely tell you your audience(s) for your final report, and help you think about how best to communicate your findings with them.

Of course, over the course of the process, you have probably thought of additional stakeholders who will be interested in your assessment. You may also want to share findings with the people who participated in stakeholder engagement so they know how their information was used.

See workbook exercise 5.1 for help with planning your communication strategy.

Step 2: Describe the impactThere are several different ways to talk about the impact of your work. The method you choose will depend on how you gathered your data, what your findings are and what you hope to achieve with the report.

Percentage population: One way to express an outcome is to describe the number of people – the proportion of your stakeholders – who experienced a positive change. For example, you might say that 75% of student representatives reported a better grasp of democratic representation through their role.

Distance travelled: Another way to talk about outcomes is to describe the magnitude of change that has taken place. For example, you might say that the average student representatives’ grasp of democratic representation increased from ‘very little’ to ‘a great deal’ over the course of their tenure.

However, where both options are open to you, consider which method will give the most fair representation of your impact – and don’t over claim! For example, you may say 100% of students reported higher confidence after participating in clubs. However, if most of those improvements moved them from ‘extremely low confidence’ to ‘low confidence’, reporting a 100% improvement might represent a greater outcome than actually occurred.

See workbook exercise 5.2 for some examples of impact assessment reports to get you started.

23

Stage 5: Report

Think of your first time as a practice run. It doesn’t need to be perfect – there’s a lot to learn and the next time will be much easier” – University of Cumbria Students’ Union

Step 3: Embed the learningAs you’ve moved through each stage of this process, you have probably come across a few issues of interest. Some of these issues will relate to the process itself – did you have enough resources and expertise to complete each stage properly? Would you recommend an additional staff member next time? Or, did you find some good sources of data that you’d like to draw on in future analyses?

It is good practice to keep a file on lessons learned so that future years’ analyses will be easier. Additionally, the findings from your first outcomes analysis can serve as a benchmark for future rounds.

Some of the issues you will have encountered will have relevance to the organisation as a whole. For example, if you struggled to find outcomes data on your stakeholders, you might consider the need for future surveying or better data collection. If outcomes were less positive than you expected in certain areas, perhaps a review of those activities is in order.

There is a lot of real value in taking the time to review learning and make recommendations for improvements in the long run.

See workbook exercise 5.3 to help you implement learning from your impact assessment.

24

Glossary

GlossaryActivity Actual interventions, programmes or services provided or facilitated

by the organisation. They may be outward facing, such as advice services, or behind the scenes, such as negotiation with the university. If the work has an impact on stakeholders, it can be considered an activity for the purpose of this analysis.

Attribution The consideration of what share of an outcome is attributable to, or results from, the organisation.

Deadweight The effect of what would have happened anyway, without your organisation’s intervention. For example, a certain number of participants in an ILM likely would have obtained a job without the programme.

Indicator Information that allows performance to be measured. It is a statistical value that links an organisation’s activities to their outputs and outcomes.

Materiality Importance and significance of information. Information is material if its omission or misrepresentation could influence stakeholder decisions.

Outcomes Changes in the individual participants (or other stakeholders) resulting from the activity: for example, a new job, increased income or improved stability in life.

Outputs The direct and tangible products from the activity: for example, the number of people trained.

Scope The range or extent of activity or influence.

Stakeholder Those people or organisations that affect or are affected by your organisation.

25

Resources

Resources – Useful Electronic Data SourcesOffice of National Statistics – Children, Education and Skills http://www.ons.gov.uk/ons/taxonomy/index.html?nscl=Children%2C+Education+and+Skills

Office of National Statistics – Labour Market http://www.ons.gov.uk/ons/taxonomy/index.html?nscl=Labour+Market

PlosONE – a free, open-access journal of peer-reviewed research http://www.plosone.org/home.action

Department for Innovation and Skills – Statistics and Analysis http://www.bis.gov.uk/analysis/statistics http://www.bis.gov.uk/analysis

Office of National Statistics – National Wellbeing Measures http://www.ons.gov.uk/ons/interactive/well-being-wheel-of-measures/index.html

26

Notes

Notes

National Union of Students

Macadam House275 Gray’s Inn RoadLondon WC1X 8QB t 0845 5210 262 f 020 7380 0794 e nusuk@nus.org.ukwww.nus.org.uk