GCN Evaluation Guide

7/30/2019 GCN Evaluation Guide

1/50

Evaluating GovernmentCommunication ActivityStandards and Guidance


2/50

Evaluating Government Communication Activity

2

Contents

Introduction 3

PROOF: ve guiding principles or evaluation 4

The big IDIA: the our-stage evaluation process 5

Stage 1: Identiy 6

Stage 2: Develop 8

Stage 3: Implement 17

Stage 4: Analyse and report 23

Conclusions 27

Appendix: Recommended metrics 28


3/50

3


IntroductionAs government communicators, were all aware o the need to make every piece o work

that we produce as eective and ecient as possible. To do this, we need to understand

whats already working well and where theres room or improvement. This in turn requires

us to evaluate our work and apply the learning rom this evaluation to uture activity.

Were also increasingly required to demonstrate how were applying evaluation in our

day-to-day jobs, through the plans and reports that we submit to the Eciency and

Reorm Group (ERG) or activities with a spend o 100,000 or more and the inormation

that we supply to eed into the annual communication plan. The revised GovernmentCommunication Network (GCN) Core Skills or Government Communicators will set out

clear evaluation standards that we should all ollow, based on our grade and the discipline

that we work in.

To help us evaluate our activity eectively, in line with expected standards, we need clarity

on what good evaluation practice looks like. This guide sets out an approach to evaluation

that should be ollowed as a minimum or all government communication activity,

regardless o size, discipline or budget. This approach is pragmatic and ocuses on helping

you to produce the best possible evaluation given the scope o your activity and the time,

resource and budget that you have available or evaluation. By ollowing the guide, you

can be condent that you will produce an evaluation that meets the required standards or

your activity and or your role.

I youre new to evaluation, use the guide to help you get started, recognising that it will

take time to build up your approach and gather whats working. Remember, a partial

evaluation is almost always better than no evaluation at all. I youre already evaluating

your activity eectively, ensure that youre ollowing the standards required o you, making

modications as necessary. Think about how you can share what youve learned with

others to help them evaluate more eectively.

By evaluating the activity that we carry out, we will be able to improve the eectiveness

and eciency o our work over time in the uture. We will also be able to demonstrate the

contribution that well-planned and executed communication activity makes to governmentoverall and hence justiy urther investment in our work.
https://gcn.civilservice.gov.uk/gcn-core-skills/https://gcn.civilservice.gov.uk/gcn-core-skills/


4/50

4


P Pragmatic best availablewithin budget, not best ever

A pre-planned, but partial, evaluation is better than no evaluation at all. Be transparent: acknowledge the gaps in your

evaluation and the implications o these gaps. Even i you are not able to ully quantiy the eect o your communication

activity, you will still be able to draw valuable learning rom the evidence that you have obtained.

R Realistic prove what you can,acknowledge what you cant

Dont worry i you can only collect a small amount o data in the short term. By establishing a robust evaluationramework that is linked to a clear set o communication objectives, you will be able to interpret and analyse whatever

inormation you gather. Over time, you will be able to build on this knowledge, increasing the amount o data that you

collect rom each subsequent activity.

O

Objective approach yourevaluation with an open mind

Be honest and constructive about what was achieved, so that we can all learn or the uture. Learn rom your successes

and rom things that didnt work as well as youd hoped. Use this to rene your uture strategy.

O Open record and shareas much as possible

Share your learning as widely as possible so that colleagues can also benet rom your experience. Work with GCN to

develop a detailed case study.

F Fully integrated integrate evaluationinto activity planning and delivery

Plan ahead. Start thinking about how to evaluate your activity as soon as possible, ideally well beore it begins. This will

help you to put the right mechanisms in place or subsequent measurement and data collection. Retrospective evaluation

is oten less eective because the right data may not have been collected or objectives may not be measurable.

PROOF: fve guidingprinciples or evaluation

Whatever the size or scope o your communication activity,ollowing these guiding principles will help to ensure thatyour evaluation is as eective as possible.


5/50


5

You can evaluate all kinds o communication activity,including press and media management, marketing andinternal communication. Follow the our-stage process set out

in Figure 1 below and you will carry out an eective evaluationin line with the ve guiding principles or evaluation.

Figure 1: The our-stage evaluation process

The big IDIA:the our-stage evaluation process

1

2

3

4

Identiythe scope o your project

Developyour evaluation plan

Implement source data to measure perormance

Analyse and reportperormance against the plan


6/50


6 Stage 1: Identiy

Stage 1: Identify thescope o your project

Checklist

Task: to dene what you need to evaluate by asking:

What activity am I evaluating?

What do I already know?

What is my evaluation expected to achieve?

Output: summary o your proposed evaluation approach

What activity are you evaluating?

Begin by establishing exactly what you are going to evaluate. What activities do you need

to include? Are these activities part o a wider communication strategy?

Identiy the time period over which you will evaluate the activity. For a one-o piece

o activity with clear start and end dates, this is generally straightorward. For ongoing

activity, you need to identiy and agree appropriate time periods or evaluation, i.e. how

oten you will assess perormance against objectives.

Examples o types o activity

Press For ongoing reputation management work, its oten most

eective to track the eect o your work over time. Identiy the key

messages that you want to track and monitor coverage on a regular

basis, providing monthly or quarterly updates rather than reporting

on individual activities.

Marketing I the activity that you are evaluating runs across a range o

channels (which might include, or example, paid-or activity,

partnerships, leafets, a website or social media), check that youve

included all o them in your plan.

Internal

communication

I you are evaluating the eect o a change management

programme, ensure that you include all the elements o the

programme in your evaluation plan (e.g. briengs to senior sta,

communication via email and the intranet, events, training etc).


7/50


Stage 1: Identiy 7

What do you already know?

Review similar previous activity that your team or others have run to see what you can

learn. This will give you a benchmark to measure perormance against and may help to

rene your objectives and your approach to the activity that you are about to run.

$ Useful tip gathering evidence or ERG exemption requests

I your activity is subject to ERG approval, use your previous evaluation results

as evidence or why you believe it will meet its objectives in sections 3 and

10 o the exemption request orm.

What is your evaluation expected to achieve?

Ask the ollowing questions to help identiy what is expected rom your evaluation:

Whatarethekeyquestionsthatyourevaluationreportneedstoanswer?

Whatlevelofdetailisexpectedfromyourreportandwhatformatdoesitneedtobein?

Isanybudgetavailableforyourevaluation?

Whowilldothework?Thismightincludeyouandothersonyourteam,research,

analysis and evaluation specialists rom within your hub or the Shared Communications

Service, an external agency or a combination o all o these.

$ Useful tip ERG evaluation standards

I your activity is subject to ERG approval, there is a standard ormat that you

must use or your evaluation report. A report template can be ound here.

Ensure that your evaluation plans are designed with this in mind

Summarising your approach to evaluation

Beore beginning work, set out your proposed approach to evaluation, basing this on the

inormation that you have gathered so ar. Depending on the scope o the activity that

youre evaluating and the person or whom you are carrying it out, this may be a detailed

proposal to stakeholders in your department, an email to your manager or a note to

yoursel. Whatever ormat you use, you can nd a useul template here that sets out the

questions you should consider.

$ Useful tip managing expectations

Sometimes, there may be a gap between the time, resource and budget

that you have available and others expectations o what your evaluation can

achieve. Discuss whether its better to increase available resource, budget or

time or to lower expectations. Decide this early in the process.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-approach-template.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-approach-template.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.doc


8/50


8 Stage 2: Develop

Stage 2: Develop yourevaluation plan

Checklist

Task: to dene how youll measure

success by:

Setting out activity objectives

Dening target audiences

Mapping out how your activitywill work

Setting perormance metrics

Agreeing metrics and targets

Output: drat evaluation plan

Setting out activity objectives

Clear and measurable communication objectives are the cornerstone o any evaluation

plan. They set out what your activity aims to achieve and the overall goals against which

you should judge success.

Your objectives should already be identied in the communication strategy or the

activity you are evaluating. It may be useul to summarise them as shown in Figure 2,

clearly demonstrating how each sub-objective links to the overall communication and

departmental objective.

Startbyidentifyingtheoveralldepartmental objective and the issue it is designed

to address. Your activity will be part o a set o interventions which link back to this

objective.

Next,identifythecommunication objective. This is the overall role that

communication is expected to play in achieving the policy objective.

Differentchannelsoractivitiesmayhavedistinctrolestoplayinachievingtheoverall communication objective. Each o these should be set out as a separate

communication sub-objective.

You may only be evaluating the eectiveness o one communication sub-objective, but

it is important to understand how this is expected to contribute towards the overall

communication objective.

Ensure that all communication objectives are clear and set out what each activity was put

in place to achieve, together with a measure o success. Always consider whether or not

you will be able to prove a communication objective has been met. I not, it will need to

be revised.

You may nd it useul to map out your objectives as shown in Figure 2.


9/50


Stage 2: Develop 9

Figure 2: Hierarchy o objectives

Sample objectives might include:

Press Departmental objective: To ensure that compliance with a new tax

regulation is above 80%.

Communication objective: To ensure that the majority o the general

public understands the reasons why complying with the regulation

benets the economy.

Press-specic sub-objective: To ensure that the public is given a air

and balanced view o the policy, via the media.

Marketing Departmental objective: To get 10,000 more people working as

community service volunteers in your area.

Communication objective: To get 40,000 people in the area to

register as potential volunteers on your community website.

Sub-objective 1: To increase the proportion o the public who

recognise the value o volunteering rom 20% to 40%.

Sub-objective 2: To get 80,000 people to visit the website and nd

out more about how to volunteer.

Sub-objective 3: To secure 40,000 incremental registrations.

Internal

communication

Departmental objective: To ensure that unauthorised sta absences

all by 50%.

Communication objective: To ensure that all sta are able to ollow

the correct processes or reporting absences rom work.

Sub-objective 1: To ensure that all sta recognise that there is a policyor reporting absences rom work and that they must ollow this.

Sub-objective 2: To ensure that all sta understand how to access the

guidance on how to report absences.

Departmentalobjective

Put in place to address

a specic issue

Includes: policy development,

policy delivery, reputationmanagement

Communicationobjective

Role that communicationwill contribute to

achieving departmental

objective

Communicationsub-objectives

Role that individualactivities or channels

will play in meeting the

overall communicationobjective


10/50

10 Stage 2: Develop


Dening target audiences

All communication activity has an end audience the people at whom it is ultimately

targeted. But an activity may also have an intermediary audience a group o people

targeted so that they will deliver the message to the end audience on your behal. Typical

intermediaries include the media, stakeholders, such as non-governmental organisations

and charities involved in delivering a policy objective, and commercial partners working

with you to deliver a piece o marketing activity.

I your activity includes an intermediary audience, you should evaluate:

howeffectivelytheintermediarywasengagedbytheactivity

howeffectivelytheintermediarycommunicatedthemessagetotheendaudience.

Audiences might include:

Press Where you are using a media engagement or PR campaign to try and

raise volunteer levels among the public overall:

Intermediary (the press): How did the media react to the activity

targeted at them? Did they eature your messages, what volume

and quality o coverage did you get or the story?

End audience (the general public): How many people volunteered asa result?

Marketing Where you are trying to engage a partner to run events promoting

volunteering on your behal:

Intermediary (partner): How eectively did you engage the partner

and how many events did they run as a result?

End audience (the general public): How many people volunteered as

a result?

Internal

communication

I you are training sta on how to communicate better with

members o the public so that customer satisaction improves:

Intermediary (sta): How eectively did you train them? Did they

put their skills into practice in their interaction with the public?

End audience (the general public): Did they notice that they received

a better service rom trained sta? Were they more satised?

$ Useful tip intermediary audiences

I your activity includes an intermediary audience, make sure you include

perormance metrics that enable you to: (1) evaluate the eect o youractivity on the intermediary; and (2) evaluate the eect o their activity on

your end audience.


11/50


Stage 2: Develop 11

Mapping out how your activity will work

Spend some time thinking about how your activity is expected to achieve its objectives. I

it is successul, what messages will the target audience(s) see, what will they think or eel

and what will they do? Mapping out the steps to success will help you identiy the right

perormance metrics to evaluate your activity. Draw on any behavioural insight1 modelling

or customer journey work that has already been done.

Setting perormance metrics

Having identied the objectives and target audiences or your activity and mapped out

how you expect it to work, you need to build a set o performance metrics. These are

the measures you will use to assess the activitys perormance against its objectives and to

identiy which elements o the activity were most and least successul.

Make sure that your evaluation plan includes a range o perormance metrics rom the

ollowing ve categories:

Figure 3: The ve types o perormance metrics or evaluation

Using these ve categories as a guide will help you to pick the right perormance metrics

or your activity. The categories can be applied equally to the ull range o press, marketing

and internal communication activity.

1 For more on behavioural insight, see MINDSPACE: Infuencing behaviour through public policy (Institute or Government/Cabinet Oce www.instituteorgovernment.org.uk/our-work/better-policy-making/mindspace-behavioural-economics)

1. InputsThe activity carried out

2. OutputsHow many people had the opportunity

to see or hear your activity?

3. Out-takesWhat was its immediate eect on them?

4. Intermediate

outcomesDid they do anything as a result o your activity?

5. OutcomesDid you achieve your overall objective?


12/50


12 Stage 2: Develop

Inputs

Inputs include details o the actual activity that has been undertaken, including the

channels that you used to communicate. The channels to include will have been identied

at Stage 1.

Examples o input metrics

Press Number o press releases sent out or engagement work carried out

Marketing Paid-or media plan

Website or digital space created

Number o partners contacted, types o message shared or requests

made o partners (intermediary audience)

Internal

communication

Number o sta events organised

Number o briengs or training sessions organised

Web content created and put on to the intranet

Include the costs o carrying out the activity in your input metrics. Include all externaland internal costs and time spent (including sta time). When comparing results or

more than one piece o activity, use a consistent methodology to record the costs and

time spent against each one. This will be essential or calculating return on marketing

investment.2

$ Useful tip choosing the right perormance metrics

At this stage, dont worry about whether you can get data or the

perormance metrics you choose. Pick those that you would need or an

eective evaluation. Stage 3 looks at how to secure data and deal with gaps.

2 For more inormation reer to Evaluating the nancial impact o public sector marketing communication: An Introduction to Payback, Return onMarketing Investment (ROMI) and Cost per Result (https://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/intro-to-payback-romi-and-cpr.pd)


13/50


Stage 2: Develop 13

Outputs

Output metrics measure the number o people who had the opportunity to see or hear

your activity, regardless o whether they recall or recognise it. Try to include:

Reach the total number o people or organisations in your target audience who were

exposed to your activity.

Frequency the number o times they saw or heard the activity.

Examples o output metrics

Press Number o pieces o coverage achieved

Frequency o exposure to coverage by end audience

Marketing Proportion o the target audience reached by media activity

Number o impressions (one page-view) on the web page

Number o stakeholders you contacted and number o contacts

made

Internal

communication

Number o sta attending events and training sessions

Number o impressions on the intranet

$ Useful tip activity mapping

Use the activity map that you created earlier to help you identiy the right

out-take, intermediate outcome and outcome perormance metrics or

your activity.

Out-takes

Out-take metrics look at the impact that the activity had on your target audiences awareness,understanding and attitude. Think about what you wanted people to recall, think or eel about

your activity and include perormance metrics that allow you to measure this.

$ Useful tip other metrics

Perormance metrics or out-takes, intermediate outcomes and nal

outcomes cannot be standardised in the same way as those or inputs and

outputs. They will need to be tailored to refect how you expect your activity

to work and what it is trying to achieve.


14/50


14 Stage 2: Develop

Examples o out-take metrics

Recall How many people are aware o your activity or the message(s) that

it is promoting?

Think How many people understand the key messages that your activity is

trying to get across?

Feel What eect has your activity had on peoples attitudes? Do they

intend to behave dierently as a result o your communication?

Intermediate outcomes

Intermediate outcome metrics capture any action taken by the target audience as a result

o your activity which may lead to the eventual end behaviour. Think about including

perormance metrics that allow you to measure the ollowing:

Examples o intermediate outcome metrics

Talk How many people discussed the activity or its message with peers,

riends and amilies? How many partners or stakeholders discussed

it with their work colleagues?

Direct response How many people responded to or otherwise interacted directlywith you as a result o the activity? This could include visiting a

website, attending training, ringing a phone line or having a

ace-to-ace discussion.

Indirect response How many people responded to or otherwise interacted with third

parties as a result o your activity? This could include people interacting

with stakeholders or partners or other local and national services.

Other actions How many people took (or claim to have taken) any other action as

a result o your activity? For example, in a stop-smoking campaign

this might include people buying books, patches or other similar

products.

Reer back to your activity map and make sure that you include measures to cover the ull

range o actions that people could have taken.


15/50

Stage 2: Develop 15


Outcomes

Outcome metrics look at the eect that an activity has had on the overall communication

and policy objectives that it was put in place to address. Include metrics that enable you

to measure whether your communication objective was met and the eect your activity

had on the wider policy objective. This could be related to changing behaviour, adopting

a service, increasing positive reputation, increasing understanding and awareness or

increasing participation.

Y The appendix to this document provides a list o recommended metrics oreach component. You can use this to help identiy the appropriate metricsor your evaluation plan. This will enable you to compare your activity with

other cross-hub work and will also ensure that your plan meets the standards

required by ERG where applicable.

Agreeing KPIs and targets

You may decide to set a small number o key perormance indicators (KPIs) based on

the perormance metrics that you have chosen or KPIs may already have been set by

your team, department or hub. KPIs are measures o success that can help you track

how well an activity is progressing towards its end objective or contributing to a broader

communication strategy.

KPIs are particularly useul where your activity wont achieve its end objective or some

time. They will enable you to track how eectively the activity is progressing towards

this end objective and may provide timely inormation to make changes to the existing

communication plan, i necessary. KPIs are easier to set when the activity has run a

number o times beore, as you will have a better eel or which metrics most accurately

predict how it will ultimately perorm.

KPIs may be single perormance metrics or combine several metrics.

Youmaychoosetobringtogetherresponsesbyphone,weborface-to-facetogivea

total number of responses.

Wherepeoplesoverallsatisfactionisdrivenbyfourorvedifferentfactors,youcould

bring these together in one composite satisfaction measure.

How to set KPIs

YoudontneedKPIstoevaluateeffectivelyonlysetthemiftheyareusefulin

helping you understand whether an activity is progressing as expected or contributing

eectively to overall goals.

EnsurethatyourKPIsaremeasurable.

SelectnomorethanveKPIsforyourendaudience(andthesameforanyintermediary

audiences). SettargetsforeachKPIandspecifythetime-frameinwhichyouexpecttoachieve

them.


16/50


16 Stage 2: Develop

Maketargetsasmeaningfulandasrealisticaspossible,drawingonpreviousresults

where applicable. The less historical data you have available, the broader your targets

should be.

Asfaraspossible,benchmarktargetsagainstotheractivitycarriedoutbyyouand

others within your team, department or hub. This will give you a broader context or

what success looks like.

Creating the evaluation plan

Your evaluation plan should bring together:

theobjectivesandtargetaudiencesthatyouwillevaluateperformanceagainst

theperformancemetrics(andKPIsandtargetsifyoureusingthem)thatyouwillusein

your evaluation.

Your team or department may already have a standard template or evaluation plans.

I not, you may nd these templates or activities with and without an intermediary

audience useul.

$ Useful tip multi-channel activity

For more complex activities, include a separate set o perormance metrics

or each activity/channel and audience.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.doc


17/50

Stage 3: Implement source datato measure perormance

Checklist

Task: to identiy and gather evaluation

data by:

Identiying available data andevidence

Creating proxies and assumptions

Using monitoring, market researchand eedback

Reviewing any remaining gaps

Agreeing who will collect data

Completing the evaluation planOutput: completed evaluation plan

At this stage, you need to source data and evidence or the perormance metrics you

identied in Stage 2. Budget, time and resource may restrict the amount o data you are

able to gather but this should not stop you evaluating your activity. Its better to produce

an evaluation report with gaps in it than to produce nothing at all, provided that you are

clear about what is missing.

Identiying available data and evidence

Begin by identiying the perormance metrics or which data is immediately available. This

will come rom three main sources:

datagatheredfromyouractivity

datagatheredbystakeholdersandpartners

existingdatasources(e.g.governmentdataandwidermediaandlifestyledata).

Data gathered rom your activityGather as much data as possible directly while your communication activity is running.

Identiy all the ways in which people can respond to it and ensure that data is being

gathered or each one. Data might include web visits, telephone calls or ace-to-ace

interaction. Also, look or ways to gather additional data rom these responses or

example, asking or peoples personal details (while remaining mindul o the requirements

o the Data Protection Act 1998) or asking or permission to contact them again or

ollow-up research.

$ Useful tip plan ahead

Always try to identiy the data sources that you plan to use in your evaluation

beore the activity actually runs. This will give you time to ensure that theright data is being gathered in the right ormat while the activity is live.




18/50


18 Stage 3: Implement

Data gathered by stakeholders and partners

Look at what data or evidence could be collected by an agency, stakeholder, partner

or colleague working with you to deliver the activity. This might include the number o

responses to an event or helpline run by a partner; any eedback received by stakeholders;

data rom surveys; competitions or promotions; and website statistics.

$ Useful tip getting data rom agencies

When youre procuring an agency to work on planning, running or

evaluating an activity or you, always ask what data they can provide to eed

into the evaluation at the procurement stage. This can then be built into their

contract. Ask or data well in advance and, where appropriate, agree to sharethe results o your evaluation.

Existing data sources

As well as resh data gathered rom current activity, you may nd it useul to look at data

sources that already exist to help you put your results in context and back them up with

supporting evidence.

For example:

Media consumption surveys such as NRS (newspapers), BARB (TV) or comScore (digital

media) provide inormation on the number o people reached by dierent media channels.

These are particularly useul or input and output measures.

Syndicated consumer lifestyle and media surveys such as TGI, TouchPoints or ACORN

can be used to build up liestyle and behavioural proles or a range o audiences. These

are particularly useul or tracking longer-term outcome measures and optimising uture

campaign planning.

Existing government demographic and research data gathered centrally or by

individual departments can be used or measuring longer-term outcomes. The Oce or

National Statistics publishes a number o surveys across all sectors o public interest.

For more inormation on how you might be able to use and access these and other

surveys, talk to colleagues in your departments or hubs research and analytics teams, the

Shared Communications Service or external media and research agencies.

Creating proxies and assumptions

I you cant get the exact data that you need or a perormance metric, look at whether

you can source a similar piece o evidence as an alternative. This is known as a proxy

measure. For example:

ifanactivityisaskingpeopletochecktheirsmokealarmsmoreregularly,agood

proxy would be the number o nine-volt batteries being sold (a type o battery almostexclusively used in smoke alarms).


19/50



Is your activity similar to other activity run in the past or which youve got accurate

results? I so, you might create an assumption that this activity will perorm in the same

way. For example:

ifyouractivityencouragespeopletosignupforastop-smokingservice,youmaynot

have the budget to carry out research to see how many people have given up smoking

as a result o using the service. However, all things being equal, you could assume that

cessation rates were the same as in previous research studies.

Using monitoring, market research and eedback

I gaps still remain in your evaluation plan once you have exhausted all the available data

sources, consider lling them using bespoke monitoring, market research and eedback.

Monitoring

You can use monitoring to track how many people your communication is reaching. The

main monitoring techniques and tools include:

PR and media monitoring: I your activity is designed to generate media coverage,

you could use an agency to track how many people it reached, how many times the

key messages were mentioned and how avourable the coverage was. I you dont have

the budget or this, consider whether you could monitor coverage yoursel, either by

subscribing to a PR-monitoring service such as Gorkana or by looking at whats being saidacross a representative selection o media channels.

Web monitoring: The Government Digital Service (GDS) measures a range o standard

digital metrics or gov.uk and other government and partner sites. Topline data is available

in the regular reports provided by GDS and more detailed analysis will be available on

request. I you are responsible or monitoring perormance o a stand-alone website, there

is a range o ree and paid-or tools that you can use to monitor perormance, including

Google Analytics.

Social media monitoring: GDS will also provide guidance on social media monitoring.

There are various tools that you can use to monitor perormance yoursel:

WhereyouractivityishostedonasocialmediasitesuchasFacebook,LinkedInor

YouTube, you can set up standard user reports to monitor interactions with your

content. Look both at how many people interact with you and at the quality o these

interactions.

IfyoushareinformationviaTwitter,monitorhowmanypeoplefollowyouandhowmany

retweet your messages. Tweetdeck enables you to analyse interactions more eectively.

Buzz-monitoringtoolsenableyoutoseewhetherpeoplearecommentingonlineabout

your message, and whether the comments are positive and rom credible sources.

Eective buzz-monitoring relies on you dening the terms that you want to monitor in

advance; the more specic you can be, the more accurate the results. There is a wide

range o ree and paid-or buzz-monitoring tools. Google Alerts is one example o aree tool.

Thereisarangeoffreeandpaid-fortoolsthatcanbeusedtomonitorwhatterms

people are searching or online. Google Trends provides a useul ree snapshot.
http://www.tweetdeck.com/http://www.tweetdeck.com/


20/50



Market research

I you have gaps in the data or out-take, intermediate outcome and outcome metrics,

then paid-or market research is generally the most eective way o lling them, provided

that budget is available. Market research or evaluation doesnt necessarily have to involve

large-scale ace-to-ace quantitative surveys. Lower-cost research methodologies can be

a useul source o insight. These include:

Omnibus studies: An omnibus study is a quantitative survey o a representative sample

o an audience (usually, the general public). The questionnaire is made up o groups o

questions placed by dierent clients, which means the overall costs are shared, making

it a relatively cheap option. Omnibus surveys may not be appropriate i your activity is

localised, your audience is niche, or i you want to ask many and/or in-depth questions

about a particular topic.

Commissioned online surveys: Standard online surveys recruit respondents rom large

panels o people who have agreed to take part in research. They can be very cost-eective

i your target audience is digitally engaged. However, check the quality o the panel and

how it is managed in advance. Online panels are not always suitable or tracking long-term

activities, as you may not wish to survey the same panel members repeatedly.

Qualitative research: Qualitative research, including discussion groups or interviews,

can be a useul alternative to quantitative research in some circumstances. It may be

appropriate or small-scale activities; when audiences are hard to reach; or when activity isonly running or a short period with no requirement to track its impact over time.

When commissioning paid-or research, you may be able to lower your costs by reducing

the size or specication o your sample, shortening the length o your questionnaire

or simpliying your reporting requirements. Research and evaluation specialists in your

marketing hub or the Shared Communications Service (SCS) can provide advice and help

with commissioning paid-or research or conducting your own online research.

Feedback

Feedback is inormal comment and opinion that you gather yoursel. It can be a valuable

alternative to robust research when there are limitations in time, resource and budget. This

may take the orm o a ew inormal telephone or ace-to-ace interviews with the primary

target audience, those involved in delivering the activity or direct engagement with the

audience; alternatively, eedback can be obtained through an online survey that you run

yoursel, or via a blog, email or text.

I possible, seek input rom colleagues in research or insight roles within your department

or SCS beore gathering eedback yoursel. They will be able to give you advice on

questionnaire design, data protection and propriety issues. They will also be able to advise

you on how to use online survey tools such as SurveyMonkey when seeking eedback.


21/50



Examples o eedback include:

Press Speaking to a ew journalists

Monitoring the type o enquiries that you get when you run a story

Marketing Counting the number o people attending an event

Inormally asking stakeholders or partners or their observations on

how an activity went

Running a survey on your website or social media space

Adding a question about your service to a call-centre script

Internal

communication

Inormal interviews with rontline sta

Feedback rom training sessions

Feedback is not scientic. It will give you anecdotal evidence, rather than statistically

robust measures. Use as many dierent sources as possible and ensure that your analysis

refects the limitations o using such inormal methods (see Stage 4 or more detail).

$ Useful tip data protection

I youre asking members o the public to provide you with personal

inormation, this will need to be gathered and stored in line with government

and industry standards. Check your departments inormation risk policy and

talk to market research specialists in your marketing hub i you need more

advice.

Reviewing any remaining gaps

Ater gathering the data, review your evaluation plan to identiy any perormance metricsthat have no data source. I gaps still exist, consider how important it is to measure that

particular perormance metric or KPI. I it is not central to the evaluation, you may choose

not to measure it at all but point out this limitation in the nal report. I a perormance

metric or KPI is crucial to your evaluation, consider asking or additional resource or budget

to measure it, and make clear the implications o not obtaining it.

Agreeing who will collect data

Beore the activity runs, agree who will gather the data or each source that you have

identied, when and in what ormat. Collecting each piece o data in the same ormat

using consistent time periods and target audiences will help with the analysis later on.


22/50



Completing the evaluation plan

You should now complete your evaluation plan by setting out the data sources that you

will use or each o your perormance metrics and by noting any gaps that still exist. Your

completed plan will now include:

objectivesandtargetaudiences

performancemetricsandKPIs,andthedatasourcesthatyouwillusetomeasureeachone

anylimitationsinyourevaluation

agreedbudgetandresourceneededtogatherthedata(signedoffbythebudget-

holder i needed).

The three Cs principles or good data collection

Whatever youre measuring, make the data that you collect as continuous, consistent

and comparable as possible.

Continuous Most communication activity aims to get people to start, stop or

continue a particular attitude or behaviour. To quantiy its eect,

try to measure peoples attitudes or behaviour beore, during and

ater the activity runs. Benchmarking beore it runs is particularly

important to demonstrate the eect o your activity. For ongoingactivity, measure perormance regularly enough to show the eect

it is having on attitudes or behaviour. The more data points you can

capture, the more obvious the trends will be.

Consistent Use consistent measures and methodology to assess activity that you

repeat or run continuously over time. Using the same wording or

questions, tracking against the same audience and collecting data

in the same way every time will enable you to measure longer-term

trends accurately.

Comparable As ar as you can, try to use the same measures or every piece

o activity that you run. Try to make your measures as similar aspossible to the ones used by other communicators in your team

and across government. This will make it easier to compare results

in the uture. The appendix gives you guidance on the types o

measures that you might want to use or your activity to help with

standardisation.


23/50



Stage 4: Analyse and reportperormance against the plan

Checklist

Task: to assess the success o your activity by:

Analysing eectiveness

Demonstrating eciency and value or money

Output: nal evaluation report

Analysing eectiveness

Once your activity has run, you will need to measure how eectively it met its objectives.

Begin by gathering data and evidence rom all your sources and bringing it together in a

centralised database or older. Check to see whether it looks correct beore beginning the

analysis. Also, check that the activity ran as planned and whether anything unexpected is

likely to have aected its perormance.

$Useful tip activity diaryYou will nd it useul to keep a diary while your activity is running, noting

down any external or operational actors that might aect perormance as

they happen (e.g. bad weather aecting event attendance; negative news

stories aecting public perceptions; or downtime on a website aecting

visitor numbers). This will make subsequent analysis easier.

How to approach analysis

Did your activity work as you expected?

At Stage 2, you mapped out how you expected your activity to work. Use this as the basis

or your analysis. Create some key hypotheses results or outcomes that you might expect

to see based on your objectives, activity map and past perormance.

Check perormance against your KPIs and targets

Have these been met? I so, what is driving success? I not, consider whether your targets

were realistic and whether your KPIs are accurate measures o success. For example, i you

have set KPIs or a new activity, you may need to consider revising these in the uture.


24/50


24 Stage 4: Analyse and report

Be objective

Analyse the ull range o potential outcomes and results or your activity. Do not ignore

results or trends that dont t with your hypotheses or with the general patterns in the

data. Instead, look or the reasons behind these. I something didnt work in the way that

you expected or ailed to meet its objectives, look at the reasons why. This will help you to

amend uture activity and so improve uture perormance.

Try to isolate any operational or external issues

Did any unoreseen operational issues aect your activitys perormance? For example:

Werethereenoughcontactcentrestafftorespondtothecallswhichyouractivitygenerated?

Didthewebsitetowhichyouweredirectingpeoplestopworking?

Didexternalfactors(e.g.negativePR,badweather,theeconomicsituation)havean

impact on your activitys success?

An activity diary can be very useul in helping you to identiy these actors.

Analyse data on a bottom-up basis

I you are analysing the eect o more than one activity or channel, analyse data on a

bottom-up basis against your evaluation plan. First, analyse the eectiveness o each

individual activity against its sub-objective. Then, look at the eect that each activity hadon the wider communication objective, considering which one had the greatest eect.

Assess the eect o communication on the policy objective

Consider the eect that communication as a whole has had on the wider policy objective,

bearing in mind the eect o other interventions designed to meet that objective. You

may not have sucient data to ully isolate the eect o communication on the policy

objective, but try to draw conclusions based on the available evidence.

Review progress

When evaluating activity that will not achieve its communication objectives or some time,

look or evidence o progress towards these based on the KPIs and perormance metrics

that you have identied.

Cross-check your conclusions

Where you have data rom a range o dierent sources, check to see i the results rom

each one are pointing towards the same conclusions. This is particularly important where

you are relying on less robust data sources such as inormal eedback. One set o results

may not always give a conclusive answer, but several pieces o evidence all pointing in the

same direction may allow you to be more condent in your conclusions.


25/50



Signpost gaps and limitations

Your evaluation plan will identiy the gaps in available data and the implications o those

gaps. Be aware o the limitations that this puts on the evaluation. Where possible, try to

draw assumptions about missing results rom the available data. For example, i you dont

know how many people have stopped smoking as a result o your stop-smoking event,

can you make an assumption about possible cessation rates based on the number o

people who tell you that theyre going to quit at the end o the event? I you cant make

such assumptions, signpost missing data within the evaluation report.

Get specialist support where necessary

Some evaluation will require more sophisticated analysis, or example econometricmodelling techniques to predict uture perormance or isolate the eect o dierent

actors on perormance. This type o analysis will generally be carried out by specialists,

either within your department or externally. Only seek external support where you are

unable to carry out the analysis within your marketing hub.

Demonstrating eciency and value or money

When you evaluate perormance, always ask yoursel whether the results that you

achieved justiy the time, resource and money that you spent. As part o your evaluation,

also consider whether the results that the activity achieved justiy the time and money

that were spent on it. Where you are able to quantiy the actual eect that the activity

had on the overall policy objective and put a nancial value on this, you should be able

to calculate the overall return on marketing investment. Otherwise, calculate the cost per

result the amount o time or money that was invested or each person who carried out

a specied action. Compare your results with other activities that you or others have run

to see which ones are most cost-eective and time-ecient and use your learning to

optimise uture activity.

Examples o return on marketing investment and valid cost per result gures include:

Return on marketing investment: Where you are able to demonstrate how many lives

have been saved as a result o a campaign to reduce speeding on residential streets and

calculate the nancial value o each lie saved, the return on marketing investment is thetotal number o lives saved multiplied by the nancial value o each lie saved minus the

cost o running the campaign.

Cost per result: Where you are not able to demonstrate how many lives have been saved

as a result o the campaign, you may instead choose to calculate the cost per result, basing

this on any action that people took ater seeing the activity. This might include the cost

per web visit (total web visits divided by total campaign cost) i the activity sends people

to a website or more inormation, or the cost per event attendee (total event attendees

divided by total campaign cost) i the activity involves running events on road saety.

Further guidance on demonstrating nancial return is available in a separate GCN

publication.3

3 Evaluating the nancial impact o public sector marketing communication: An Introduction to Payback, Return on Marketing Investment (ROMI)and Cost Per Result (https://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/intro-to-payback-romi-and-cpr.pd)


26/50


26 Stage 4: Analyse and report

The evaluation report

You will have agreed the ormat or your report at Stage 1. Check that this is still correct.

Whatever ormat your report is in, always try to ollow these principles:

beobjectiveandincludeallresults,positiveandnegative

alwaysprovideclearconclusionsandrecommendationswhatshouldbedoneasa

result o this report and by whom?

separatefactfromopinionandrecommendationssothatotherscanreviewthe

evidence on which you have based your conclusions

acknowledgegapsinthedataandtheirimplications

stateanyassumptionsthatyouhavemadeandwhatyoubasedtheseon

includereferencesforalldatasourcesusedinyourreport

includeappropriategraphsandimagestoaidinterpretation.

A template or your evaluation report is included here. I you are submitting an evaluation

report to ERG, you must use the ollowing template.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/GCN-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/ERG-Evaluation-report-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/GCN-Evaluation-report-template-.doc


27/50


Conclusions 27

ConclusionsAdopting the principles, processes and practical examples set out in this guide will

enable you to design and implement evaluation plans or the ull range o government

communication activity to the standards required by ERG and the Communication Delivery

Board. By ollowing the recommended approach, you can also be condent that your

work will meet the revised GCN competencies or government communicators.

Adopting good evaluation standards will also enable you both to demonstrate the

contribution that your communication activity makes towards achieving overall policy

objectives and to make a well-inormed business case to invest urther resource andbudget in your work. Sharing your learning with other government communicators in your

team, department or arms-length body and hub, and across GCN more widely, will enable

others to benet rom your knowledge and insight as well.

Finally, remember that it can take time to ully implement the approach set out in this

guide and to gather the data needed to understand what is driving success. Begin

gradually, recognising that it is always better to produce a partial evaluation than nothing

at all, provided that you are clear on the limitations o your end report.

This guide and the examples that it contains refect knowledge and experience drawn

rom a wide range o colleagues rom across the government communication network.

Further comment, eedback and examples are always welcome. Please contact

[email protected].

Helpul contacts and resources

I you want more help or support in applying the content in this guide, talk to your hub

lead or to the evaluation specialists within your hub. The GCN website includes their

contact details.

There are also a number o teams within the Government Communication Centre who

may be able to help:

The Campaigns and Strategy Team works with the communication hubs to dene

good practice evaluation standards across government and to ensure that these are

refected in all annual communication plans and ERG submissions.

TheEvaluation Team in the Shared Communications Service can provide urther

advice on how to apply these evaluation standards to specic projects; they can

oer practical support in planning and conducting your evaluation and advice on the

procurement o external agencies and should be your rst point o contact or general

evaluation queries.

TheGovernment Procurement Service will provide access to specialist external

agencies, i required.

TheGovernment Communication Network runs training courses on evaluation and

the GCN website contains a wealth o inormation and access to groups and individuals

to help you with your evaluation. Go to https://gcn.civilservice.gov.uk/ or more

inormation.
https://gcn.civilservice.gov.uk/https://gcn.civilservice.gov.uk/https://gcn.civilservice.gov.uk/https://gcn.civilservice.gov.uk/


28/50


28 Appendix

Appendix

Recommended metrics

Introduction

This appendix gives examples o the types o perormance metric that you should

consider including in your evaluation plan. They are split by discipline:

1. Press

2. Marketing

3. Internal communication

Within each discipline, perormance metrics are urther split by channel and

objective type. Choose the set that is most relevant to your activity, adapt it as

appropriate and include it in your evaluation plan. I you are using more than one

channel, you will need to pick separate sets or each channel that your activity uses.

I your activity reaches the end audience via an intermediary (e.g. the media or

a partner), use separate sets o metrics to measure:

how eectively your activity engaged the intermediary;

how eectively your intermediary engaged the end audience on your behal.

Templates or activities with and without an intermediary audience are available on

the GCN website.

$

Useful tip adapting perormance metrics to your activity

Ensure that you are clear on your activitys objectives, the channelsthat you are using and the key messages that you are promoting.

This will enable you to adapt the example perormance metrics to

t your activity.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/Evaluation-plan-template-2.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.doc


29/50

Appendix 29


1. Press activity

Press activity includes proactive publicity (activity which proactively promotes

a policy or your organisation), reactive media handling (activity put in place to

respond to a specic issue or event or to rebut inaccurate coverage o this issueor event) and media briefng and handling sessions that you organise to support

ministers and senior policy ocials in their work. All types o press activity have

an intermediary audience (e.g. the media or other spokesperson (a minister or

senior ocial)) and an end audience (generally members o the public).

Choose the perormance metrics that are most relevant or your activity rom the

list below.

$

Useful tip evaluating intermediary and end audiences

The nal outcome or your intermediary audience should always orm

the input or your end audience use this evaluation plantemplate to

help you.

Inputs or intermediary audience

The number and nature o press or media activities that you carry out. This

might include:

Proactivepublicity the number o PR activities, media briengs and packagesissued to the media

Reactivemediahandling the number o corrections, reactive statements andrebuttals issued, the number o interviews arranged

Mediabriefngandhandling the number o media briengs and mediahandling sessions that you organise or ministers and ocials

Any costs incurred in running the activity, time and internal resources used.

Outputs or intermediary audience

Proactivepublicity the number o media contacts that you reach with youractivity, the number o times that you contact them, the messages that you

pass onReactivemediahandling the number o media contacts that you reach with

your corrections, reactive statements and rebuttals, the number o interviews

that take place

Mediabriefngandhandling the number o briengs and training sessionsthat you organise, the inormation and skills that you pass on.
https://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.dochttps://gcn.civilservice.gov.uk/wp-content/uploads/2012/07/Evaluation-plan-template-.doc


30/50


30 Appendix

Out-takes or intermediary audience

Proactivepublicity the attitude o the media overall (and key media contacts

where applicable) towards the message that you are promoting or towardsyour organisation more generally

Reactivemediahandling the attitude o the media towards the issue that youare working on and your handling o it. Have you changed their knowledge

or attitude?

Mediabriefngandhandling what do ministers and senior ocials thinkabout the brieng and training that you provide? Do they nd it useul?

Do they intend to put it into practice?

Intermediate outcomes or intermediary audience

Proactivepublicity the number o media contacts that you reach with youractivity, the number o times that you contact them, the messages that you

pass on

Reactivemediahandling the number o media contacts that you reach withyour corrections, reactive statements and rebuttals, the number o interviews

that take place

Mediabriefngandhandling knowledge and skills that ministers and ocialsgain as a result o your brieng or training sessions.

Final outcomes or intermediary audience/Inputs or end audience (these

perormance metrics are the same)

The volume and quality o media coverage achieved by your activity. This might

include:

Proactivepublicity number o pieces o coverage achieved, accuracy ocoverage, avourability o coverage, key message penetration, quotes and

interviews used

Reactivemediahandling number o pieces o coverage or interviewscontaining your responses, amendments or corrections, overall accuracy o

coverage, avourability o coverage, the amount o negative coverage that has

been prevented as a result o your work

$Useful tip measuring a reduction in negative coverage

It is oten ar harder to measure the negative coverage that has

been avoided as a result o the corrections and rebuttals that

you issue. By keeping a record o the contacts that you have with

the media or each issue that you work on, over time it becomes

easier to demonstrate how your interventions are aecting

coverage in the longer term.

Mediabriefngandhandling the number o times that your contacts use theknowledge and skills that you have passed on in their contact with the media,

the volume o coverage achieved as a result o these contacts, accuracy and

avourability o coverage, key message penetration.


31/50

Appendix 31


Outputs or end audience

The number or percentage o your end audience reached by the media activity

and how oten they saw it.

Out-takes or end audience

Use the perormance metrics included in section 2.1 (purchased marketing

activity) to measure what your end audience recall, think or eel about the

media activity.

Intermediate outcomes or end audience

Depending on the nature o the messages that you promote, you may expect

your end audience to take some orm o action as a result o seeing the media

coverage. This might include:

Seeking inormation rom you

Seeking inormation rom other sources

Registering or a service, product or inormation

Starting, stopping or continuing a particular behaviour.


activity) to measure each o these actions where relevant.

Final outcomes or end audience

Final outcome measures should assess whether your activity met its overall

communication objective, and its eect on the overall policy or reputation

objective that you are working to and the eect that this has had. Choose

perormance metrics that enable you to measure this.


32/50


32 Appendix

2. Marketing activity

Choose the perormance metrics that are most relevant rom the ollowing list,

based on the type o activity that you are evaluating:

2.1 Purchased media

2.2 Websites, social media and other digital spaces

2.3 Public and stakeholder engagement

2.4 Partnership activity(including public advocacy).

2.1 Purchased media (TV, radio, press, outdoor, digital advertising, paid-or

search, direct marketing)

$Useful tip evaluating multi-channel activityI your activity includes more than one paid-or channel, remember to

include perormance metrics to measure the eectiveness o each one.

Inputs

The number o people you plan to reach with your activity and requency o

exposure. This might include:

Estimated reach, coverage and requency (TV, press, radio, outdoor)

The number o impressions served (digital advertising)

The number o planned clicks (paid-or search)

The number o inserts produced, the number o leaets planned to be

distributed (direct marketing)

The costs (media and production) incurred (by channel).

Outputs

The number o people actually reached by your activity and the number o

times they were exposed to your message.

$Useful tip sourcing data

I an agency is booking media or organising distribution o direct

marketing materials, it will be able to provide input and output data

or you.


33/50

Appendix 33


Out-takes

Think about the key messages that you want your activity to get across to your

end audience what will they be thinking or eeling i its been successul? Adaptthe ollowing perormance metrics to enable you to measure their reaction,

choosing those which are most relevant or your activity.

$Useful tip identiying key messages

Use your activity map to help you to identiy what your key

messages are and how you expect your target audience(s) to

react to them.

Have they seen it (RECALL)?

Are the end audience aware o the key/supporting message that you are

promoting?

Have they seen any advertising, communication or publicity about the key/

supporting message?

Can they describe this advertising, communication or publicity? Where have

they seen the advertising, communication or publicity?

Do they recognise your activity when its shown to them (this might include

advertising, leaets, letters, event invitations etc produced by you or press

coverage and partnership activity delivered via an intermediary)?

Do they understand the activitys key messages (THINK)?

Did the end audience like the activity?

Did they think it was relevant to them?

Was it clear and engaging?

Did it capture their attention?

Did they understand the key/supporting message that the activity was

promoting?

Has it impacted on their views? Do they intend to do anything as a result (FEEL)?What would you want the end audience to eel or think i the activity was

successul? On this basis, what eect did the activity have on your end/

intermediary audiences attitudes?

Do the end/intermediary audience eel more positive about your department

and its work as a result o the activity? Are they willing to support you in your

work or to advocate your department more widely?

Do the end/intermediary audience intend to take the action specied in the

key/supporting message in the coming weeks or months (depending on the

time-rame or your activity)?


34/50


34 Appendix


Intermediate outcome metrics measure the actions that people have taken

as a result o your activity. These will vary depending on your communicationactivity, but might include:

a) Seeking inormation rom you

b) Seeking inormation rom other sources

c) Registering or a service, product or inormation

d) Starting, stopping or continuing a particular behaviour

e) Interacting with you or with others online or ofine.

Suggested perormance metrics or each o these intermediate outcomes aregiven below choose the one(s) that is/are most appropriate or your activity

and adapt as appropriate.

a) Seeking inormation rom you

Sometimes, you will ask people to contact you directly or more inormation.

Always try to include a range o metrics that cover who contacted you, how

they contacted you and the depth o contact. Even i you dont directly ask

people to contact you, its always worth checking to see whether youve had

uplit in contact as a result o your activity. Suggested perormance metrics or

the dierent contact channels include:

Call centre:

The number o people calling

What drove them to call (your activity, activity carried out by

intermediaries, e.g. media, stakeholders, other actors)

The reason or their call

How many callers you passed relevant inormation on to

Digital activity (advertising):

The number o people clicking on digital advertising or text links

(split by creative, site)

The number visiting your website or social media space as a result o

advertising

Website or social media space (NB: data or these metrics will be

available rom the Government Digital Service (GDS)):

Source o visits (how people came to the site)

Average length o time spent on the site (and number spending more

than 30 seconds on the site)

The number o pages visited

The number watching any videos or embedded content on your site

(starting to watch and watching whole clip)

The number seeking relevant inormation

Bounce rate (percentage only visiting one page)


35/50

Appendix 35


Intermediate outcomes (continued)

a) Seeking inormation rom you (continued)

Events/ace-to-ace The number attending events as a result o activity you ran to promote them

The number visiting you or your sta in person at other times

What drove them to visit (your activity, other actors)

How many you passed relevant inormation on to

Correspondence

Number o emails received (positive/negative)

Number o letters received (positive/negative).

b) Seeking inormation rom other sources

I your activity is being delivered by stakeholders or partners, ask them whether

they can gather data on how people seek inormation using the perormance

metrics listed under section (a) above.

Other sources that you might want to include are:

People searching or inormation on your activity online

People who claim to have asked riends, amilies or other organisations or

inormation or advice ollowing your activity.

c) Registering or a service, product or inormation

Your activity might ask people to register or a service, more inormation or a

product, or to attend an appointment with you or a stakeholder. I so, you might

want to include measures that look at:

The number o people seeking inormation on the service, product or

inormation eatured in your activity (see previous section)

The number o people requesting a product eatured in your activity (and

where they heard about this product)

The number who actually receive this product

The number who go on to use this product

The number o people signing up to receive inormation in the uture

The number o people starting the registration process or a service or

product (and where they heard about it)

The number o people completing the registration process

The people using the service that they have registered or and how they use it


36/50


36 Appendix


c) Registering or a service, product or inormation (continued)

$Useful tip choosing the right perormance metrics oryour activity

The perormance metrics that you choose here will vary

depending on the nature o your activity. For example, i youre

running an ongoing programme to support people as they give

up smoking, you might record how many people attend an initial

meeting or read the inormation that you have sent to them

and then record how many attend subsequent meetings,

request urther inormation etc. Creating an activity map can

be a helpul way o identiying how you would expect people

to use the service, and can help you to choose the rightperormance metrics.

The number o people making an appointment with you or a stakeholder

The number o people attending this appointment.

d) Starting, stopping or continuing a particular behaviour

Your activity may ask people to start, stop or continue a particular behaviour.

I so, you might want to include perormance metrics that look at:

The number o people undertaking that behaviour beore the activity ran

The number o people who take the relevant steps to change their behaviouras a result o your activity

The number o people who continue with the changed behaviour over time.

e) Interacting with you or with others online or ofine

You may want people to pass your message on or to discuss it with others.

I youre using social media, you may want them to join an online community

or contribute to content on your website.

You may nd the ollowing metrics useul, depending on what media youre using:

The number o people who pass your message on or discuss your activity orthe message that it is promoting with others

The number o people who claim to have discussed your message with

others (online and ofine)

The number passing on or sharing your content digitally (e.g. tweets,

retweets, sharing link on Facebook, via Digg, Delicious, StumbleUpon,

Reddit etc)

Amount o talk on digital orums, blogs, websites etc (tracked via buzz

monitoring)

Amount o this talk that is positive (and accurate)


37/50

Appendix 37



e) Interacting with you or with others online or ofine (continued)

The number o people who interact with you online depending on thedigital space that you use, this might include:

The number o Facebook ans or riends, the number liking your content

The number o Facebook ans or riends who are active (i.e. return to the

site regularly)

The number o people posting content on your site

The number o people commenting on your content (e.g. on your own

site, Facebook, YouTube).

$

Useful tip passing the message on

I your activity asks someone to pass a message on on yourbehal, also include metrics that enable you to measure the eect

o this message on those to whom they speak.

Final outcomes

Final outcome measures should assess whether your activity met its overall

communication objective, and its eect on the overall policy objective that

you are working to and the eect that this has had. Choose perormance

metrics that enable you to measure the activitys impact on both.

2.2 Websites, social media and other digital spacesGDS will produce standard reporting dashboards or gov.uk. These include

standard perormance metrics that measure outputs, out-takes and intermediate

outcomes. Bespoke reports or gov.uk and other government websites and social

media spaces will also be available. Further inormation on how to get this data will

be available soon via the GCN website. Always ensure that you integrate these

perormance metrics with wider measures that look at the eect o your website

or social media activity. These might include:

Inputs

The website, social media or other digital space and related content created

to meet communication objectives

The cost and resource used to create this website, social media space or

content.

Outputs (available via GDS reports)

GDS can provide perormance metrics that you can use to look at how many

people were exposed to your digital content. These will include:

The number o the end audience visiting the site or digital space at least once

during the evaluation period (unique users)

The source o visitor reerrals.


38/50


38 Appendix

Out-takes (available via GDS reports)

What people think, eel or recall about the webspace. Section 2.1 gives more

detail on the types o perormance metric that you might want to include.You may also be able to iner attitudes and understanding rom comments

that people leave on your website or social media space.

Intermediate outcomes (available via GDS reports)

GDS can provide a number o standard metrics that measure how people

interact with your site or digital space. These will include:

Average length o time spent on site

Bounce rate

The number watching any videos or embedded content on your site (starting

to watch and watching whole clip)

The number o pages visited.

Also consider including metrics that look at actions carried out on social media

websites, such as:

Number o likes on Facebook

Number o retweets on Twitter.

Always ensure that you tie these standard metrics back to your website or

digital spaces objectives and consider what you are trying to get people to

do (register, read a particular piece o content, retweet content etc). This will

enable you to interpret the data appropriately.

Final outcomes

Final outcome measures should assess whether your website or digital space

met its overall communication objective, and its eect on the overall policy

objective that you are working to and the eect that this has had. Choose

perormance metrics that enable you to measure the activitys impact on both.

2.3 Public and stakeholder engagement

Public or stakeholder engagement activity is put in place to secure eedback on orsupport or a specic policy area. Depending on the activity that you are running,

you may use a range o channels including events, digital and purchased channels.

Inputs

The activity that you carry out to promote your initiative. Depending on the

nature o your engagement activity, this might include:

The number o invitations to events that you send out

The number o requests to provide eedback that you issue

Activity that you carry out to publicise your engagement activity

The website that you create to gather eedback

The number and nature o contacts that you make with specic

stakeholders.


39/50

Appendix 39


Outputs

Depending on the nature o your engagement activity, these might include:

The number o people that you reach via your publicity, invitations andrequests

The number o people attending the event that you organised

The number o people who visit your website or digital space (see

section 2.2 or more detail on digital metrics)

The number o conversations that you have with a specic stakeholder as

a result o the contacts that you make.

Out-takes

Think about what you want the audience that you are engaging to recall, think

or eel as a result o your engagement activity. Oten there will be a specicpolicy that you want them to advocate or support. Include relevant out-take

measures to assess levels o support or your message and your department.


activity) to help to measure the depth o engagement and attitudes towards

your policy area.


Intermediate outcome metrics measure the actions people have taken as

a result o your activity. These will vary depending on your communication

activity, but typical intermediate outcomes or stakeholder and public

engagement include:

a) Providing usable eedback

b) Attending an event

c) Passing your message on to others.

Suggested perormance metrics or each o these intermediate outcomes are

given below choose the one(s) that is/are most appropriate or your activity

and adapt as appropriate.

a) Providing usable eedback

You may be running activity to increase engagement among specic audiences

possibly to seek help in developing policy or to ask or eedback or consult on a

particular policy area. I so, you might include measures that look at:

The number o people who respond to your engagement activity in any way.

This might include:

Comments or eedback received on your website, in writing, by phone,

ace-to-ace

The quality o eedback and comments received

The number o people attending public meetings or hearings

The number o people agreeing to help to develop a specic policy area

(generally in relation to stakeholder engagement)

The amount o usable eedback incorporated into uture policy development

or engagement activity.


40/50


40 Appendix


b) Attending an event

You may invite stakeholders or members o the public to attend events so thatyou can share inormation or increase engagement. Depending on what your

event is designed to do, you might want to think about including perormance

metrics that look at:

The number o event attendees who interact with you or your sta during

the event

The number o event attendees who undertake specic actions or

behaviours in line with messages promoted at the event

The number who thought the event was good and useul

The number who understand the messages that you were promoting.

c) Passing your message on to others

You may want people to pass on specic messages rom your engagement

activity. You may nd the ollowing metrics useul, depending on what media

youre using:

The number o stakeholders or members o the public who pass your message

on and discuss your activity or the message that it is promoting with others

The number o stakeholders or members o the public who claim to have

discussed your message with others (online and ofine):

The number passing on or sharing your content digitally (e.g. tweets,

retweets, sharing link on Facebook, via Digg, Delicious, StumbleUpon,

Reddit etc)

Amount o digital talk on orums, blogs, websites etc (tracked via buzz

monitoring)

Amount o positive (and accurate) digital talk.

$Useful tip passing the message on

I your activity asks someone to pass a message on on your

behal, also include metrics that enable you to measure the eect

o this message on those to whom they speak.


41/50

Appendix 41


2.4 Partnership activity (including public advocacy)

In partnership activity a partner is used as an intermediary to advocate your

work, pass on a message or deliver a service to an end audience on your behal

typically in the areas o policy delivery and reputation management. Partnersmay include commercial, not-or-prot and government bodies as well as

individual stakeholders.

$Useful tip evaluating partnership activity

All partnership activity will have an intermediary audience the partner

that you are engaging. Always think about who your end audience is and

what you are asking the partner to do to reach them on your behal.

Inputs or intermediary audience

Details o your overall partnership engagement strategy and the supportingactivity that you carry out to engage stakeholders or partners so that they will

pass on your message on your behal. This might include:

The number o partners that you contact by phone, email or ace-to-ace

The number o partners that you invite to events, training etc and the

number o events and training sessions that are held

The number o times that you contact each partner (particularly when

building relationships and hence advocacy over time)

The number and nature o key messages that you pass on to each partner(particularly when building relationships/advocacy)

Any costs incurred in contacting partners, time and internal resources used.

Outputs or intermediary audience

The number o partners who are exposed to the activities that you run and the

messages that you promote. This might include:

The number o partners successully reached by your activity (the number

with whom you are able to share your messages)

The number o partners attending events or training that you have organised

The quality and range o messages shared with each partner over time

(particularly when building advocacy).

Out-takes or intermediary audiences

Think about what you want partners to think or eel as a result o the contact

that you make. This might include:

The number o partners who support your activity

The number o partners who are interested in nding out more about your

initiative or in working with you.


42/50


42 Appendix

Intermediate outcomes or intermediary audiences

Intermediate outcomes will depend on the nature o contact that you make with

partners and the response that you are seeking rom them. They might include:

The number o partners contacting you or more inormation

The number o partners discussing the activity with you

The number o partners expressing an interest in working with you

The number o partners agreeing to undertake a specic activity or pass on a

specic message or you.

Final outcomes or intermediary/Inputs or end audience

The number and/or nature o messages passed on or activities carried out to

promote messages on your behal by partners. This might include:

The number o partners advocating the work o your department or team:

The number o partners passing on specic messages about your

department and its work (through conversations with others, interviews,

media coverage, digital interaction etc)

The number o partners oering more general support or your department

The number o messages passed on by each individual partner, content o

each message, avourability o each message

The number passing on or sharing your content digitally (e.g. tweets, retweets,

sharing link on Facebook, via Digg, Delicious, StumbleUpon, Reddit etc)

The number o partners carrying out activities to prom

GCN Evaluation Guide

Documents

Transcript of GCN Evaluation Guide