1

Case Study

Introduction

“When someone comes to use our services, they want to know it will be effective.”

(Quote from a project research participant)

Welcome to our Case Study module, one of eight online modules created as part of the

Developing a Culture of Evaluation project. This project was conducted through a

partnership between Community Literacy of Ontario and Literacy Link South Central.

This module, like all of the modules developed for this project, aims to provide practical,

useful information, with a focus on small non-profits with limited budgets and limited staff

resources. The modules have been written to meet the needs of the non-profit sector as

identified during online research, key informant interviews and a provincial survey

conducted at the outset of the project.

These are the eight online modules available from our project:

We hope that our online modules are helpful to you in the important work that you do.

1 Introduction to Evaluation

2 Planning for Evaluation

3 Collecting Data

4 Analyzing Data

5 Taking Action

6 Communicating the Results

7 Case Study: The Evaluation Cycle in Action

8 Trends in Evaluation

2

Case Study

Module Overview

In our online training modules, Community Literacy of Ontario has shared information,

resources and tools to lead non-profits through the evaluation cycle and the process of

encouraging a culture of evaluation.

In this Case Study module, a fictitious non-profit organization, called “Main Street

Community Services” will take you through all of the components of the evaluation cycle:

We created this Case Study module to showcase the evaluation cycle in action, and to

show you how the tools, templates and resources could be used to help your non-profit

organization conduct an evaluation.

For our case study, we chose a small non-profit that wanted to conduct an evaluation of

one part of their organization. We did this because small organizations are overwhelmingly

the most common type of non-profit in Ontario. Your organization may be larger, or you

may be conducting an evaluation of a different topic, or of your entire organization;

however, the same simple steps overviewed in our case study would apply in different

scenarios.

3

Case Study

The following topics are covered in this module:

Introduction to Our Case Study

Planning for Evaluation: Case Study

Collecting Data: Case Study

Analyzing Data: Case Study

Taking Action: Case Study

Communicating the Results: Case Study

Introduction to Our Case Study

Main Street Community Services (MSCS) is a small, Ontario-based non-profit, which serves

a medium sized city of 80,000 people. MSCS runs a food bank and a small clothing depot.

It has a full-time, paid staff of two (an Executive Director and an Office Coordinator), and

30 dedicated volunteers (including 10 board members). MSCS is a registered charity and it

has existed for 20 years.

Main Street Community Services has an annual budget of $150,000. It receives 50% of its

funding from the Provincial Government, 30% from the Federal Government, and the

remaining 20% of its operating funds from donations from businesses and individuals. The

food and clothing are donated by community partners – businesses, corporations and

individuals.

MSCS has a website, a Facebook page, a blog and a monthly electronic newsletter (or e-

newsletter) featuring agency news and events. They also have a paper-based brochure,

and they put up posters in the community about their services. In addition, MSCS has a

bulletin board at the office with all the latest news about their organization and

community.

4

Case Study

So why does Main Street Community Services want to conduct an evaluation?

While MSCS has an excellent reputation in its community, the Board of Directors and staff

have noticed a few worrisome communication trends over the past year:

Fewer people are attending their annual fundraising event.

Articles and press releases submitted to the local media are less often used while other community organizations seem to effortlessly appear in the local press.

MSCS set up a Facebook page, but it has been a disappointment. After one year, they have few followers and almost no comments, likes or other interaction with followers of their Facebook page.

When MSCS speaks to potential volunteers and donors, they often hear, “So what do you do exactly? I’ve never heard of your organization before.”

MSCS has amazing stories to tell about the important work they are doing, but no one seems to want to listen!

Accordingly, the Board of Directors and staff of Main Street Community Services decided

that it was time to evaluate their communication tools, in order to understand what works,

what doesn’t work, and what improvements need to be made.

They decided to follow the evaluation cycle and the templates, tools and resources in the

online modules from our “Developing a Culture of Evaluation” project. This is their

evaluation story.

5

Case Study

Planning for Evaluation: Case Study

The first phase of the cycle is “Planning for Evaluation”,

which makes perfect sense. After all, if you don’t plan where

you are going, what will be done and who will be involved,

well it’s like setting off on a camping trip with no map and no

camping equipment! Disaster is sure to follow.

Are You Ready?

MSCS first carefully considered the “Are You Ready” issues raised in our “Planning for

Evaluation” module. They especially considered why they might conduct this evaluation,

and whether it was manageable for them. These questions were critical ones, especially in

light of their limited staff and volunteer capacity to undertake additional work. They

decided that because understanding and improving their communication methods was a

critical issue for Main Street Community Services, and because they were undertaking a

small-scale evaluation, rather than a complete organizational review, this evaluation was

achievable for them.

An additional positive factor was that staff, volunteers, and board members were highly

motivated to conduct this evaluation. Almost all internal stakeholders understood the

importance and also wanted to get involved in evaluation by starting with a small,

manageable evaluation task. Internal stakeholders also thought it was very important to

undertake this evaluation in order to begin the process of building a culture of evaluation

at Main Street Community Services.

Main Street Community Services also believed in the importance of hearing from external

stakeholders regarding their communication tools. The Board and staff believed that they

were not engaging with external stakeholders effectively any more. It is a noisy world, with

many non-profits and causes competing for community attention. MSCS had the sense

that, while their mission was critical, their voice was becoming lost.

In short, the staff, volunteers and Board of Directors of Main Street Community Services

believed that there was enough urgency, motivation, skills and commitment amongst the

staff and volunteer team to conduct an evaluation of their communication tools and

methods.

6

Case Study

Assessing Risk

As recommended in our “Planning for

Evaluation” module, the Board and staff of

MSCS asked themselves the following question:

“Are there any risks associated with conducting

this evaluation?”

Main Street Community Services then used our

risk assessment framework to brainstorm and

record potential risks:

Evaluation Risk Assessment

Identified Risk Likelihood Impact Mitigation Strategies

Evaluation feedback is mainly received from stakeholders who are currently engaged with MSCS, and we don’t hear from those who are not already connected with MSCS.

High High The Board, staff and volunteer evaluation team must ensure that they all make strong efforts to promote the evaluation and collect data from a broad array of stakeholders, outside of the usual communication channels and methods.

A high degree of negative input is received regarding our communication tools.

Low High Be aware of problem areas and the potential impact on staff and volunteers upon receiving this input. Be prepared to figure out how to incorporate this feedback into the evaluation process.

7

Case Study

Evaluation Risk Assessment

Identified Risk Likelihood Impact Mitigation Strategies

Great communication strategies are identified in the evaluation but we do not have the resources to address them.

High Medium During all phases of the evaluation, remind stakeholders of organizational capacity limitations. Create a “great ideas” file, so ideas that cannot be immediately implemented are not lost.

Be open to stopping some current activities in order to start taking more effective, new initiatives.

Finding out that our stakeholders think we’ve been doing our communication all wrong.

Low High There is little that can be done to mitigate this risk. This would require a major re-orientation of our communication strategy. However, this would be critical information for us to know, in order to be an effective organization.

A communication tool that is highly valued by our staff is not valued by our stakeholders.

Medium Medium Prepare our staff ahead of time that changes to any and all communication tools may occur. Remind all stakeholders that change is normal and healthy.

Momentum and motivation is lost amongst stakeholders if there is a lack of follow-through on evaluation results.

Low High Leaders in the organization and in the evaluation process (the Executive Director, the Board of Directors and key volunteers) must commit to the evaluation process, including taking action.

8

Case Study

What Will You Evaluate?

Main Street Community Services had a specific goal in mind: to evaluate their

communication tools. As such, they did a small, focused evaluation, as opposed to a full-

scale, comprehensive, organizational evaluation. MSCS understood that, even though they

were conducting a focused evaluation on one topic, communication is still a broad and

many-facetted issue.

Accordingly, to get a better sense of WHAT they needed to evaluate in terms of their

communication, MSCS informally talked to clients, volunteers, donors, and funders about

their communication tools. Then, at a monthly meeting of the Board of Directors, the

Board and staff took the information they had gathered, plus their own knowledge of

communication issues, and conducted a SWOT analysis using the template from our

“Planning for Evaluation” module. They assessed the internal factors (strengths and

weakness) and the external factors (opportunities and threats) facing their non-profit

organization and recorded them in the following chart.

SWOT Analysis

Strengths

Website

Stories of hope and change

Newsletter

Strong volunteer and donor base

Organizational reputation

Weaknesses

Limited staff capacity for communication

Not effective at social media

Not strategic in our communication

Aren’t sure what is working and what’s not working well

Opportunities

Strengthening our presence on social media to engage stakeholders

Potential of partnering with other non-profits to jointly promote local community services

Learning from others who are more effective at communication

Threats

Government funding is shrinking and it seems like the larger non-profits who are most likely to receive grants

It is becoming ever more difficult to engage the community at large in our fundraising events

Competition for media coverage is fierce

This SWOT analysis helped Main Street Community Services to focus its evaluation goals on

communication tools and strategies that were the most important.

9

Case Study

Developing Evaluation Goals

Main Street Community Services knew it was

important to have a clear goal (or goals) in mind to

help them stay focused. Therefore, the next step

taken by Main Street Community Services was to

take the information from the initial planning, their

readiness and risk assessment, and the SWOT

analysis to develop a goal (or goals) for their

evaluation (as recommended in our “Planning for

Evaluation” module).

The organization also learned from our module that setting a SMART goal for their

evaluation was more likely to result in a successful process. (Remember, SMART goals are

Specific, Measureable, Achievable, Realistic and Time-Bound.)

Accordingly, the following evaluation goal was developed by Main Street Community

Services:

In the next eight months, Main Street Community Services will conduct an

evaluation of our communication tools in order to improve the way we

communicate with stakeholders

What Type of Evaluation?

Main Street Community Services looked at the three major types of

evaluation featured in our “Planning for Evaluation” module:

1. Goals-based

2. Process-based

3. Outcomes-based

Because their evaluation goal was most closely related to understanding organizational

processes related to communication, Main Street Community Services decided that the

type of evaluation they were conducting would be process-based.

10

Case Study

Who Will Evaluate?

Deciding who would evaluate was a key step in the process for Main Street Community

Services. Following the key points set out in our module, MSCS considered the following

questions:

Would they follow a team approach of

having a combination of staff, board and

volunteers?

Would they assign responsibilities to one

staff member?

Would they seek to hire an external

evaluator?

In deciding who should be responsible for the evaluation, several important factors came

into play. First was cost. MSCS did not have extra funds to pay for an external evaluator.

So, while they valued the independent viewpoint that would be brought by an external

evaluator, and would have greatly liked to spare their staff and volunteers the

responsibility and time commitment of conducting the evaluation, they decided that an

external evaluator was not feasible for them.

A second factor was capacity. Their two staff members were already working at maximum

capacity (and often beyond), so adding on substantial additional work was not viable. A

third consideration was skill sets. Although no one person had all the required skills, as a

group, the staff, volunteers and board members of Main Street Community Services

believed they had both the skills and capacity to conduct their evaluation.

Lastly, MSCS wanted to use this evaluation process to help build a culture of evaluation in

their non-profit organization, and involving multiple stakeholders was an excellent way to

accomplish this.

11

Case Study

Developing an Evaluation Plan of Action

The final planning step taken by Main Street Community Services was to develop an

evaluation plan of action. They used the handy template developed in our “Planning for

Evaluation” module to think through the necessary action items, who would be

responsible, the timelines and the results.

Evaluation Plan of Action

Action Item Who? When? Cost Result

Identify stakeholders to involve in the evaluation and inform them about the evaluation process

Staff with help from Board and volunteers

Ongoing Staff time Stakeholders are informed about the evaluation goals and processes

Brainstorm and decide:

What methods will be used to collect data

Who will be responsible for collecting the data

Timelines

Board of Directors and staff

September board meeting

No additional costs; part of regular meeting

Data collection plan

Collect the data Assigned staff, volunteers and board members

October and November

Staff and volunteer time

Budget of $250 for paper surveys, snacks focus groups

Report on the results of the data collection phase

12

Case Study

Action Item Who? When? Cost Result

Analyze the data and make recommendations

Executive Director

December Staff time Report on evaluation results and draft recommendations

Review the results and recommendations

Board of Directors and staff

January board meeting

No additional costs; part of regular meeting

Evaluation results are collated into a report on the key findings

Based on the results & recommendations, decide:

What actions will be taken

Who will do what

Costs (if any)

Timelines

Special meeting of the Board of Directors, several volunteers and staff

February Staff and volunteer time

Budget of $50 for snacks and beverages

Report on approved recommendations, actions to be taken, responsibilities, costs and timelines

Implement the plan of action

Assigned staff, volunteers and board members

February to April

Will be determined in the action plan

Appropriate actions are taken to improve the communication tools and strategies

Communicate the results

Staff April to June

Staff time Stakeholders are informed about the evaluation results

13

Case Study

Cost of Staff Time

The Board and staff realized that extra staff time would be needed to conduct the

evaluation. They estimated that the Executive Director would need 32 extra hours of staff

time and the Office Manager would need an extra 24 hours. To compensate staff for the

additional time required for the evaluation, the Board decided that, during the summer,

they would give the ED three days off in lieu of pay while the Office Manager would

receive two days off in lieu of pay. Two volunteers stepped forward to help the staff with

the equivalent of a days’ worth of work each, thereby lessening the cost to the

organization.

Furthermore, both the Board and the staff hoped that conducting an evaluation of their

communication tools would help MSCS to be more strategic in their communication. They

believed that the evaluation might not only tell them what they should be doing more of,

but also what perhaps they should stop doing! Potentially, the evaluation process could

lead to savings in staff time.

Conclusion

And that brings us to the end of how Main Street Community Services effectively planned

for evaluation. With a solid plan in place, they were ready to take on the work of

evaluating their communications.

14

Case Study

Collecting Data: Case Study

The second phase of the evaluation cycle is “Collecting Data”. Deciding the type of data

you should be collecting and the most effective ways to gather that data are critical

elements of conducting evaluation in your non-profit organization. Let’s see how Main

Street Community Services worked through this important step.

As a small, non-profit organization, Main Street

Community Services knew that they needed to

stay focused. They could not possibly collect

every bit of data regarding their communication

tools, but rather, MSCS wanted to focus its

efforts on collecting data that would help them

make evidence-based decisions about improving

their communication tools and strategies. To

keep focused, Main Street Community Services

followed the recommended steps in our

“Collecting Data” module and asked themselves

the following key questions.

What do we want to know?

Main Street Community Services had already developed a clear evaluation goal. Having a

clear goal made it easy for them to clearly understand what they needed to know in the

data collection phase. Simply put, MSCS wanted to know how they could improve the way

they communicate with stakeholders.

What data do we need?

In order to understand and assess their communication tools, methods and strategies,

MSCS needed data on the following four key questions:

1. What communication tools are working well?

2. What tools are working somewhat effectively but need improvement?

3. What communication tools are no longer effective at all?

4. Are there new ways of communicating that should be implemented?

15

Case Study

From whom do we need to gather input?

Because communication has a broad-based reach that impacts virtually all stakeholders,

MSCS knew that they needed to gather data from a broad array of people, including:

Clients

Staff

Board members

Volunteers

Partner organizations

Community organizations

Funders and donors

What tools will we use to colle ct the data we need?

Main Street Community Services considered the chart in our “Collecting Data” module,

which overviewed the pros and cons of various data collection methods. They thought

about their organizational capacity and resources, the skills and interests of their staff and

volunteers, the need to gain input from a wide variety of stakeholders, and the importance

of communication to their organization.

MSCS had never evaluated their communication

tools and methods before, so they did not have

any pre-existing data to use. They did have an

informal sense of the strengths and weaknesses of

their communications, and they knew that this

informal information could be captured in focus

groups and surveys.

16

Case Study

Based on these factors, MSCS selected the following data collection tools, which were

geared to their different stakeholder groups:

Data Collection Method Target Audience

Online survey Any stakeholder

Print survey Clients

Two face-to-face focus groups One for clients

One for Board members, staff and volunteers

15 key informant interviews Funders, partners, community organizations and donors

Open questions posted on Facebook Any stakeholder

Facebook analytics Any stakeholder

In order to collect quality data, they

reviewed the best practices, found in our

“Collecting Data” module, to learn how to

ask the right questions (including field

testing the tools to ensure clarity and reduce

bias).

This helped them to effectively gather data

via focus groups, surveys, key informant

interviews and other sources.

17

Case Study

What resources are available to collect the data?

As a small, non-profit organization, Main Street Community Services had limited resources

available to support this evaluation. As noted earlier, they were unable to allocate funds

for an external evaluator, nor provide additional funding for staff (although provisions for

time off in lieu of pay were made for staff). However, the Board of Directors, volunteers

and staff all agreed that they would work as a team to undertake all aspects of the

evaluation, including data collection.

A small budget of $250 was made available for data collection to cover the costs of printing

the paper surveys, as well as for providing snacks and beverages for the two focus groups.

Other

Who will collect data?

Main Street Community Services elected to use a team approach of staff, volunteers and

board members to collect the data. This type of approach ensured buy-in from all levels of

MSCS, and helped to build a culture of evaluation in their organization.

Data collection responsibilities were allocated as follows:

Staff:

Created, distributed and promoted the online survey

Created, distributed and promoted the print survey

Reviewed the social media analytics

Volunteers:

Hosted a focus group with Board members,

staff and volunteers

Posted questions on Facebook

Board of Directors:

Conducted 15 key informant interviews

Hosted a focus group with clients

18

Case Study

Finally, Main Street Community Services used the template found in our “Collecting Data”

module to compile their plan for data collection.

Evaluation Plan: Data Collection

Who? What? When?

Executive Director Create the online and print surveys

October 1st

Office Coordinator Distribute and promote the online and print surveys

October 1st – 31st

Two volunteers Host a focus group with Board members, staff and volunteers

October 20th

Chair and Vice-Chair of the Board of Directors

Host a focus group with clients November 1st

Five Board members each conducted three interviews

Conduct 15 key informant interviews

October 15th – November 10th

Executive Director Review social media analytics November 1st to 15th

Two social media volunteers

Post several open questions on Facebook

October 15th to November 30th

Conclusion

By keeping a clear focus, using a team approach and following our structured process,

Main Street Community Services collected the data they needed to address their

evaluation goal of improving the way they communicate with stakeholders. Now they were

ready to move on to the next step: making sense of the data they had collected by

analyzing it carefully.

19

Case Study

Analyzing Data: Case Study

The third phase of the evaluation cycle is “Analyzing Data”. Organizing, collating and

analyzing the data are important steps. After all, you need to make sense of the data in

order to share the results and implications, as well as to make decisions and take action

based on your evaluation findings.

As we noted in our “Analyzing Data” module, “Successful evaluation isn’t just about

gathering the right data, it’s also about how you use that data to effect positive change –

large or small – in your organization.”

Main Street Community Services knew that they were evaluating a specific component of

their non-profit organization, as opposed to doing a comprehensive evaluation of all of

their services. As such, they believed it was extremely important for them to stay focused

in order to keep all evaluation tasks manageable.

Compile and Organize the Data

Their first step was to sort the data into manageable pieces. Staff and volunteers from Main

Street Community Services prepared a brief Evaluation Results Report which summarized the

results of each type of data collected. They collected data in these seven ways:

1. Online survey

2. Print survey

3. Focus group with Board members, staff and volunteers

4. Focus group with clients

5. 15 key informant interviews

6. Gathering input on Facebook

7. Facebook analytics

Main Street Community Services recorded the two types of data they received:

Formal, quantitative data (percentages, ranking, ratios, etc.)

o This data came in the form of closed survey questions and Facebook analytics.

Informal, qualitative data (conversations, comments, ideas, stories, etc.)

o This data came from the focus groups, open-ended survey questions, key

informant interviews and responses to the questions posted on Facebook.

20

Case Study

Once the data was compiled into a Data Summary Report, the next step was to organize

the data in a way that would be useful to their evaluation goals. To help them more easily

notice trends and issues, Main Street Community Services decided to organize the data by

going back to the four key questions they asked in the evaluation:

1. What communication tools are working well?

2. What tools are working somewhat effectively but need improvement?

3. What communication tools are no longer effective at all?

4. Are there new ways of communicating that should be implemented?

For each of these four questions, MSCS grouped and recorded relevant results from all

applicable data collection methods into a written summary which they included in their

Data Summary Report.

Analyze the Data

With the evaluation data organized and compiled into their Data Summary Report, the

data was ready to be analyzed. The Board of Directors and staff of Main Street Community

Services reviewed the data compiled according to collection method, and brainstormed

the key themes and trends emerging from the data. They summarized their findings in the

following chart:

Analyzing the Data – By Data Collection Method

Data Collection Method Key Findings

Online survey Stakeholders found the look of the website to be outdated

Stakeholders liked the Facebook page, but found that the page was not active enough

Most stakeholders have never heard of the MSCS blog, and those who had visited it thought it was very outdated.

Stakeholders valued the e-newsletter

21

Case Study

Analyzing the Data – By Data Collection Method

Data Collection Method Key Findings

Print survey Clients found the website hard to navigate

Clients liked the bulletin board at the office

Clients liked the Facebook page, but were intimidated to participate

Focus group with clients Clients found the website to be static and boring

Clients were not aware of the e-newsletter or MSCS blog

Clients liked to have the brochures to share with their friends and family

Clients wished special organizational events were set up on Facebook, so that they could easily invite their friends

They wanted to see more stories of people like them

Focus group with Board members, staff and volunteers

The Board, staff and volunteers thought the website needed some minor tune-ups

They greatly valued the e-newsletter and agency bulletin board

Half valued the Facebook page, while the other half thought it was a waste of time

They thought that the brochure had value, but that it was outdated

Staff did not have the time to keep the blog up to date

Open questions on Facebook

People wanted more content, more stories, more links to events, and a more dynamic organizational presence on Facebook

22

Case Study

Analyzing the Data – By Data Collection Method

Data Collection Method

Key Findings

15 key informant interviews

Stakeholders found the website to be overly complicated

Stakeholders didn’t find the brochure helpful, as they wanted to access organizational information online

Stakeholders liked having the posters to put up in their offices

Stakeholders wanted to know more about MSCS and about client success stories, but found that this type of information was not readily available

They didn’t have a clear idea of MSCS’ achievements

No one had ever visited the MSCS blog

External stakeholders thought the MSCS’s Facebook page had potential and that they should be on Twitter as well

Open questions on Facebook

People wanted more content, more stories, more links to events, and a more dynamic organizational presence on Facebook

Facebook analytics Upon reviewing Facebook Insights for the three months, it was noted that the most popular posts (i.e., the ones that were liked, shared and commented upon the most) were posts about client success stories and achievements

The majority of Facebook posts were seen by at least 75% of the page’s followers, although less than 5% of followers interacted with the posts (liked, shared or commented)

One post was “boosted” (meaning that they paid Facebook a small fee to promote the post) during the month; this post reached 10x the usual number of Facebook users and was shared 25 times

Following the post’s boosting, 10 new page likes were received

By looking at the compiled data according to data collection method and putting the key

findings into this chart, some strong messages about organizational strengths and

weaknesses in the area of communication quickly began to emerge.

23

Case Study

Next, the Board of Directors and staff of Main Street Community Services looked again at

the Data Summary Report, but this time they analyzed the data by looking at the four key

evaluation questions.

Their findings were summarized in the following chart:

Analyzing the Data – By Evaluation Question

Evaluation Questions Key Findings

What communication tools are working well?

Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.

Internal stakeholders greatly valued the bulletion board.

Their posters were working well as a communication tool.

What tools are working somewhat effectively but need improvement?

Stakeholders valued their website as a communication tool, but found the MSCS website to be outdated and not user-friendly.

Stakeholders valued the Facebook page, but thought that it should be used more dynamically. Followers visit the page, although they aren’t actively interacting with it. Analytics are beginning to show more clearly the type of posts that generate interaction, so staff can use this knowledge for future posts. The only boosted post was very successful, so boosting should be considered for select posts.

Brochures received mixed reviews depending on the stakeholder group questioned.

What communication tools are no longer effective at all?

The blog was not valued by stakeholders, and the staff did not have time to keep it up to date.

Are there new ways of communicating that should be implemented?

External stakeholders often mentioned that MCSC should set up a Twitter account.

All stakeholders wanted to see videos implemented. This reflects the worldwide trend of the popularity of video, and statistics show greater interaction with communication materials when video is included.

24

Case Study

Conclusion

By compiling and organizing the data in a simple yet

effective way, the Board of Directors and staff were

able to efficiently conduct an analysis of the data. This

analysis revealed some strong strengths, weaknesses,

opportunities and trends in the communication tools

and strategies used by Main Street Community Services.

With a clear picture in hand of what stakeholders were

telling them about communication in their organization,

Main Street Community Services was ready to move on

to the next step (and perhaps the most important and

neglected phase of all) – using the data to take action to

improve services.

Taking Action: Case Study

The fourth phase of our evaluation cycle is “Taking Action”. It can sometimes be difficult to

summon the energy and time to implement the results of your evaluation. However,

implementing your evaluation results in order to create a stronger and more effective non-

profit organization lies at the very foundation of why you engaged in evaluation in the first

place.

So, take heart, and let’s follow along with Main Street Community Services to see how they

used our “Taking Action” module to implement their evaluation results in order to meet

their goal of improving the way they communicate with stakeholders.

Main Street Community Services followed the “Five Steps to Taking Action” described in

our module:

1. Assess What the Data is Telling You

2. Set Your Priorities

3. Develop Your Recommended Actions

4. Create an Action Plan

5. Implement and Monitor Your Action Plan

25

Case Study

Step 1: Assess What the Data is Telling You

The Board of Directors, staff and several volunteers

of Main Street Community Services met together and

carefully reviewed the results of their data analysis.

Then they asked the all-important question: “What is

the Data Telling You?”

They used the “What is the Data Telling You?”

template from our module to record the results.

“What is the Data Telling You?”

Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.

Internal stakeholders greatly valued the bulletion board.

Their posters were working well as a communication tool.

Stakeholders valued their website as a communication tool. However, their website is outdated and not user-friendly.

Stakeholders valued the MCSC Facebook page. However, their Facebook page is not actively used, and MSCS should develop new strategies to create a more engaging and dynamic page. Facebook analytics has provided information about the type of posts that are most successful.

Brochures received mixed reviews depending on the stakeholder group.

Their blog was not valued by stakeholders, and the staff did not have time to keep it up to date.

External stakeholders often mentioned that MCSC should set up a Twitter account.

All stakeholders wanted to see videos implemented as a communication tool.

26

Case Study

Step 2: Set Your Priorities

Main Street Community Services’ evaluation data certainly revealed many communication

issues! However, as a small, non-profit organization with a limited budget and capacity,

MSCS was not able to implement all of the communications needs identified. Instead, they

had to set priorities based on factors such as the importance of the issue to MSCS, their

relation to the organization’s mission and their ability to implement the change.

Main Street Community Services used our “Prioritizing Issues Template” to help them set

priorities. For the purposes of this case study, we will use just one of the issues identified

by MSCS.

Prioritizing Issues Template

Our Facebook page should be used more actively and dynamically

Does this issue have a strong link to your organization’s values and mission?

Yes Somewhat No

Notes: Yes. Our stakeholders value our Facebook page. It is an important and cost-effective way for us to communicate.

Does this issue provide a clear benefit to your clients or other stakeholders?

Yes Somewhat No

Notes: If we can learn to be more dynamic on Facebook, we will be able to communicate more effectively with our stakeholders.

Would your key stakeholders be supportive if changes were made to this issue?

Yes Somewhat No

Notes: Yes. Stakeholders are driving the request to change.

Do your staff members / volunteers / other stakeholders have the ability (time and skill level) to implement changes to this issue?

Yes Somewhat No

Notes: One staff member and several volunteers have intermediate skills in this area. One volunteer in particular is eager to learn more and take on this challenge.

27

Case Study

Do you have the resources (financial and capacity) to address the needs identified in this issue?

Yes Somewhat No

Notes: We have limited financial resources, but Facebook is free. It is staff and volunteer time that will be impacted. However, since we plan to stop hosting our blog, some capacity should be freed up.

Do you have the ability to influence this issue or is it outside of your ability to control?

Yes Somewhat No

Notes: We have complete control over our Facebook page and are limited only by our financial and HR resources.

Is this an issue we can and should be addressing now?

Yes Somewhat No

Notes: Yes. This issue is manageable and important for us.

Based on the above, on a scale of 1 to 10 (with 1 being low, and 10 being high), what priority level would you assign to this issue? Priority Level: 9

28

Case Study

Using this same “Prioritizing Issues Template”, Main Street Community Services assessed

all of the communications issues identified in the evaluation. The following priorities for

action were developed, and designated as high, medium or low priority, or assigned to a

“no change needed” category.

High Priority

Stakeholders valued their website as a communication tool. However, their website is outdated and not user-friendly.

Stakeholders valued the MCSC Facebook page. However, their Facebook page should be used more actively and dynamically.

The blog was not valued by stakeholders, and the staff did not have time to keep it up to date.

All stakeholders wanted to see the use of videos implemented.

Medium Priority

External stakeholders often mentioned that MCSC should set up a Twitter account.

Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.

Low Priority

Brochures received mixed reviews depending on the stakeholder group.

No Change Needed at this Time

Internal stakeholders greatly valued the bulletin board.

Their posters were working well as a communication tool.

29

Case Study

Step 3: Develop Your Recommended Actions

Main Street Community Services developed four clear

priorities for action in order to meet its goal of

improving communication with stakeholders:

1. Stakeholders valued their website as a

communication tool. However, their website is

outdated and not user-friendly.

2. Stakeholders valued the MCSC Facebook page.

However, their Facebook page should be used

more actively and dynamically.

3. The blog was not valued by stakeholders, and

the staff did not have time to keep it up to date.

4. All stakeholders wanted to see videos

implemented as a communication tool.

Next they turned their priorities into clear, strong, actionable recommendations, as

follows:

Our Recommended Actions

1. Main Street Community Services will conduct an in-depth update of its website.

2. MSCS will use its Facebook page more actively and dynamically.

3. MSCS will cease hosting its blog and delete it immediately.

4. MSCS will assess costs, staff and volunteer skills sets, as well as research potential grant or foundation opportunities in order to begin the process of implementing videos as a communication tool.

30

Case Study

Step 4: Create an Action Plan

Main Street Community Services had four clear, actionable recommendations to help the

organization meet its goal of improving communications with stakeholders. The next step

was to develop a plan of action for implementing each of the recommendations. They kept

in mind the tried and true statement that “A goal without a plan in just a wish.”

Main Street Community Services used our “Action Plan Template” from the “Taking

Action” module to set timelines, responsibilities, available resources and implementation

strategies for each of the four recommendations.

In our case study, we will showcase their strategy for implementing one of their

recommendations: “MSCS will use its Facebook page more actively and dynamically”.

31

Case Study

Action Plan Template

WHAT? MSCS will use its Facebook page more actively and dynamically.

WHY?

Stakeholders valued the MCSC Facebook page. However, their Facebook page is not actively used, and MSCS should develop new strategies to create a more engaging and dynamic page.

HOW? WHO? HOW MUCH? WHEN?

Two volunteers will visit the Facebook pages of other non-profits with similar missions to search for best practices and new ideas.

Volunteers Free Next six weeks

The Executive Director will commit to posting two Facebook posts per week.

Executive Director

Two hours off per month in lieu of pay

Immediate and ongoing

Two volunteers will commit to creating one post each per week on Facebook.

Volunteer Free Immediate and ongoing

The Executive Director or Board Chair will find a well-known person from the community to do a monthly guest post.

Guest Speaker

Free Monthly and ongoing

The Board, volunteers and clients will actively post comments or questions, and to like and share MSCS Facebook postings.

Board, volunteers and clients

Free Ongoing

MSCS will pay to boost Important Facebook posts.

Executive Director

$50 annually As needed

The Office Coordinator will set up all future events held by MSCS as Facebook events.

Office Coordinator

Incorporate into regular staff time

Ongoing

32

Case Study

Step 5: Implement and Monitor Your Action Plan

Main Street Community Services now had an Action Plan that clearly set out its priorities,

tasks, timelines, resources and responsibilities. They were ready to take the final steps of

implementing and monitoring their plan of action. Work days in a non-profit are busy, and

it’s easy to fall behind on good intentions, even with the best of plans! Regular check-ins

can save the day and keep everyone motivated and on track.

MSCS reviewed the strategies for implementing and monitoring an action plan in our

“Taking Action” module, and used our “Monitoring Your Action Plan Template” to help

keep everyone on track and moving forward.

In our case study, we will showcase their strategy for monitoring their action plan by

tracking the implementation of one of their action items: “Two volunteers will visit the

Facebook pages of other non-profits with similar missions to search for best practices and

innovative ideas.”

33

Case Study

Monitoring Your Action Plan Template

ACTION: Two volunteers will visit the Facebook pages of other non-profits with similar missions to search for best practices and innovative ideas.

WHO?

Our plan:

Two volunteers will conduct the research.

Actual:

One volunteer resigned due to health issues.

One volunteer did some of the research but needed help from the Office Coordinator.

Notes / Follow-up / Changes:

This was too much work for one volunteer. The Office Coordinator agreed to help out.

HOW MUCH?

Our plan:

Free (volunteer time only)

Actual:

6 hours of staff time

Notes / Follow-up / Changes:

More staff time was needed. Since the Office Coordinator was no longer maintaining the blog, she had these 6 extra hours to help out.

WHEN?

Our plan:

Six week timeline

Actual:

With one less volunteer, an extra week was needed to complete this task.

Notes / Follow-up / Changes:

A week delay did not affect this action. This action was successfully completed!

34

Case Study

Conclusion

By following our five step process to taking action on evaluation results, Main Street

Community Services was able to assess what the data was telling them, set their priorities,

develop their recommended actions, create an action plan, and identify a process for

monitoring implementation.

Using our “Taking Action” module, MSCS kept the process straightforward and clear. They

successfully developed a plan of action that would help them meet their evaluation goal of

improving communication with stakeholders.

With a successful process and plan in hand, Main Street Community Services arrived at the

final step of the evaluation cycle: communicating the results!

35

Case Study

Communicating the Results: Case Study

The fifth and final phase of our evaluation cycle is “Communicating the Results”. By this

point, Main Street Community Services had planned the evaluation, collected and analyzed

data and created an action plan for implementation. Those are very impressive tasks! Now,

there is just one more important phase left to be carried out in the evaluation cycle:

communicating the results.

Main Street Community Services knew that organizational communication is of critical

importance. In fact, that is why they chose to evaluate their communication strategies and

tools in the first place! MSCS knew that they should communicate their evaluation results

for a variety of good reasons, including profiling organizational success and positive

outcomes, showing organizational transparency, increasing credibility and promoting a

culture of evaluation. However, as a small, non-profit organization with few resources,

Main Street Community Services also knew they needed to be strategic in how they

communicated the results.

MSCS followed the step-by-step process outlined in our “Communicating the Results”

module to figure out the “what, why, who and how” of communicating their evaluation

results. Let’s follow along and see how they did it.

What Evaluation Results Should You Communicate and Why?

Main Street Community Services knew that

their stakeholders are busy people. So,

although they had prepared a summary report

on their evaluation results, they believed that

this report was more for internal use and that

stakeholders would only be interested in the

key findings.

To decide what to share, the Board and staff

of MSCS asked themselves two key questions:

1. What are the key findings of our evaluation?

2. Of these findings, what needs to be communicated to our stakeholders and why?

36

Case Study

Their responses to these questions are summarized in this template from our module:

What evaluation results should you communicate and why?

What are the key findings? Does this evaluation result need to be communicated?

Why are we sharing (or not sharing) this result?

Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.

IN PART. This finding will be used internally for staff to develop strategies to increase client awareness of the e-newsletter.

MSCS wanted to better engage clients by encouraging them to subscribe to the e-newsletter.

Internal stakeholders greatly valued the bulletin board.

No. MSCS is not changing their strategy for this item.

No change in strategy. Internal impact only.

Stakeholders valued their website as a communication tool. However, it is outdated and not user-friendly.

YES. The highlights of this finding should be communicated broadly.

Their website is a criticial communication tool for all stakeholders.

Stakeholders valued the MCSC Facebook page. However, their page is not actively used, and MSCS should create a more engaging page.

YES. The highlights of this finding should be communicated broadly.

Facebook is an important communication tool for many stakeholders.

Their blog was not valued by stakeholders, and the staff did not have time to keep it up to date.

YES. Since an activity is completely ceasing, it is important to let stakeholders know.

It is important to share this news for organizational transparency.

All stakeholders wanted to see videos implemented as a communication tool.

YES. Stakeholders need to know that MSCS heard their input on this issue.

This is a new strategic area for MSCS. Active communication is key.

37

Case Study

Who Needs to Know our Evaluation Results?

As with all non-profit organizations, Main Street Community Services had a wide array of

stakeholders. They knew it was important to keep stakeholders informed during the evaluation

in order to increase awareness of the evaluation process and results, gain support for needed

changes, show respect for stakeholder input and build a culture of evaluation at MSCS.

However, they also knew that some of the evaluation findings would be of more relevance and

interest to certain stakeholder groups than others. Accordingly, the Board and staff asked

themselves, “Which of our stakeholder groups need to know about this evaluation result?”

Main Street Community Services then used the template from our module to answer these

questions.

Who Needs to Know Our Evaluation Results?

What were the key findings? Who needs to know?

Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.

Staff

Clients

Internal stakeholders greatly valued the bulletin board.

Internal stakeholders

Stakeholders valued their website as a communication tool. However, it is outdated and not user-friendly.

All stakeholders

Stakeholders valued the MCSC Facebook page. However, their page is not actively used, and MSCS should develop new strategies to create a more engaging page.

All stakeholders

Their blog was not valued by stakeholders, and the staff did not have time to keep it up to date.

All stakeholders

All stakeholders wanted to see videos implemented as a communication tool.

All stakeholders

38

Case Study

How Should You Communicate Your Evaluation Results?

Main Street Community Services discovered the “what, why and who” of communicating their

evaluation results. Now, they needed to figure out how to communicate strategically with the

major stakeholder groups.

There are so many ways to communicate, including formal and informal communication, and via

traditional and new media. Main Street Community Services considered what needed to be

communicated, who needed to know and then assessed the best way to communicate each

evaluation result in a way that would be meaningful to their stakeholders.

This chart records what they decided:

How Should We Communicate Our Evaluation Results?

Action Item Who needs to know?

Communication methods

Monthly e-newsletter

Staff

Clients

Personal conversations with clients

E-mails to clients

Update client information packages

Bulletin board

Internal stakeholders

Post a notice on the bulletin board

Website All stakeholders

Share via all communication channels

Also, prepare a major re-launch strategy when the website is revamped

Facebook page

All stakeholders

Share via all communication channels

When new strategies are ready, promote a post on Facebook announcing the changes

Add links to Facebook on emails, website

Consider holding a virtual “launch event”

Blog All stakeholders

Share via e-newsletter, website, Facebook, email, and via a farewell Blog posting

Videos All stakeholders

Share via all communication channels

39

Case Study

Effective Communication Strategies

As you can imagine, the Board, staff and volunteers of Main Street Community Services

were both excited and proud of figuring out the “what, why, who and how” of

communicating their evaluation results! In fact, they were so proud of their achievements

that they wanted to “up their communications game” by trying out a few new

communication methods. Accordingly, they reviewed the “Communicating Using New

Media” part of our module.

Based on what they learned, MSCS implemented some new communication strategies:

They used pictures to share their evaluation results.

Main Street Community Services created their own graphic images using Canva to

share results (see image below).

They shared quotes from the evaluation.

They even tried to make an infograph using Piktochart. This was, at first,

unsuccessful, but at least they got started and expanded their knowledge.

All we can say is, “job well done” to the amazing team at Main Street Community

Services!

40

Case Study

Conclusion

We hope that our Case Study module has helped you to clearly see the evaluation cycle in

action, as we walked alongside Main Street Community Services while they conducted

these five critical steps:

Planning for Evaluation

Collecting Data

Analyzing Data

Taking Action

Communicating the Results

It is our goal that these online training modules will help you to build a culture of

evaluation in your non-profit organization. Furthermore, we hope that our modules have

given you the inspiration to know that evaluation in non-profit organizations can be

manageable, and can lead to lasting improvements for our clients and communities.

Be sure to continue the evaluation conversation. Here are some ways to do that:

During the life of the “Developing a Culture of Evaluation” project (it ends in March

2017), visit our website to join our online discussion group, online clinics and/or

webinars so that we can continue to learn together.

Stay connected with the Ontario Nonprofit Network, Imagine Canada and other

organizations with expertise in non-profit evaluation.

Participate in conversations about evaluation. Talk to your fellow non-profit

organizations, as well as your volunteers, Board of Directors, funders, donors and

community members.

Share your thoughts with us via our Facebook page or on Twitter.

“True genius resides in the capacity for evaluation of uncertain,

hazardous, and conflicting information.”

(Sir Winston Churchill)

41

Case Study

Acknowledgements

The Case Study module is one of eight online modules created as part of the Developing a

Culture of Evaluation project. All project resources can be freely accessed online at:

www.communityliteracyofontario.ca/evaluation-culture.

This project was conducted through a partnership between Community Literacy of Ontario

and Literacy Link South Central.

The Case Study module was written by Joanne Kaattari.

This project was funded through the Partnership Project of the Ontario Ministry of

Citizenship, Immigration and International Trade. We are very grateful for their support.

The views expressed in this module are the views of Community Literacy of Ontario and

Literacy Link South Central and do not necessarily reflect those of the Province.

Date of publication: March 2016.

All information and weblinks provided in this module were accurate at the time of

publication.

©Copyright Community Literacy of Ontario and Literacy Link South Central.