Post on 21-Mar-2018
1
Case Study
Introduction
“When someone comes to use our services, they want to know it will be effective.”
(Quote from a project research participant)
Welcome to our Case Study module, one of eight online modules created as part of the
Developing a Culture of Evaluation project. This project was conducted through a
partnership between Community Literacy of Ontario and Literacy Link South Central.
This module, like all of the modules developed for this project, aims to provide practical,
useful information, with a focus on small non-profits with limited budgets and limited staff
resources. The modules have been written to meet the needs of the non-profit sector as
identified during online research, key informant interviews and a provincial survey
conducted at the outset of the project.
These are the eight online modules available from our project:
We hope that our online modules are helpful to you in the important work that you do.
1 Introduction to Evaluation
2 Planning for Evaluation
3 Collecting Data
4 Analyzing Data
5 Taking Action
6 Communicating the Results
7 Case Study: The Evaluation Cycle in Action
8 Trends in Evaluation
2
Case Study
Module Overview
In our online training modules, Community Literacy of Ontario has shared information,
resources and tools to lead non-profits through the evaluation cycle and the process of
encouraging a culture of evaluation.
In this Case Study module, a fictitious non-profit organization, called “Main Street
Community Services” will take you through all of the components of the evaluation cycle:
We created this Case Study module to showcase the evaluation cycle in action, and to
show you how the tools, templates and resources could be used to help your non-profit
organization conduct an evaluation.
For our case study, we chose a small non-profit that wanted to conduct an evaluation of
one part of their organization. We did this because small organizations are overwhelmingly
the most common type of non-profit in Ontario. Your organization may be larger, or you
may be conducting an evaluation of a different topic, or of your entire organization;
however, the same simple steps overviewed in our case study would apply in different
scenarios.
3
Case Study
The following topics are covered in this module:
Introduction to Our Case Study
Planning for Evaluation: Case Study
Collecting Data: Case Study
Analyzing Data: Case Study
Taking Action: Case Study
Communicating the Results: Case Study
Introduction to Our Case Study
Main Street Community Services (MSCS) is a small, Ontario-based non-profit, which serves
a medium sized city of 80,000 people. MSCS runs a food bank and a small clothing depot.
It has a full-time, paid staff of two (an Executive Director and an Office Coordinator), and
30 dedicated volunteers (including 10 board members). MSCS is a registered charity and it
has existed for 20 years.
Main Street Community Services has an annual budget of $150,000. It receives 50% of its
funding from the Provincial Government, 30% from the Federal Government, and the
remaining 20% of its operating funds from donations from businesses and individuals. The
food and clothing are donated by community partners – businesses, corporations and
individuals.
MSCS has a website, a Facebook page, a blog and a monthly electronic newsletter (or e-
newsletter) featuring agency news and events. They also have a paper-based brochure,
and they put up posters in the community about their services. In addition, MSCS has a
bulletin board at the office with all the latest news about their organization and
community.
4
Case Study
So why does Main Street Community Services want to conduct an evaluation?
While MSCS has an excellent reputation in its community, the Board of Directors and staff
have noticed a few worrisome communication trends over the past year:
Fewer people are attending their annual fundraising event.
Articles and press releases submitted to the local media are less often used while other community organizations seem to effortlessly appear in the local press.
MSCS set up a Facebook page, but it has been a disappointment. After one year, they have few followers and almost no comments, likes or other interaction with followers of their Facebook page.
When MSCS speaks to potential volunteers and donors, they often hear, “So what do you do exactly? I’ve never heard of your organization before.”
MSCS has amazing stories to tell about the important work they are doing, but no one seems to want to listen!
Accordingly, the Board of Directors and staff of Main Street Community Services decided
that it was time to evaluate their communication tools, in order to understand what works,
what doesn’t work, and what improvements need to be made.
They decided to follow the evaluation cycle and the templates, tools and resources in the
online modules from our “Developing a Culture of Evaluation” project. This is their
evaluation story.
5
Case Study
Planning for Evaluation: Case Study
The first phase of the cycle is “Planning for Evaluation”,
which makes perfect sense. After all, if you don’t plan where
you are going, what will be done and who will be involved,
well it’s like setting off on a camping trip with no map and no
camping equipment! Disaster is sure to follow.
Are You Ready?
MSCS first carefully considered the “Are You Ready” issues raised in our “Planning for
Evaluation” module. They especially considered why they might conduct this evaluation,
and whether it was manageable for them. These questions were critical ones, especially in
light of their limited staff and volunteer capacity to undertake additional work. They
decided that because understanding and improving their communication methods was a
critical issue for Main Street Community Services, and because they were undertaking a
small-scale evaluation, rather than a complete organizational review, this evaluation was
achievable for them.
An additional positive factor was that staff, volunteers, and board members were highly
motivated to conduct this evaluation. Almost all internal stakeholders understood the
importance and also wanted to get involved in evaluation by starting with a small,
manageable evaluation task. Internal stakeholders also thought it was very important to
undertake this evaluation in order to begin the process of building a culture of evaluation
at Main Street Community Services.
Main Street Community Services also believed in the importance of hearing from external
stakeholders regarding their communication tools. The Board and staff believed that they
were not engaging with external stakeholders effectively any more. It is a noisy world, with
many non-profits and causes competing for community attention. MSCS had the sense
that, while their mission was critical, their voice was becoming lost.
In short, the staff, volunteers and Board of Directors of Main Street Community Services
believed that there was enough urgency, motivation, skills and commitment amongst the
staff and volunteer team to conduct an evaluation of their communication tools and
methods.
6
Case Study
Assessing Risk
As recommended in our “Planning for
Evaluation” module, the Board and staff of
MSCS asked themselves the following question:
“Are there any risks associated with conducting
this evaluation?”
Main Street Community Services then used our
risk assessment framework to brainstorm and
record potential risks:
Evaluation Risk Assessment
Identified Risk Likelihood Impact Mitigation Strategies
Evaluation feedback is mainly received from stakeholders who are currently engaged with MSCS, and we don’t hear from those who are not already connected with MSCS.
High High The Board, staff and volunteer evaluation team must ensure that they all make strong efforts to promote the evaluation and collect data from a broad array of stakeholders, outside of the usual communication channels and methods.
A high degree of negative input is received regarding our communication tools.
Low High Be aware of problem areas and the potential impact on staff and volunteers upon receiving this input. Be prepared to figure out how to incorporate this feedback into the evaluation process.
7
Case Study
Evaluation Risk Assessment
Identified Risk Likelihood Impact Mitigation Strategies
Great communication strategies are identified in the evaluation but we do not have the resources to address them.
High Medium During all phases of the evaluation, remind stakeholders of organizational capacity limitations. Create a “great ideas” file, so ideas that cannot be immediately implemented are not lost.
Be open to stopping some current activities in order to start taking more effective, new initiatives.
Finding out that our stakeholders think we’ve been doing our communication all wrong.
Low High There is little that can be done to mitigate this risk. This would require a major re-orientation of our communication strategy. However, this would be critical information for us to know, in order to be an effective organization.
A communication tool that is highly valued by our staff is not valued by our stakeholders.
Medium Medium Prepare our staff ahead of time that changes to any and all communication tools may occur. Remind all stakeholders that change is normal and healthy.
Momentum and motivation is lost amongst stakeholders if there is a lack of follow-through on evaluation results.
Low High Leaders in the organization and in the evaluation process (the Executive Director, the Board of Directors and key volunteers) must commit to the evaluation process, including taking action.
8
Case Study
What Will You Evaluate?
Main Street Community Services had a specific goal in mind: to evaluate their
communication tools. As such, they did a small, focused evaluation, as opposed to a full-
scale, comprehensive, organizational evaluation. MSCS understood that, even though they
were conducting a focused evaluation on one topic, communication is still a broad and
many-facetted issue.
Accordingly, to get a better sense of WHAT they needed to evaluate in terms of their
communication, MSCS informally talked to clients, volunteers, donors, and funders about
their communication tools. Then, at a monthly meeting of the Board of Directors, the
Board and staff took the information they had gathered, plus their own knowledge of
communication issues, and conducted a SWOT analysis using the template from our
“Planning for Evaluation” module. They assessed the internal factors (strengths and
weakness) and the external factors (opportunities and threats) facing their non-profit
organization and recorded them in the following chart.
SWOT Analysis
Strengths
Website
Stories of hope and change
Newsletter
Strong volunteer and donor base
Organizational reputation
Weaknesses
Limited staff capacity for communication
Not effective at social media
Not strategic in our communication
Aren’t sure what is working and what’s not working well
Opportunities
Strengthening our presence on social media to engage stakeholders
Potential of partnering with other non-profits to jointly promote local community services
Learning from others who are more effective at communication
Threats
Government funding is shrinking and it seems like the larger non-profits who are most likely to receive grants
It is becoming ever more difficult to engage the community at large in our fundraising events
Competition for media coverage is fierce
This SWOT analysis helped Main Street Community Services to focus its evaluation goals on
communication tools and strategies that were the most important.
9
Case Study
Developing Evaluation Goals
Main Street Community Services knew it was
important to have a clear goal (or goals) in mind to
help them stay focused. Therefore, the next step
taken by Main Street Community Services was to
take the information from the initial planning, their
readiness and risk assessment, and the SWOT
analysis to develop a goal (or goals) for their
evaluation (as recommended in our “Planning for
Evaluation” module).
The organization also learned from our module that setting a SMART goal for their
evaluation was more likely to result in a successful process. (Remember, SMART goals are
Specific, Measureable, Achievable, Realistic and Time-Bound.)
Accordingly, the following evaluation goal was developed by Main Street Community
Services:
In the next eight months, Main Street Community Services will conduct an
evaluation of our communication tools in order to improve the way we
communicate with stakeholders
What Type of Evaluation?
Main Street Community Services looked at the three major types of
evaluation featured in our “Planning for Evaluation” module:
1. Goals-based
2. Process-based
3. Outcomes-based
Because their evaluation goal was most closely related to understanding organizational
processes related to communication, Main Street Community Services decided that the
type of evaluation they were conducting would be process-based.
10
Case Study
Who Will Evaluate?
Deciding who would evaluate was a key step in the process for Main Street Community
Services. Following the key points set out in our module, MSCS considered the following
questions:
Would they follow a team approach of
having a combination of staff, board and
volunteers?
Would they assign responsibilities to one
staff member?
Would they seek to hire an external
evaluator?
In deciding who should be responsible for the evaluation, several important factors came
into play. First was cost. MSCS did not have extra funds to pay for an external evaluator.
So, while they valued the independent viewpoint that would be brought by an external
evaluator, and would have greatly liked to spare their staff and volunteers the
responsibility and time commitment of conducting the evaluation, they decided that an
external evaluator was not feasible for them.
A second factor was capacity. Their two staff members were already working at maximum
capacity (and often beyond), so adding on substantial additional work was not viable. A
third consideration was skill sets. Although no one person had all the required skills, as a
group, the staff, volunteers and board members of Main Street Community Services
believed they had both the skills and capacity to conduct their evaluation.
Lastly, MSCS wanted to use this evaluation process to help build a culture of evaluation in
their non-profit organization, and involving multiple stakeholders was an excellent way to
accomplish this.
11
Case Study
Developing an Evaluation Plan of Action
The final planning step taken by Main Street Community Services was to develop an
evaluation plan of action. They used the handy template developed in our “Planning for
Evaluation” module to think through the necessary action items, who would be
responsible, the timelines and the results.
Evaluation Plan of Action
Action Item Who? When? Cost Result
Identify stakeholders to involve in the evaluation and inform them about the evaluation process
Staff with help from Board and volunteers
Ongoing Staff time Stakeholders are informed about the evaluation goals and processes
Brainstorm and decide:
What methods will be used to collect data
Who will be responsible for collecting the data
Timelines
Board of Directors and staff
September board meeting
No additional costs; part of regular meeting
Data collection plan
Collect the data Assigned staff, volunteers and board members
October and November
Staff and volunteer time
Budget of $250 for paper surveys, snacks focus groups
Report on the results of the data collection phase
12
Case Study
Action Item Who? When? Cost Result
Analyze the data and make recommendations
Executive Director
December Staff time Report on evaluation results and draft recommendations
Review the results and recommendations
Board of Directors and staff
January board meeting
No additional costs; part of regular meeting
Evaluation results are collated into a report on the key findings
Based on the results & recommendations, decide:
What actions will be taken
Who will do what
Costs (if any)
Timelines
Special meeting of the Board of Directors, several volunteers and staff
February Staff and volunteer time
Budget of $50 for snacks and beverages
Report on approved recommendations, actions to be taken, responsibilities, costs and timelines
Implement the plan of action
Assigned staff, volunteers and board members
February to April
Will be determined in the action plan
Appropriate actions are taken to improve the communication tools and strategies
Communicate the results
Staff April to June
Staff time Stakeholders are informed about the evaluation results
13
Case Study
Cost of Staff Time
The Board and staff realized that extra staff time would be needed to conduct the
evaluation. They estimated that the Executive Director would need 32 extra hours of staff
time and the Office Manager would need an extra 24 hours. To compensate staff for the
additional time required for the evaluation, the Board decided that, during the summer,
they would give the ED three days off in lieu of pay while the Office Manager would
receive two days off in lieu of pay. Two volunteers stepped forward to help the staff with
the equivalent of a days’ worth of work each, thereby lessening the cost to the
organization.
Furthermore, both the Board and the staff hoped that conducting an evaluation of their
communication tools would help MSCS to be more strategic in their communication. They
believed that the evaluation might not only tell them what they should be doing more of,
but also what perhaps they should stop doing! Potentially, the evaluation process could
lead to savings in staff time.
Conclusion
And that brings us to the end of how Main Street Community Services effectively planned
for evaluation. With a solid plan in place, they were ready to take on the work of
evaluating their communications.
14
Case Study
Collecting Data: Case Study
The second phase of the evaluation cycle is “Collecting Data”. Deciding the type of data
you should be collecting and the most effective ways to gather that data are critical
elements of conducting evaluation in your non-profit organization. Let’s see how Main
Street Community Services worked through this important step.
As a small, non-profit organization, Main Street
Community Services knew that they needed to
stay focused. They could not possibly collect
every bit of data regarding their communication
tools, but rather, MSCS wanted to focus its
efforts on collecting data that would help them
make evidence-based decisions about improving
their communication tools and strategies. To
keep focused, Main Street Community Services
followed the recommended steps in our
“Collecting Data” module and asked themselves
the following key questions.
What do we want to know?
Main Street Community Services had already developed a clear evaluation goal. Having a
clear goal made it easy for them to clearly understand what they needed to know in the
data collection phase. Simply put, MSCS wanted to know how they could improve the way
they communicate with stakeholders.
What data do we need?
In order to understand and assess their communication tools, methods and strategies,
MSCS needed data on the following four key questions:
1. What communication tools are working well?
2. What tools are working somewhat effectively but need improvement?
3. What communication tools are no longer effective at all?
4. Are there new ways of communicating that should be implemented?
15
Case Study
From whom do we need to gather input?
Because communication has a broad-based reach that impacts virtually all stakeholders,
MSCS knew that they needed to gather data from a broad array of people, including:
Clients
Staff
Board members
Volunteers
Partner organizations
Community organizations
Funders and donors
What tools will we use to colle ct the data we need?
Main Street Community Services considered the chart in our “Collecting Data” module,
which overviewed the pros and cons of various data collection methods. They thought
about their organizational capacity and resources, the skills and interests of their staff and
volunteers, the need to gain input from a wide variety of stakeholders, and the importance
of communication to their organization.
MSCS had never evaluated their communication
tools and methods before, so they did not have
any pre-existing data to use. They did have an
informal sense of the strengths and weaknesses of
their communications, and they knew that this
informal information could be captured in focus
groups and surveys.
16
Case Study
Based on these factors, MSCS selected the following data collection tools, which were
geared to their different stakeholder groups:
Data Collection Method Target Audience
Online survey Any stakeholder
Print survey Clients
Two face-to-face focus groups One for clients
One for Board members, staff and volunteers
15 key informant interviews Funders, partners, community organizations and donors
Open questions posted on Facebook Any stakeholder
Facebook analytics Any stakeholder
In order to collect quality data, they
reviewed the best practices, found in our
“Collecting Data” module, to learn how to
ask the right questions (including field
testing the tools to ensure clarity and reduce
bias).
This helped them to effectively gather data
via focus groups, surveys, key informant
interviews and other sources.
17
Case Study
What resources are available to collect the data?
As a small, non-profit organization, Main Street Community Services had limited resources
available to support this evaluation. As noted earlier, they were unable to allocate funds
for an external evaluator, nor provide additional funding for staff (although provisions for
time off in lieu of pay were made for staff). However, the Board of Directors, volunteers
and staff all agreed that they would work as a team to undertake all aspects of the
evaluation, including data collection.
A small budget of $250 was made available for data collection to cover the costs of printing
the paper surveys, as well as for providing snacks and beverages for the two focus groups.
Other
Who will collect data?
Main Street Community Services elected to use a team approach of staff, volunteers and
board members to collect the data. This type of approach ensured buy-in from all levels of
MSCS, and helped to build a culture of evaluation in their organization.
Data collection responsibilities were allocated as follows:
Staff:
Created, distributed and promoted the online survey
Created, distributed and promoted the print survey
Reviewed the social media analytics
Volunteers:
Hosted a focus group with Board members,
staff and volunteers
Posted questions on Facebook
Board of Directors:
Conducted 15 key informant interviews
Hosted a focus group with clients
18
Case Study
Finally, Main Street Community Services used the template found in our “Collecting Data”
module to compile their plan for data collection.
Evaluation Plan: Data Collection
Who? What? When?
Executive Director Create the online and print surveys
October 1st
Office Coordinator Distribute and promote the online and print surveys
October 1st – 31st
Two volunteers Host a focus group with Board members, staff and volunteers
October 20th
Chair and Vice-Chair of the Board of Directors
Host a focus group with clients November 1st
Five Board members each conducted three interviews
Conduct 15 key informant interviews
October 15th – November 10th
Executive Director Review social media analytics November 1st to 15th
Two social media volunteers
Post several open questions on Facebook
October 15th to November 30th
Conclusion
By keeping a clear focus, using a team approach and following our structured process,
Main Street Community Services collected the data they needed to address their
evaluation goal of improving the way they communicate with stakeholders. Now they were
ready to move on to the next step: making sense of the data they had collected by
analyzing it carefully.
19
Case Study
Analyzing Data: Case Study
The third phase of the evaluation cycle is “Analyzing Data”. Organizing, collating and
analyzing the data are important steps. After all, you need to make sense of the data in
order to share the results and implications, as well as to make decisions and take action
based on your evaluation findings.
As we noted in our “Analyzing Data” module, “Successful evaluation isn’t just about
gathering the right data, it’s also about how you use that data to effect positive change –
large or small – in your organization.”
Main Street Community Services knew that they were evaluating a specific component of
their non-profit organization, as opposed to doing a comprehensive evaluation of all of
their services. As such, they believed it was extremely important for them to stay focused
in order to keep all evaluation tasks manageable.
Compile and Organize the Data
Their first step was to sort the data into manageable pieces. Staff and volunteers from Main
Street Community Services prepared a brief Evaluation Results Report which summarized the
results of each type of data collected. They collected data in these seven ways:
1. Online survey
2. Print survey
3. Focus group with Board members, staff and volunteers
4. Focus group with clients
5. 15 key informant interviews
6. Gathering input on Facebook
7. Facebook analytics
Main Street Community Services recorded the two types of data they received:
Formal, quantitative data (percentages, ranking, ratios, etc.)
o This data came in the form of closed survey questions and Facebook analytics.
Informal, qualitative data (conversations, comments, ideas, stories, etc.)
o This data came from the focus groups, open-ended survey questions, key
informant interviews and responses to the questions posted on Facebook.
20
Case Study
Once the data was compiled into a Data Summary Report, the next step was to organize
the data in a way that would be useful to their evaluation goals. To help them more easily
notice trends and issues, Main Street Community Services decided to organize the data by
going back to the four key questions they asked in the evaluation:
1. What communication tools are working well?
2. What tools are working somewhat effectively but need improvement?
3. What communication tools are no longer effective at all?
4. Are there new ways of communicating that should be implemented?
For each of these four questions, MSCS grouped and recorded relevant results from all
applicable data collection methods into a written summary which they included in their
Data Summary Report.
Analyze the Data
With the evaluation data organized and compiled into their Data Summary Report, the
data was ready to be analyzed. The Board of Directors and staff of Main Street Community
Services reviewed the data compiled according to collection method, and brainstormed
the key themes and trends emerging from the data. They summarized their findings in the
following chart:
Analyzing the Data – By Data Collection Method
Data Collection Method Key Findings
Online survey Stakeholders found the look of the website to be outdated
Stakeholders liked the Facebook page, but found that the page was not active enough
Most stakeholders have never heard of the MSCS blog, and those who had visited it thought it was very outdated.
Stakeholders valued the e-newsletter
21
Case Study
Analyzing the Data – By Data Collection Method
Data Collection Method Key Findings
Print survey Clients found the website hard to navigate
Clients liked the bulletin board at the office
Clients liked the Facebook page, but were intimidated to participate
Focus group with clients Clients found the website to be static and boring
Clients were not aware of the e-newsletter or MSCS blog
Clients liked to have the brochures to share with their friends and family
Clients wished special organizational events were set up on Facebook, so that they could easily invite their friends
They wanted to see more stories of people like them
Focus group with Board members, staff and volunteers
The Board, staff and volunteers thought the website needed some minor tune-ups
They greatly valued the e-newsletter and agency bulletin board
Half valued the Facebook page, while the other half thought it was a waste of time
They thought that the brochure had value, but that it was outdated
Staff did not have the time to keep the blog up to date
Open questions on Facebook
People wanted more content, more stories, more links to events, and a more dynamic organizational presence on Facebook
22
Case Study
Analyzing the Data – By Data Collection Method
Data Collection Method
Key Findings
15 key informant interviews
Stakeholders found the website to be overly complicated
Stakeholders didn’t find the brochure helpful, as they wanted to access organizational information online
Stakeholders liked having the posters to put up in their offices
Stakeholders wanted to know more about MSCS and about client success stories, but found that this type of information was not readily available
They didn’t have a clear idea of MSCS’ achievements
No one had ever visited the MSCS blog
External stakeholders thought the MSCS’s Facebook page had potential and that they should be on Twitter as well
Open questions on Facebook
People wanted more content, more stories, more links to events, and a more dynamic organizational presence on Facebook
Facebook analytics Upon reviewing Facebook Insights for the three months, it was noted that the most popular posts (i.e., the ones that were liked, shared and commented upon the most) were posts about client success stories and achievements
The majority of Facebook posts were seen by at least 75% of the page’s followers, although less than 5% of followers interacted with the posts (liked, shared or commented)
One post was “boosted” (meaning that they paid Facebook a small fee to promote the post) during the month; this post reached 10x the usual number of Facebook users and was shared 25 times
Following the post’s boosting, 10 new page likes were received
By looking at the compiled data according to data collection method and putting the key
findings into this chart, some strong messages about organizational strengths and
weaknesses in the area of communication quickly began to emerge.
23
Case Study
Next, the Board of Directors and staff of Main Street Community Services looked again at
the Data Summary Report, but this time they analyzed the data by looking at the four key
evaluation questions.
Their findings were summarized in the following chart:
Analyzing the Data – By Evaluation Question
Evaluation Questions Key Findings
What communication tools are working well?
Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.
Internal stakeholders greatly valued the bulletion board.
Their posters were working well as a communication tool.
What tools are working somewhat effectively but need improvement?
Stakeholders valued their website as a communication tool, but found the MSCS website to be outdated and not user-friendly.
Stakeholders valued the Facebook page, but thought that it should be used more dynamically. Followers visit the page, although they aren’t actively interacting with it. Analytics are beginning to show more clearly the type of posts that generate interaction, so staff can use this knowledge for future posts. The only boosted post was very successful, so boosting should be considered for select posts.
Brochures received mixed reviews depending on the stakeholder group questioned.
What communication tools are no longer effective at all?
The blog was not valued by stakeholders, and the staff did not have time to keep it up to date.
Are there new ways of communicating that should be implemented?
External stakeholders often mentioned that MCSC should set up a Twitter account.
All stakeholders wanted to see videos implemented. This reflects the worldwide trend of the popularity of video, and statistics show greater interaction with communication materials when video is included.
24
Case Study
Conclusion
By compiling and organizing the data in a simple yet
effective way, the Board of Directors and staff were
able to efficiently conduct an analysis of the data. This
analysis revealed some strong strengths, weaknesses,
opportunities and trends in the communication tools
and strategies used by Main Street Community Services.
With a clear picture in hand of what stakeholders were
telling them about communication in their organization,
Main Street Community Services was ready to move on
to the next step (and perhaps the most important and
neglected phase of all) – using the data to take action to
improve services.
Taking Action: Case Study
The fourth phase of our evaluation cycle is “Taking Action”. It can sometimes be difficult to
summon the energy and time to implement the results of your evaluation. However,
implementing your evaluation results in order to create a stronger and more effective non-
profit organization lies at the very foundation of why you engaged in evaluation in the first
place.
So, take heart, and let’s follow along with Main Street Community Services to see how they
used our “Taking Action” module to implement their evaluation results in order to meet
their goal of improving the way they communicate with stakeholders.
Main Street Community Services followed the “Five Steps to Taking Action” described in
our module:
1. Assess What the Data is Telling You
2. Set Your Priorities
3. Develop Your Recommended Actions
4. Create an Action Plan
5. Implement and Monitor Your Action Plan
25
Case Study
Step 1: Assess What the Data is Telling You
The Board of Directors, staff and several volunteers
of Main Street Community Services met together and
carefully reviewed the results of their data analysis.
Then they asked the all-important question: “What is
the Data Telling You?”
They used the “What is the Data Telling You?”
template from our module to record the results.
“What is the Data Telling You?”
Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.
Internal stakeholders greatly valued the bulletion board.
Their posters were working well as a communication tool.
Stakeholders valued their website as a communication tool. However, their website is outdated and not user-friendly.
Stakeholders valued the MCSC Facebook page. However, their Facebook page is not actively used, and MSCS should develop new strategies to create a more engaging and dynamic page. Facebook analytics has provided information about the type of posts that are most successful.
Brochures received mixed reviews depending on the stakeholder group.
Their blog was not valued by stakeholders, and the staff did not have time to keep it up to date.
External stakeholders often mentioned that MCSC should set up a Twitter account.
All stakeholders wanted to see videos implemented as a communication tool.
26
Case Study
Step 2: Set Your Priorities
Main Street Community Services’ evaluation data certainly revealed many communication
issues! However, as a small, non-profit organization with a limited budget and capacity,
MSCS was not able to implement all of the communications needs identified. Instead, they
had to set priorities based on factors such as the importance of the issue to MSCS, their
relation to the organization’s mission and their ability to implement the change.
Main Street Community Services used our “Prioritizing Issues Template” to help them set
priorities. For the purposes of this case study, we will use just one of the issues identified
by MSCS.
Prioritizing Issues Template
Our Facebook page should be used more actively and dynamically
Does this issue have a strong link to your organization’s values and mission?
Yes Somewhat No
Notes: Yes. Our stakeholders value our Facebook page. It is an important and cost-effective way for us to communicate.
Does this issue provide a clear benefit to your clients or other stakeholders?
Yes Somewhat No
Notes: If we can learn to be more dynamic on Facebook, we will be able to communicate more effectively with our stakeholders.
Would your key stakeholders be supportive if changes were made to this issue?
Yes Somewhat No
Notes: Yes. Stakeholders are driving the request to change.
Do your staff members / volunteers / other stakeholders have the ability (time and skill level) to implement changes to this issue?
Yes Somewhat No
Notes: One staff member and several volunteers have intermediate skills in this area. One volunteer in particular is eager to learn more and take on this challenge.
27
Case Study
Do you have the resources (financial and capacity) to address the needs identified in this issue?
Yes Somewhat No
Notes: We have limited financial resources, but Facebook is free. It is staff and volunteer time that will be impacted. However, since we plan to stop hosting our blog, some capacity should be freed up.
Do you have the ability to influence this issue or is it outside of your ability to control?
Yes Somewhat No
Notes: We have complete control over our Facebook page and are limited only by our financial and HR resources.
Is this an issue we can and should be addressing now?
Yes Somewhat No
Notes: Yes. This issue is manageable and important for us.
Based on the above, on a scale of 1 to 10 (with 1 being low, and 10 being high), what priority level would you assign to this issue? Priority Level: 9
28
Case Study
Using this same “Prioritizing Issues Template”, Main Street Community Services assessed
all of the communications issues identified in the evaluation. The following priorities for
action were developed, and designated as high, medium or low priority, or assigned to a
“no change needed” category.
High Priority
Stakeholders valued their website as a communication tool. However, their website is outdated and not user-friendly.
Stakeholders valued the MCSC Facebook page. However, their Facebook page should be used more actively and dynamically.
The blog was not valued by stakeholders, and the staff did not have time to keep it up to date.
All stakeholders wanted to see the use of videos implemented.
Medium Priority
External stakeholders often mentioned that MCSC should set up a Twitter account.
Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.
Low Priority
Brochures received mixed reviews depending on the stakeholder group.
No Change Needed at this Time
Internal stakeholders greatly valued the bulletin board.
Their posters were working well as a communication tool.
29
Case Study
Step 3: Develop Your Recommended Actions
Main Street Community Services developed four clear
priorities for action in order to meet its goal of
improving communication with stakeholders:
1. Stakeholders valued their website as a
communication tool. However, their website is
outdated and not user-friendly.
2. Stakeholders valued the MCSC Facebook page.
However, their Facebook page should be used
more actively and dynamically.
3. The blog was not valued by stakeholders, and
the staff did not have time to keep it up to date.
4. All stakeholders wanted to see videos
implemented as a communication tool.
Next they turned their priorities into clear, strong, actionable recommendations, as
follows:
Our Recommended Actions
1. Main Street Community Services will conduct an in-depth update of its website.
2. MSCS will use its Facebook page more actively and dynamically.
3. MSCS will cease hosting its blog and delete it immediately.
4. MSCS will assess costs, staff and volunteer skills sets, as well as research potential grant or foundation opportunities in order to begin the process of implementing videos as a communication tool.
30
Case Study
Step 4: Create an Action Plan
Main Street Community Services had four clear, actionable recommendations to help the
organization meet its goal of improving communications with stakeholders. The next step
was to develop a plan of action for implementing each of the recommendations. They kept
in mind the tried and true statement that “A goal without a plan in just a wish.”
Main Street Community Services used our “Action Plan Template” from the “Taking
Action” module to set timelines, responsibilities, available resources and implementation
strategies for each of the four recommendations.
In our case study, we will showcase their strategy for implementing one of their
recommendations: “MSCS will use its Facebook page more actively and dynamically”.
31
Case Study
Action Plan Template
WHAT? MSCS will use its Facebook page more actively and dynamically.
WHY?
Stakeholders valued the MCSC Facebook page. However, their Facebook page is not actively used, and MSCS should develop new strategies to create a more engaging and dynamic page.
HOW? WHO? HOW MUCH? WHEN?
Two volunteers will visit the Facebook pages of other non-profits with similar missions to search for best practices and new ideas.
Volunteers Free Next six weeks
The Executive Director will commit to posting two Facebook posts per week.
Executive Director
Two hours off per month in lieu of pay
Immediate and ongoing
Two volunteers will commit to creating one post each per week on Facebook.
Volunteer Free Immediate and ongoing
The Executive Director or Board Chair will find a well-known person from the community to do a monthly guest post.
Guest Speaker
Free Monthly and ongoing
The Board, volunteers and clients will actively post comments or questions, and to like and share MSCS Facebook postings.
Board, volunteers and clients
Free Ongoing
MSCS will pay to boost Important Facebook posts.
Executive Director
$50 annually As needed
The Office Coordinator will set up all future events held by MSCS as Facebook events.
Office Coordinator
Incorporate into regular staff time
Ongoing
32
Case Study
Step 5: Implement and Monitor Your Action Plan
Main Street Community Services now had an Action Plan that clearly set out its priorities,
tasks, timelines, resources and responsibilities. They were ready to take the final steps of
implementing and monitoring their plan of action. Work days in a non-profit are busy, and
it’s easy to fall behind on good intentions, even with the best of plans! Regular check-ins
can save the day and keep everyone motivated and on track.
MSCS reviewed the strategies for implementing and monitoring an action plan in our
“Taking Action” module, and used our “Monitoring Your Action Plan Template” to help
keep everyone on track and moving forward.
In our case study, we will showcase their strategy for monitoring their action plan by
tracking the implementation of one of their action items: “Two volunteers will visit the
Facebook pages of other non-profits with similar missions to search for best practices and
innovative ideas.”
33
Case Study
Monitoring Your Action Plan Template
ACTION: Two volunteers will visit the Facebook pages of other non-profits with similar missions to search for best practices and innovative ideas.
WHO?
Our plan:
Two volunteers will conduct the research.
Actual:
One volunteer resigned due to health issues.
One volunteer did some of the research but needed help from the Office Coordinator.
Notes / Follow-up / Changes:
This was too much work for one volunteer. The Office Coordinator agreed to help out.
HOW MUCH?
Our plan:
Free (volunteer time only)
Actual:
6 hours of staff time
Notes / Follow-up / Changes:
More staff time was needed. Since the Office Coordinator was no longer maintaining the blog, she had these 6 extra hours to help out.
WHEN?
Our plan:
Six week timeline
Actual:
With one less volunteer, an extra week was needed to complete this task.
Notes / Follow-up / Changes:
A week delay did not affect this action. This action was successfully completed!
34
Case Study
Conclusion
By following our five step process to taking action on evaluation results, Main Street
Community Services was able to assess what the data was telling them, set their priorities,
develop their recommended actions, create an action plan, and identify a process for
monitoring implementation.
Using our “Taking Action” module, MSCS kept the process straightforward and clear. They
successfully developed a plan of action that would help them meet their evaluation goal of
improving communication with stakeholders.
With a successful process and plan in hand, Main Street Community Services arrived at the
final step of the evaluation cycle: communicating the results!
35
Case Study
Communicating the Results: Case Study
The fifth and final phase of our evaluation cycle is “Communicating the Results”. By this
point, Main Street Community Services had planned the evaluation, collected and analyzed
data and created an action plan for implementation. Those are very impressive tasks! Now,
there is just one more important phase left to be carried out in the evaluation cycle:
communicating the results.
Main Street Community Services knew that organizational communication is of critical
importance. In fact, that is why they chose to evaluate their communication strategies and
tools in the first place! MSCS knew that they should communicate their evaluation results
for a variety of good reasons, including profiling organizational success and positive
outcomes, showing organizational transparency, increasing credibility and promoting a
culture of evaluation. However, as a small, non-profit organization with few resources,
Main Street Community Services also knew they needed to be strategic in how they
communicated the results.
MSCS followed the step-by-step process outlined in our “Communicating the Results”
module to figure out the “what, why, who and how” of communicating their evaluation
results. Let’s follow along and see how they did it.
What Evaluation Results Should You Communicate and Why?
Main Street Community Services knew that
their stakeholders are busy people. So,
although they had prepared a summary report
on their evaluation results, they believed that
this report was more for internal use and that
stakeholders would only be interested in the
key findings.
To decide what to share, the Board and staff
of MSCS asked themselves two key questions:
1. What are the key findings of our evaluation?
2. Of these findings, what needs to be communicated to our stakeholders and why?
36
Case Study
Their responses to these questions are summarized in this template from our module:
What evaluation results should you communicate and why?
What are the key findings? Does this evaluation result need to be communicated?
Why are we sharing (or not sharing) this result?
Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.
IN PART. This finding will be used internally for staff to develop strategies to increase client awareness of the e-newsletter.
MSCS wanted to better engage clients by encouraging them to subscribe to the e-newsletter.
Internal stakeholders greatly valued the bulletin board.
No. MSCS is not changing their strategy for this item.
No change in strategy. Internal impact only.
Stakeholders valued their website as a communication tool. However, it is outdated and not user-friendly.
YES. The highlights of this finding should be communicated broadly.
Their website is a criticial communication tool for all stakeholders.
Stakeholders valued the MCSC Facebook page. However, their page is not actively used, and MSCS should create a more engaging page.
YES. The highlights of this finding should be communicated broadly.
Facebook is an important communication tool for many stakeholders.
Their blog was not valued by stakeholders, and the staff did not have time to keep it up to date.
YES. Since an activity is completely ceasing, it is important to let stakeholders know.
It is important to share this news for organizational transparency.
All stakeholders wanted to see videos implemented as a communication tool.
YES. Stakeholders need to know that MSCS heard their input on this issue.
This is a new strategic area for MSCS. Active communication is key.
37
Case Study
Who Needs to Know our Evaluation Results?
As with all non-profit organizations, Main Street Community Services had a wide array of
stakeholders. They knew it was important to keep stakeholders informed during the evaluation
in order to increase awareness of the evaluation process and results, gain support for needed
changes, show respect for stakeholder input and build a culture of evaluation at MSCS.
However, they also knew that some of the evaluation findings would be of more relevance and
interest to certain stakeholder groups than others. Accordingly, the Board and staff asked
themselves, “Which of our stakeholder groups need to know about this evaluation result?”
Main Street Community Services then used the template from our module to answer these
questions.
Who Needs to Know Our Evaluation Results?
What were the key findings? Who needs to know?
Their monthly e-newsletter was valued by most stakeholders. Clients, however, were not sufficiently aware that they could subscribe to it.
Staff
Clients
Internal stakeholders greatly valued the bulletin board.
Internal stakeholders
Stakeholders valued their website as a communication tool. However, it is outdated and not user-friendly.
All stakeholders
Stakeholders valued the MCSC Facebook page. However, their page is not actively used, and MSCS should develop new strategies to create a more engaging page.
All stakeholders
Their blog was not valued by stakeholders, and the staff did not have time to keep it up to date.
All stakeholders
All stakeholders wanted to see videos implemented as a communication tool.
All stakeholders
38
Case Study
How Should You Communicate Your Evaluation Results?
Main Street Community Services discovered the “what, why and who” of communicating their
evaluation results. Now, they needed to figure out how to communicate strategically with the
major stakeholder groups.
There are so many ways to communicate, including formal and informal communication, and via
traditional and new media. Main Street Community Services considered what needed to be
communicated, who needed to know and then assessed the best way to communicate each
evaluation result in a way that would be meaningful to their stakeholders.
This chart records what they decided:
How Should We Communicate Our Evaluation Results?
Action Item Who needs to know?
Communication methods
Monthly e-newsletter
Staff
Clients
Personal conversations with clients
E-mails to clients
Update client information packages
Bulletin board
Internal stakeholders
Post a notice on the bulletin board
Website All stakeholders
Share via all communication channels
Also, prepare a major re-launch strategy when the website is revamped
Facebook page
All stakeholders
Share via all communication channels
When new strategies are ready, promote a post on Facebook announcing the changes
Add links to Facebook on emails, website
Consider holding a virtual “launch event”
Blog All stakeholders
Share via e-newsletter, website, Facebook, email, and via a farewell Blog posting
Videos All stakeholders
Share via all communication channels
39
Case Study
Effective Communication Strategies
As you can imagine, the Board, staff and volunteers of Main Street Community Services
were both excited and proud of figuring out the “what, why, who and how” of
communicating their evaluation results! In fact, they were so proud of their achievements
that they wanted to “up their communications game” by trying out a few new
communication methods. Accordingly, they reviewed the “Communicating Using New
Media” part of our module.
Based on what they learned, MSCS implemented some new communication strategies:
They used pictures to share their evaluation results.
Main Street Community Services created their own graphic images using Canva to
share results (see image below).
They shared quotes from the evaluation.
They even tried to make an infograph using Piktochart. This was, at first,
unsuccessful, but at least they got started and expanded their knowledge.
All we can say is, “job well done” to the amazing team at Main Street Community
Services!
40
Case Study
Conclusion
We hope that our Case Study module has helped you to clearly see the evaluation cycle in
action, as we walked alongside Main Street Community Services while they conducted
these five critical steps:
Planning for Evaluation
Collecting Data
Analyzing Data
Taking Action
Communicating the Results
It is our goal that these online training modules will help you to build a culture of
evaluation in your non-profit organization. Furthermore, we hope that our modules have
given you the inspiration to know that evaluation in non-profit organizations can be
manageable, and can lead to lasting improvements for our clients and communities.
Be sure to continue the evaluation conversation. Here are some ways to do that:
During the life of the “Developing a Culture of Evaluation” project (it ends in March
2017), visit our website to join our online discussion group, online clinics and/or
webinars so that we can continue to learn together.
Stay connected with the Ontario Nonprofit Network, Imagine Canada and other
organizations with expertise in non-profit evaluation.
Participate in conversations about evaluation. Talk to your fellow non-profit
organizations, as well as your volunteers, Board of Directors, funders, donors and
community members.
Share your thoughts with us via our Facebook page or on Twitter.
“True genius resides in the capacity for evaluation of uncertain,
hazardous, and conflicting information.”
(Sir Winston Churchill)
41
Case Study
Acknowledgements
The Case Study module is one of eight online modules created as part of the Developing a
Culture of Evaluation project. All project resources can be freely accessed online at:
www.communityliteracyofontario.ca/evaluation-culture.
This project was conducted through a partnership between Community Literacy of Ontario
and Literacy Link South Central.
The Case Study module was written by Joanne Kaattari.
This project was funded through the Partnership Project of the Ontario Ministry of
Citizenship, Immigration and International Trade. We are very grateful for their support.
The views expressed in this module are the views of Community Literacy of Ontario and
Literacy Link South Central and do not necessarily reflect those of the Province.
Date of publication: March 2016.
All information and weblinks provided in this module were accurate at the time of
publication.
©Copyright Community Literacy of Ontario and Literacy Link South Central.