Saskatchewan Liquor & Gaming Authority HOST Saskatchewan Conference March 14 - 15, 2010.
Introduction - wiki.usask.ca · Web viewProcess Evaluation Plan. Prepared for the Heart and...
Transcript of Introduction - wiki.usask.ca · Web viewProcess Evaluation Plan. Prepared for the Heart and...
Process Evaluation PlanPrepared for the Heart and Stroke Foundation of
Saskatchewan
December 6, 2010
Prepared By University of Saskatchewan MPH Students:Sunisha Neupane, Stuart Lockhart, Heather Davis, Paul Duong, Alim
Gillani, Prachi Bandivadekar
Table of Contents
1. Introduction.............................................................................................................................1
2. Purpose..................................................................................................................................3
3. Proposed Evaluation Design..................................................................................................5
3.1 Evidence-based Indicators for C.A.R.E...................................................................................73.1.1 Community Action..........................................................................................................83.1.2 Advocacy....................................................................................................................... 93.1.3 Research...................................................................................................................... 103.1.4 Education..................................................................................................................... 11
4. Data Sources and Data Collection Methods........................................................................12
4.1 Data Sources........................................................................................................................ 134.2 Data Collection Methods.......................................................................................................144.3 C.A.R.E Process Evaluation Strategies................................................................................15
4.3.1 Community Action........................................................................................................164.3.2 Advocacy..................................................................................................................... 174.3.3 Research..................................................................................................................... 184.3.4 Education..................................................................................................................... 19
5. Data Analysis.......................................................................................................................20
5.1 Analyzing Qualitative Data....................................................................................................215.2 Analyzing Quantitative Data..................................................................................................22
6. Evaluation Resources..........................................................................................................24
6.1 Budget.................................................................................................................................. 256.1 Human Resources................................................................................................................25
7. Conclusion............................................................................................................................26
7.1 Summary.............................................................................................................................. 277.2 Recommendations................................................................................................................28
Appendix 1 Evaluation Plan Checklist..............................................................................................30
Appendix 2 Evaluation Plan Checklist..............................................................................................31
2
1. Introduction
Established in 1956, the Saskatchewan Heart Foundation (a provincial division National
Heart Foundation of Canada) founded by a visionary group of Canadians who had a dream to
turn the tide on heart disease. Later renamed to the Heart and Stroke Foundation (HSF) with an
effort to more accurately reflect the scope of the foundations vision and hopes for the future.
Growing both nationally and provincially the HSF continues to endeavor to put heart health on
the public agenda, to elevate researchers, alleviate the burden of heart disease, and translate
the knowledge emerging from research to education for Canadians about their hearts. Since
the establishment of the HSF more than $1 billion has been raised and invested into
innovationative heart and stroke research. Moreover, in an effort to address need where it is
most required more than 80 per cent of donations directly support research and education
programs in the province where the funds are raised. With the increasing burden of heart and
stroke-related diseases increases to impact the lives of so many Saskatchewan residents the
Heart & Stroke Foundation of Saskatchewan (HSFS) continue to receive impressive financial
support striving to reach new milestones in research support, health promotion, and advocacy.
These inspiring results are only made possible by the ongoing and valuable efforts of volunteers
and donors, and the success of HSF programs across this province and the country from coast
to coast (Heart & Stroke Foundation Canada, 2010).
The HSFS with the support and guidance of visionary doctors and researchers, staff
and, the corporate community and the community at-large, plays a relevant leadership role in
the study, prevention and reduction of disability and death from heart disease and stroke in
Saskatchewan. Heart disease continues to be an important public health problem in
Saskatchewan that requires a focus on solutions at the population-level. The HSFS is dedicated
to reducing the risk factors that are associated with heart disease and stroke in addition to
improving outcomes and instruction for survivors (Heart & Stroke Foundation Saskatchewan,
2010).
Heart disease and stroke are two of the three leading causes of death and the leading
cause of disability in Canada. Canadian baby boomers are now moving into their middle years
which are expected to result in a large increase in death and disability due to heart disease and
3
stroke. (Heart & Stroke Foundation Saskatchewan, 2010). This underlines the importance of
primary, secondary and tertiary prevention strategies. With Saskatchewan ranking high in its
smoking and obesity rates, it is as pertinent as ever to push residents to engage in healthy
eating, physical activity and tobacco cessation (Buller, 2010). Research has shown that
negative lifestyle choices such as sedentary lifestyle, smoking, porr nutritional choices, etc. play
a synergistic role in the development of heart and stroke related-diseases, and via education we
can change these negatives into positives and endeavor for a more heart smart Saskatchewan.
Collaboration between governments, researchers, health care practitioners, communities, and
individuals are necessary to extend the reaches of heart healthy programming and education
across the province (Buller, 2010) (Heart & Stroke Foundation Saskatchewan, 2010).
The HSFS is currently developing a strategy in health promotion that will benefit
communities throughout the province. Their aim is to eliminate heart and stroke related disease
and reduce their impact through the advancement of research and apply researched based
knowledge to the advocacy and promotion of healthy heart smart living. In order to determine
the efficacy of the programs within this model and their adherence to the initial guidelines set
forward, it is necessary to evaluate the Health Promotion process. The goals set forth in these
programs are:
To keep people healthy so that they do not suffer from heart or stroke related-disease in
the first place.
To prevent progressive, more debilitating disease in those affected by heart disease,
stroke and ischemic attack.
To reduce the incidence of death and disability from cardiovascular disease and stroke.
Focusing primarily on prevention, this document will lay the foundation for the evaluation
steps involved in the Heart and Stroke Foundation's health promotion programs. The
Community Action, Advocacy, Research and Education (C.A.R.E.) approach is a strategy that is
being used to achieve the previously stated goals. Although, these elements are interrelated, a
separate focus on each allows us to better describe the role that each can play in health
promotion programs. With defined target communities outlined by HSFS’s Community Action
Plan (Kaar, 2010) C.A.R.E. aims to give the communities ownership of the prevention programs
and processes. By engaging the community and providing a sense of ownership, HSFS can
4
identify strong leaders and companions situated within the community, leveraging the most it
can out of local resources. This will allow all parties to efficiently and effectively work within the
organization capacity and build for the future while continuing the ongoing process of
community development. C.A.R.E. will provide each community a support system, skills and
knowledge aligning with HSFS mission and vision. This will only lead to strong networks and
partnerships for sustainable programs for the short and long term. Working with individual
communities HSFS can develop programs that will best involve the local health issues and
prioritize common health issues found throughout Saskatchewan. With success in mind the
C.A.R.E. programs we outline clearly articulated goals mechanisms for decision making and
evaluations based on predefined outcomes and indicators to be achieved specific to each
section of C.A.R.E. (Kaar, 2010) (Issel, 2009).
5
Figure 1.1 Heart & Stroke Foundation Saskatchewan Health Promotion Logic Model. (Adapted
from: Mission in Action. Heart & Stroke Foundation Saskatchewan (Buller, 2010).
6
2. Purpose
Process evaluations are an important tool when it comes to the implementation of all
program types. They help us to understand why or why not a program was successful and they
can determine the effectiveness of a program prior to the investment of valuable resources
(Saunders, et al, 2005). This makes process evaluations beneficial from a financial standpoint
as well as an operative stance. Program implementation can be quite variable depending on
many factors so it is constructive to measure program performance in many regions and
communities throughout the province. To implement this strategy and standardize the C.A.R.E.
approach to health promotion, we have generated a process evaluation plan for the activities
and methodologies that will be carried out to improve the heart health of the people of
Saskatchewan.
The framework that we used for this evaluation follows the format of the Centers for
Disease Control and Prevention (CDC), Division for Heart Disease and Stroke Prevention
(CDC, 2006). Within this structure we took specific steps to investigate the quality,
completeness, exposure, satisfaction, participation rates, recruitment and context of the
programs that are going to be used in the C.A.R.E. approach. Looking to the literature for
guidance, we found evidence for key indicators that can be used to look at varying aspects of
community action, advocacy, research and education. For each indicator that we found, we
generated questions that are necessary to measure the achievement of the activities performed.
The answers to these questions can be found from many different sources which is why we
identified appropriate locations to find this data. While some of the necessary data is already in
existence through administrative records, logs, etc, others must be collected directly from
communities as primary data. The collection techniques and methodologies are also defined in
this report. Finally, the analysis of the data must be done in order to understand the results of
the data and to draw conclusions that can impact recommendations for revision.
7
3. Proposed Evaluation Plan
3.1 Evidence-based Indicators for C.A.R.E
3.1.1 Community Action
In developing criteria to evaluate the measure of success in implementing aspects of the
Community Action component of HSFS’s program strategies, it was determined that due to
measurement limitations most criteria would be based in self-reflection and would not be
quantifiable. In addition, because the Community Action Coordinator (CAC) plays such a direct
role in this part of the overall strategy, many of the criteria are directed at the performance of
this person. The performance of these people will stem directly from their ability to lead
effectively. The term best describing the two persons - who will work directly with communities
in enabling them, inspiring them, and mobilizing them - is leader. From this viewpoint, their
performance as leaders should be monitored.
Measuring Success of Group Interaction Sessions
The indicators that have been chosen for this portion of the C.A.R.E. strategy are
directed at measuring the success of community sessions from the viewpoint of community
members in attendance. The reason their input is valued more than the that of HSFS employees
is because community action is about grassroots involvement and must ultimately shape to the
mould of the community. In this sense, community action would be better termed community-
based action.
Measuring Coordinators’ Success as a Leader
The role of the Community Action Coordinator is to facilitate the community action
process. This involves communicating with community leaders to begin the community
engagement and action process; facilitating sessions of problem identification, solution
formulation, and monitoring and evaluation; and providing resources and support to
communities initiating projects. The HSFS’s presence at community action sessions will also
have an education component. Teaching community members about other community action
projects in other parts of the province will help to motivate the community about the potential
8
that it has to create a heart healthy community. A renowned theorist about critical education,
Paulo Freire of Brazil, articulated the ideas of empowerment education that has contributed so
much to discussion about engagement within communities. “Education should have as one of
its main tasks to invite people to believe in themselves. It should invite people to believe they
have the knowledge (Bernstein & Wallerstein, 1988).” Empowerment education should be an
essential component of the CAC in delivery of community action sessions, though evaluation of
the CAC should be based on a more complete framework.
The criteria recommended to evaluate the leadership performance of the Community
Action Coordinator will be based upon the Kouzes and Posner Leadership Model. The criteria
used are derived from four of the Five Practices of Exemplary Leadership: Model the Way,
Inspire a Shared Vision, Enable Others to Act, Encourage the Heart. This leadership model is
built upon years of research by Jim Kouzes and Barry Posner about what it is that is essential to
get extraordinary things done in organizations. The research was based on qualitative accounts
of thousands of “personal best” stories.
3.1.2 Advocacy
All the programs are benefited from proper alliance, and it is necessary to evaluate it in
the context of the health program. The HSF in its program has recommendations for
development and implementations of tobacco control programs, food banks, government
agencies to implement various programs and policies etc. It also needs various collaborations
for delivery of services to various smaller communities at the local level. Within C.A.R.E. the
HSFS will use advocacy to form collaborations or partnerships for mutual benefit and the
common objective reduction of heart and stroke related-disease. Partnerships will be forged
with a person or an organization. Moreover, HSFS may form mutual strategic alliances at the
government, partner organizations, community or local levels to benefit wider range of
residents. With HSFS mission the involvement of partners from the public and private sectors is
needed to enhance sustainable health capacity of a community, and remove barriers that
enable the programs of C.A.R.E to work effectively (Heart & Stroke Foundation Canada, 2010).
The goals of health promotion programs are to provide lasting change in the community
that will lead to reduction of heart and stroke related-disease. Schwartz et al. (1993) report that
advocacy is facilitated at two levels. The first level involves formation of partnerships with
9
community organizations and citizens committing to advance program efforts. The second
involves environmental and policy interventions that are likely to remain and to result in positive
permanent change (Elder et al. 1993) (Schwartz et al. 1993).
Private Partnerships
Creating partnerships with members and organizations that are established in the
community will attract new stakeholders which are vital to sustaining programs. Thus, it further
engages the community and respected members of the community, not only legitimizing the
programs efforts, but furthering the ownership of the success of the program and the
community’s role in them. HSF has a base of long lasting partnerships already, however many
of these partnerships are in the larger centres, C.A.R.E. will be providing programs throughout
the province making the establishment of new local and provincial partnerships essential. In
smaller communities support by local organizations will only further aid in program success.
However, evaluating the support and satisfaction of partnerships is essential for maintaining
existing partnerships and forming long lasting new partnerships (Elder et al. 1993) (Schwartz et
al. 1993).
Public Partnerships
Formation of partnerships with any level of government could lead to successful policy
interventions for the reduction of heart and stroke related-disease. A successful partnership
would have a foundation of evidence such as data provided by evaluations. Moreover,
successful government partnerships would use HSF input within the context of social-planning
model, so that the development and implementation of policies are relevant to health
behaviours. Whether they are school or worksite policies or community and provincial
legislature, policies are often implemented to suppress undesirable or unhealthy behaviour.
Heart smart health promotion involves getting people to eat, drink, smoke less or not at all and
be more physically active. To some extent, individuals will always retain the “right to choose” to
be unhealthy. However, in many cases it is not a choice, but an intrinsic cycle perpetuated by
lack of education and social supports, only furthering the critical role of public partnerships
(Elder et al. 1993) (Schwartz et al. 1993).
10
3.1.3 Research
Bridging the gap between research findings and practical applications is essential to
population health intervention strategies. Many health promotion programs have failed because
evidence-based knowledge was not optimally transferred to practice. Zapka et al. cited three
types of potential program failure: (a) theory failure (the wrong strategy, messages, or vehicle
chosen for the intervention); (b) program implementation failure (the right strategy, messages,
and medium were planned but not implemented); and (c) measurement failure (the right theory
for the program, which was implemented as planned, however, the measurement methods were
not sensitive enough or not valid or reliable) (Zapka, Goins, Pbert, & Ockene, 2004). In order to
avoid these pitfalls, the Heart and Stroke foundation recognizes the importance of knowledge
transfer and exchange. As such, they have dedicated their efforts to support “the development
and application of policy- and practice- relevant research in a way that draws on the contribution
of both researchers and knowledge users” (Heart & Stroke Foundation of Canada, 2009).
Evaluating knowledge transfer and exchange activities is necessary to determine which
aspects work and areas that can be improved upon through modification and refinement. After
all, effective knowledge transfer and exchange efforts treat knowledge as a means to improve
practice and situations by having positive impacts, rather than as an end in itself (Sperling, Von
Sychowski, & Zarinpoush, 2007). Keeping this in mind, we developed a framework to facilitate
the evaluation of knowledge transfer and exchange activities. These activities include:
distribution and accessibility, dissemination, presentations, training sessions, workshops,
conferences, seminars, and meetings. By evaluating these activities, we can measure the
change in: user’s capacity to apply knowledge, integration of evidence into decision-making
process, collaboration between knowledge producers and users, cultural shift within the
organization or a community of practice.
Capacity to use knowledgeConducting effective knowledge transfer empowers participants to apply the gained
knowledge to real-life situations and problems in their own contexts. Thus, knowledge transfer
and exchange activities that focuses on the application of knowledge – rather than merely the
provision of knowledge – can lead to an increase in the capacity of users to interpret and apply
what they have learned (Sperling et al., 2007). The Saskatchewan Heart Health Program
11
(SHHP) is a good example of building individual capacity. A capacity-rich individual was defined
as: One who clearly articulates their health promotion values and principles, enhances skills of
self and others, are computer literate, partners within and across Health Districts and sectors,
records and shares experiences, steps out of “comfort zones,” engages in reflective practice,
and learns continuously were considered (Ebbesen et al., 2001). The program conducted an
annual survey (Health Promotion Contact (HPC) Profile) in the fall of 1998 to get an
understanding of individual capacity in each of the Health Districts. They found that prominent
resources contributing to individual capacity were personal commitment, enthusiasm and
energy, along with management support for and understanding of health promotion. Key
barriers to individual health promotion capacity included: lack of time; conflicting responsibilities;
lack of shared understanding of health promotion; reluctance to attempt more challenging
approaches for fear of previous work being de-valued; and the long time frame implicit in health
promotion (Ebbesen et al., 2001).
Integrating evidence into decision-makingThe knowledge that is produced and exchange should help practitioners make well
informed decisions about the various policies, programs, and projects in which they are involved
(Sperling et al., 2007). Hence, it is important to know if knowledge transfer and exchange efforts
have influenced decision-making process. Once more, the SHHP can be used as a reference
for evidence-based decision making. Participants of this program spoke most often of using
research findings to encourage reflective practice through the design and process of their data
collection (Haalboom, Robinson, Elliott, Cameron, & Eyles, 2006). Facilitated sessions were
held for researchers and practitioners to make sense of research findings and develop action
plans tailored to the local and provincial context (Riley et al., 2009).
CollaborationActivities that support collaboration among researchers and practitioners can promote
the creation of knowledge that is current, relevant, and readily applied by users (Sperling et al.,
2007). Therefore, measuring the level of collaboration created through knowledge transfer and
exchange is important. Perhaps the best example of successful collaboration would be the
Canadian Heart Health Initiative (CHHI). The CHHI was groundbreaking in its attempt to bring
together researchers and public health leaders to jointly plan, conduct, and act on relevant
12
evidence (Riley et al., 2009). The data from this project helped the coalitions to recognize each
other’s weaknesses and strengths, highlighted opportunities to build rural community capacity
and contributed to their evolution to broader wellness coalitions (Haalboom et al., 2006). For
example, in Prince Edward Island (PEI), research had an influence on heart health
implementation where feeding back research results contributed to better quality and more
comprehensive programming based on data from evaluations (Haalboom et al., 2006).
Cultural shiftCultural differences between researchers and practitioners can significantly affect
knowledge exchange. Thus, it is desirable to foster a culture that encourages ongoing dialogue
between knowledge producers and users which can produce an environment whereby both
groups will better understand each other’s perspectives, experiences, languages, and needs
(Sperling et al., 2007). Again, evidence from the CHHI provides a good understanding of how
the relationship between knowledge producers and users can lead to positive results. In the
CHHI, leadership from well-positioned, influential people (scientists and public health
professionals) within the public health system was especially critical to mobilizing a productive
and deeply engaged pan-Canadian community (Riley et al., 2009). The conference of Principal
Investigators brought together provincial and federal CHHI leaders to facilitate the exchange of
plans, ideas, and tools (Riley et al., 2009). This allowed researchers and practitioners to draw
on each other’s experiences and expertise during proposal development, implementation,
analysis, and write-up phases.
3.1.4 Education
An important component to any health promotion strategy is education of the community
and its individuals about initiatives and programs that are available in the community. To convey
the messages of heart healthy living and risk factor prevention, workshops, seminars, training
events and programs can be used to bring people together, provide helpful information and
create a positive learning environment. Sharing primary prevention tools with community
members allows them the opportunity to make decisions to take control of their health and make
improvements if needed. Media messages are another part of education that play an important
role in communicating with the public. They increase awareness and provide healthy living
messages at a population level.
13
Marketing Tactics
In order to understand whether the educational tools, programs and marketing
techniques are being achieved successfully for a specific community, we can use a set of
indicators to determine their performance in the context of this health promotion strategy. We
can evaluate these indicators based on their adherence to implementation as intended, dosage
or amount of education provided, satisfaction and context.
In this evaluation, we must determine whether media messages put forth by the Heart
and Stroke Foundation are simple and easy to understand and how often they are carried out.
From the North Karelia project in Finland, it has been shown that media messages must be
simple and run as frequently as possible for them to be retained and comprehended (McAlister,
1982). The appropriateness of the venue for communication must also be assessed on a per
community basis. Saskatchewan is very expansive geographically and as such, extra thought
must be taken into the approach for conveying heart healthy living messages. If the right
communication techniques are being utilized, then community awareness of programs offered
by the Heart and Stroke Foundation for primary, secondary and tertiary prevention should most
likely be at a high level. This can be measured to evaluate marketing success (Issel, 2009).
Program Dosage
An important indicator to consider for educational programs, workshops, seminars and
events is the amount of sessions that are run and the number of participants who attend (Issel,
2009). This data can help guide the evaluation team in understanding the usage of existing
programs and determine whether there is a need to expand operations or to scale back due to
lack of participation. This can implicate future funding opportunities if attendance is outside of its
predetermined threshold range. If advertising of sessions is inadequate, then that aspect may
have to be addressed as well. Quality and completeness of programs should also be looked at
as an indicator in each community. The completeness of the intended program or educational
marketing tool can determine whether it will be successful or not.
Since every city and town differs in size, geography, population composition and health
needs, we must determine which programs are required in each community. Understanding the
needs of a specific community is essential to find out whether the services that are being offered
14
are in fact the right ones. Also, we would like to measure whether the program diversity
provided is adequate.
Program Satisfaction
Satisfaction with educational materials, advertising and programs will be evaluated from
a staff, volunteer and participant perspective. These indicators are extremely valuable to know
due to the significant impact that they have on the program as a whole. If those involved are not
satisfied with the program, they are not likely to become engaged in the educational messages
that they are being given. This is also important from a funding perspective because funding
agencies are interested in knowing the satisfaction level of the participants in the programs they
are funding (Issel, 2009).
These indicators as a whole provide insight into the success or lack thereof, the activities
being carried out to accomplish the goals initially set out. By measuring them, we can
understand which areas need more focus and how we can attempt to make improvements in
the education component of the health promotion strategy.
4. Data Sources and Data Collection Methods
4.1 Data Sources
The data, which is collected and analyzed by someone else other than HSF staff, is
secondary data for HSF. Secondary data should always be obtained from a reliable source so
that it is already validated and ready to use. It is essential to be careful to select only relevant
and representative data, which fits in the research that is being done. Here is a list of sources
that might be helpful for HSFS to obtain the secondary data depending on what type of
information is required:
Saskatchewan health insurance data files provided by Statistics Canada.
The Canadian Community Health survey, which provides summary every two years
(Neudorf, 2008).
15
Data from local hospitals regarding heart disease
File and document review of HSFS program documents, proposal, research materials
related to the program.
Public health surveillance data
The health plan employer data and information set (HEDIS). Along with other health
indicators, HEDIS measures controlling high blood pressure (Neudorf, 2008).
Meeting minutes (participants, issues discussed, follow-through)
4.2 Data Collection Methods
Primary data is collected to explain and validate the need of interest (Issel, 2009). Data
collection is required to answer all the questions raised in the indicators table. Few of the data
collection methods that are suggested in the table are discussed in further depth.
Key Informant Interview
Key informant interview provide significant information through in-depth questions and
probing. Detailed and rich data can be gathered in an easy and inexpensive way. This method
is useful when the information from specific people or specific group in the community is
required. For example, HSF is suggested to use this method to get the information on the group
who are not accessing the programs being offered. The key informants for HSF can also be the
people who are at risk for heart disease. HSF can use this method to build and strengthen the
relationship with important key informants and stakeholders in the community. A few
experienced, analytical, knowledgeable and informative individuals should also be interviewed
(McKillip, 1987). The researcher should try to recognize the understanding and motivation of the
community through the interview. However, while analyzing the data, special attention needs to
be paid on not evaluating the program solely based on key informant interview because
sometimes it might not represent the whole community. If the resources allow H&S to hire a
researcher to conduct the interviews it can be well structured, consistent (as one person does
all key informants) and easy to analyze the information received. As the question arises how
many interviews to conduct, it should be enough to make sure that different community groups
and people with different experiences are adequately represented (McKillip, 1987).
Community Survey
16
Surveys can be easy to administer and cheaper for Saskatoon acknowledging that the
communities are quite spread out. It is convenient if the participants wish to remain anonymous
as well. Community surveys work perfectly in getting diverse and specific information.
Step 1 (Developing questionnaire): The questions for community survey needs to
descriptive and objective. Too many questions at a single time might discourage community
members to participate in the survey. Before developing the survey questions, it is necessary
that HSF is clear on who the target population are and are the findings going to be useful?
There are four types of question structures that can be followed. The types and examples are
given below from Dillman, 1978.
i. Open ended (example: What should be done in order to improve access to Heart and
Stroke Foundation’s programs?
ii. Close-ended with ordered response (example: How often do you go to program/workshop
offered through HSF?
iii. Close ended with unordered response (example: Which programs do you attend most
frequently? List answers for participants to circle)
iv. Partially close-ended (example: Programs offered at which site of HSF is easiest to
access?)
Step 2 (Sample selection): Once the questions are ready, population needs to be sampled. Key
informant already will have some sort of criteria to select the participants so surveys can be
done by random sampling.
Step 3 (conducting survey): Three ways that are suggested for HSF to conduct their community
surveys are:
i. Online survey: Online survey can reach large number of population in easy and
inexpensive manner. SurveyMonkey (http://www.surveymonkey.com/) is a free and reliable
website for online survey.
17
ii. Mailing survey: Mailing out survey works good as well. The issue can arise when not many
participants mail back their survey. Hence, to mitigate that problem, HSF can partner with
grocery stores in the city and ask help in collecting back the survey. If the grocery stores agree
on holding a drop off box for the surveys, there would be a bigger participant rate.
iii. Booth in malls: If few booths are run in a busy mall, it is possible that large number of
people can be drawn to fill out the survey questionnaire. Few small incentives (candy, pen,
water bottle) can also be distributed to attract more people.
Attendance Log
Attendance log will serve the purpose of keeping track of how many people are
attending the offered workshops and programs. It can also help in identifying few key informants
based on attendance who might be able to answer questions such as
Why does someone attend one workshop in four months?
What motivates someone to attend all the workshops?
Here is an example of the attendance log that can be used:
Attendence log for Heart and Stoke information session Facillitator: Time: Site: Date:
NameDo you always come to this site?
Do you like attending programs offered by HSF?
Can you contact you if we have any questions regarding participant satisfaction?
Kim Crowe No Yes Yes
18
Session Evaluation Forms
After every session, an evaluation should be done to understand participant’s level of
satisfaction. Separate evaluation form for the program staff and the facilitator is suggested. This
way it is participatory and meaningful to both participants and facilitators. The only motive
behind evaluation form is to enrich the participant’s and staff’s experience. There should be
session (or program) specific evaluation form, which will help in improving the specific program.
More general evaluation form might not serve the same purpose. Example of an evaluation form
is given in the appendix.
Concluding Discussion
Concluding discussion also plays a similar role as evaluation form, but is more in depth.
Discussion can sometime take more time than expected; this method can be used when there is
no time restriction. This is a great method to collect feedback. Concluding discussion is also
beneficial in summarizing the session and making a list of lessons learnt.
4.3 C.A.R.E Process Evaluation Strategies
4.3.1 Community Action
4.3.2 Advocacy
In order to understand if advocacy is being successful and meeting goals, the use of a
set of indicators will be used to determine the effects of advocacy on the HSFS health
promotion program. Evaluation based on these indicators will allow HSF to know if Advocacy
was implemented as it was intended, if its meeting its objective, if there be any modifications
and the outcome (Heart and Stroke Foundation of Saskatchewan, 2010)(Issel, 2009).
In this evaluation, HSFS’s C.A.R.E. program needs to determine the quality and
quantities of collaborations that the HSFS has made for its program, the benefit that the HSFS
and the alliance agency is achieving and if it is overall benefit to the program. It is very important
to determine if the alliance is proper, and if it is meeting its objective. We can determine this by
knowing the kind of collaborations the HSFS has, with what kind of agencies, big or small,
19
government or private etc. the appropriateness of the alliance agency is also important for
determining the proper outcome of the program (Elder et al. 1993) (Schwartz et al. 1993).
Some of the Questions that can be asked to determine these indicators are
Do HSFS existing partnerships/alliances/collaborations support the strategic plan is the
alliance sustainable?
Is there adequate efficacy, efficiency and effectiveness?
Are the community board members adequately involved?
Benefit from the alliance to the HSFS and the partner agency?
HSFS also needs to ensure that the mission and goals of C.A.R.E are met, while also
considering the impact of partnership on the potential participant outcome and how collaboration
could benefit participants/program, and at what cost (Karr, 2010).
Some of the indicators to determine the success of a collaboration suggested by the
Public Health Agency of Canada are: determination of the proposed project, benefits of each of
the organizations of entering into the collaborations, timeline and if specific objectives are met
(Public Health Agency of Canada, 2010).
HSFS also needs to evaluate if their collaborations have a positive effect on the
community through the C.A.R.E. program. Factors such as the effectiveness of the program on
the community’s public health, HSFS will need to take into account the effects of advocacy.
This means if advocacy is meeting its objective, reaches the community and is valued by the
community. HSFS also needs to determine participation of community at the local level in
implementing strategies if the strategies are broad enough to reach each aspect of the plan, if
the advocacy is centered on developing community needs and enhancing the participation from
the community in a specific program or to all programs within the scope of C.A.R.E. (Klein &
Adelman, 2005) (Borden & Perkins, 1999).
4.3.3 Research
Given the wide range of knowledge transfer and exchange activities and large
geographical area of Saskatchewan, it is important to use multiple data collection methods. In
20
order to collect process data for measuring indicators from these activities, we suggest the
following methods: surveys/questionnaire, interview, organizational records, activity logs, and
focus groups.
Pre- and post-test questionnaires/surveys and Interviews will be useful for assessing
participants’ capacity to use knowledge and their ability to integrate evidence into decision-
making. Questions that need to be considered include:
1. Are the participants able to apply the gained knowledge to real-life situations and
problems in their own local contexts?
2. Do practitioners use the evidence-based findings to make well informed decisions about
the various programs and projects in which they were involved?
3. Does the activity provide enough opportunities for knowledge exchange?
4. Are there any follow-up actions that participants should do?
5. Do participants have resources that are accessible?
6. How satisfied are the participants?
The questionnaire/survey can be conducted before and immediately after the activity has
taken place to gauge the participant’s attitude and receive feedback. A follow-up in-depth
interview (face-to-face or telephone) can be conducted after 1 month. This gives the participants
time to practice what they have learned. The questions should be open-ended prompts about
the participants’ overall goals as well as specific goals related to heart disease and stroke health
promotion program if any, suggestions for improvement, enthusiasm level and attitude since the
activity, and anecdotal examples of integrating evidence into practice.
When it comes to monitoring the cultural shift and collaboration between researchers
and practitioners, organizational records, activity logs, and focus groups can be used. Some
guiding questions that need to be addressed are:
1. Is there ongoing dialogue between knowledge producers and users?
2. Are researchers open to practitioners’ inputs to ensure that the results of research are
relevant to the current context of heart and stroke health promotion?
21
3. Do researchers and practitioners embrace the shift from their traditional roles to one that
is collaborative, dynamic, and reciprocal?
4. Is there sufficient investment of time in developing relationships?
5. Is there actual implementation of study recommendations?
6. Are there any barriers to knowledge transfer and exchange?
Organizational records and activity are valuable data sources to establish baseline
information. They will be able to provide information such as meeting minutes and notes,
attendance, resource allocation, participation etc. When baseline data has been gathered,
conducting key informant interviews and then a focus group will help determine if the activities
were successful in meeting their objectives. Success should be judged by whether the existence
of knowledge brokers increased exchange between partners, increased decision maker uptake
of new evidence in health promotion, and helped the province accomplish their goals (CIHR,
2006). In addition, if it was found that the multidisciplinary group that assembled for the activities
continued to communicate and meet, it would be a good indicator of success.
4.3.4 Education
The indicators for education tools, as described previously in the report, are extremely
valuable in conducting this process evaluation. To measure them however, we designed a plan
to attain the proper data to evaluate them.
We developed a set of questions pertaining to each indicator to get a better idea of what
information is needed. For example, when looking at the marketing of educational messages,
we found that it would be important to use indicators such as simplicity of media messages,
frequency of media messages and methods of communication. In each of these examples we
identified key questions that needed to be answered:
Are media campaigns simple, attractive and well-placed?
Are advertising methods recognized by members of the community?
Are advertisements run with sufficient frequency?
Are the methods of communication/advertising appropriate for the community given its
resources, size, population, etc.?
22
Are the methods of communication reaching the target population?
Is it a setting where the target population can be effectively reached?
The answers to each of these questions lie in different places which is why we must
consult multiple sources to find the data that we need. In each of these questions there is likely
no existing data meaning that it must be collected directly. For the qualitative data such as the
comprehension, attractiveness and location of media, a survey should be conducted amongst
the target population in the community. To make the survey resources efficient, questions
addressing a variety of indicators should be asked at the same time. Therefore the community
survey should address not only marketing questions, but others as well such as:
Are community members aware of the programs being offered?
Is the target population properly identified and are messages customized for them?
Is there an appropriate number of events /seminars/ workshops being run to address
different demographics in community?
These surveys can be conducted in a written, telephone, or online manner depending on
the resources that are available in the specific community. If staff resources are available to
distribute the written and telephone surveys in the community, then those can be used. If not
however, a central location with additional staff can administer the online survey. Participants
can be selected randomly or written surveys can be distributed at local gathering sites such as
grocery stores, Some basic guidelines to write survey questions state that easier questions
should be placed first, words that will provoke bias or an emotional response should be avoided,
and that similar questions should be placed together in a logical order (Community Tool Box,
2010). Examples of questions that can be asked in one of these surveys are provided in
Appendix II.
With respect to the educational programs operating in the community, it is valuable to
know the dosage or amount that is being completed. Questions we are interested in for this
component of the evaluation include:
Are there an appropriate number of program sessions that a community needs?
o How many?
o How many participants are attending each session?
23
o Is this number in compliance with the acceptable rate?
If attendance is low, how come?
o How many different types of programs in the community are there to address
multiple heart healthy strategies?
These questions can be addressed using data sources that should be available from
program staff. Tracking program logs will determine the number of participants and the number
of sessions that are being run. A threshold number for each is set in advance to determine
whether the program standards are being met. These are helpful pieces of information to gather
for future planning in terms of expansion or contraction of the program (Issel, 2009). We also
need to find out through administrative records and program logs, how many different types of
programs are in the community. This data is useful to know which risk factors are being targeted
and to see if there is a need for other programs to be implemented.
Satisfaction by participants, staff and volunteers is essential to understanding if the goals
and objectives of the programs, marketing strategies, events, seminars or workshops are being
met. The questions to be focused on for these indicators are:
Do program sessions meet their specific goals or objectives?
Is the program what participants were expecting?
How satisfied are participants with the program?
Did participants accomplish the goals of the session and did they increase their
knowledge of heart healthy strategies?
Do participants plan on using this knowledge and applying it to their lives?
Would participants recommend this program to others?
Are there enough training sessions?
Do training sessions develop the ability to teach/run the program in a way that meets the
program objectives?
Collection of this data will need to be done directly by staff or administration for the
program itself. This is done by conducting surveys of staff, participants and volunteers either
randomly or on a voluntary basis. Again, surveys can be written, telephone or online depending
on resources and staff available. Measurement of program satisfaction is extremely important
24
and is often necessary for future funding opportunities. This being the case, care must be taken
to develop specific questions that are going to be asked. Staff, volunteers and participants will
be answering similar questions but surveys should be tailored specifically for each role. If
desired, the Likert rating scale can be used. This involves ranking satisfaction level on a scale
between 1 and 5 with a score of "5" representing "very satisfied" and "1" being " not satisfied"
(Issel, 2009). Qualitatively focused questions such as "Would you recommend this program to
others?" and others listed above, are very beneficial to establish the success of the program,
event, seminar or workshop. Satisfaction surveys should be completed after every session to
ensure that each one is meeting the goals and expectations that were originally set out.
A full summary table of the methods described above, is provided in the Appendix.
5. Data Analysis
To come to a conclusion, it is essential that the data collected are managed, processed
and analyzed. The whole process evaluation and data collection for it is done to achieve
valuable feedback on how to make the programs better and accessible to larger number of
population in the community. Data analysis can be done in many different ways following
various techniques. There are numerous software to make the data analysis easier. In this
section qualitative and quantitative data analysis methods will be discussed for the collected
data. Issel suggests pay special attention to avoid fallacies such as Ecological fallacy during the
data analysis process. Ecological fallacy is “the assumption that a group characteristic applies
to all individuals within that group” (Issel, 2009).
5.1 Analyzing Qualitative Data
The data collected as comments and feedback through evaluation forms, concluding
discussions, open ended survey questions and key informant interviews need to be analyzed in
a qualitative manner. Qualitative approach gives opinion both literally and metaphorically to
stake holders and makes them feel valued (Issel, 2009). There are four elements that need to
be present in qualitative data, which are credibility (internal validity), transferability (external
validity), dependability (reliability) and conformability (findings truly from respondents) (Issel,
25
2009). Researcher should be aware which data collection method requires qualitative analysis
before starting the data collection as it is important to get answers that have all four elements.
Qualitative analysis should be done by a HSF staff who are familiar with all the programs and
workshops being offered. This is essential to ensure the data collected is completely
understood. Quantitative data can be analyzed by an outsider where as this method cannot be.
Three steps to go about for qualitative analysis that Issel suggests are
Overview of analytic process: The step includes
Developing a format and constructing a consistent structure for all collected qualitative
data.
Understanding and interpreting the meaning of what was said by the participants
Coding data (developing codes for certain type of answer)
Generating categories to compare data
Defining the categories (makes easier to separate which data belongs to which
category)
Linking the data
Software: There are a few common computer software used in qualitative analysis such
as NUD*IST, Ethnograph and ATLAS-ti. There are many software programs that can organize
and manage the data; however, staffs still need to be responsible to analyze it.
Issues to consider: Data can get messy and confusing. It is also a tedious task so the
evaluator needs to be patient. It requires trained evaluator(s) and data collector(s) so that the
process runs smoothly. Trained personnel can also ensure unbiased, consistent and reliable
data results (Issel, 2009).
The figure from Dey (1993) demonstrates a diagrammatic format for the steps involved in
qualitative data analysis.
26
Figure: The steps involved in qualitative data analysis.
5.2 Analyzing Quantitative DataNumerical data enables to do quantitative data analysis. It is crucial to do quantitative
analysis so that HSF can compare the results from different programs in a measurable way.
Data collected from all three survey methods needs to be stored in a computer database to
make the analysis faster and more efficient. Microsoft word excel spreadsheet is an example of
computer program that enables the data analyst to organise and analyse the data efficiently.
Alternative software for analyzing data are Statistical Package for the Social Sciences (SPSS)
or Power analysis software. SPSS is widely used to compare the data and to do statistical
analysis of the data. If the resources allow, HSF should consider hiring a graduate student to do
the data analysis using SPSS.
The software that HSF wishes to use for data analysis should be decided before the
data entry begins (Issel, 2009). Quantitative analysis result in findings such as the participant’s
rate, which HSF site is most visited, how many programs are useful and attended, how many
more programs to offer etc. The data needs to be organized, correct and up to date. Errors in
data entry process can lead to false conclusion and more difficulties. The researcher should
check for outliers, which are the values that are outside of the normal range (Issel, 2009).
27
Outliers should be discarded because they prevent from giving a representative result. Deciding
on removing or keeping the outliers can also be done on a case by case basis.
It is helpful to have the research plan in front of you while doing the analysis; it acts as a
guide and reminds of the goal. Here is an example of a research process to show where the
quantitative data falls (Bryman & Duncan, 1990).
Figure: Showing where data analysis falls in the research process.
28
6. Evaluation Resources
6.1 BudgetThough each individual program within C.A.R.E. will have their own specific budget,
variance between communities of similar sizes should be compared on an ongoing basis. This
will determine the extent how well each communities C.A.R.E. programs are performing with the
projected program expenditures. Also, issues such as locations for programs and variations in
schedules may also impact budget as well as effectiveness of programs budget variance may
be important in distinguishing differences between communities. Sufficient variance is an alert
that the programs are not being delivered as planned. Outside salary of staff members contract
to carry out the interviews and focus groups for evaluation, there are no budgetary constraints
(Issel, 2009).
6.2 ResourcesWhen implementing the C.A.R.E. evaluation HSFS must realize that the staff, volunteers
and there quality and training are its most valuable asset. HSFS staff and volunteers have a
history of commitment to the reducing the effect of heart and stroke related-disease. The
magnitude of the C.A.R.E evaluation will call for an expanded volunteer and staff base to
adequately provide viable results. Due to the fact that the programs are province wide, with
each target community varying in size HSFS community action coordinators will have to be
vigilant in order to ensure C.A.R.E. is being implemented efficiently, effectively and with a level
of efficacy. To evaluate C.A.R.E a number of focus groups and key informant interviews will be
undertaken, these will require committed staff members that are trained with competency,
knowledge, and skills accompanied with the correct attitude to carry out and mediate
discussions to acquire appropriate data for the evaluations success. These, staff members and
the data received are essential in verify the programs sustainability. C.A.R.E.’s wide reach
throughout the province would call for varying numbers of staff, small communities under
10,000residents (such as Lloydminster, Kindersley, Estevan, etc.), will only require one staff
member to conduct interviews and mediate discussions. Whereas, larger communities up to
35,000 residents (such as Prince Albert, Moose Jaw, the Battlefords, etc.) will require three staff
29
members. Metropolitan centres such as Saskatoon and Regina will require five trained staff
members (Issel, 2009).
The majority of data on C.A.R.E will be received from surveys and questionnaires which
can largely be collected by volunteers. The range of volunteers will also vary depending on
community size. However, many of these surveys and questionnaires can be delivered during
the programs themselves, or via existing events and fundraisers put on by HSFS. By using
existing events such as the Saskatoon dragon boat festival, Hoops for hearts and fundraisers
like SaskEnergy Big Bike a C.A.R.E questionnaire can be attached to the participant
registration. Though this will not be a representative sample; however, the data will still be
relevant. Annual events such as the door to door fundraising campaign in February will allow a
vehicle to distribute a short survey and will be more representative of the provincial population.
Moreover, another vehicle for data collection would be jump rope for heart where numerous
schools across the province participate, with the participating children a survey can be sent
home for the parents. Using existing programs and events will allow for data to be collected
using the existing pool of volunteers. Not only will this reduce the number of volunteers needed
but using existing programs will allow HSFS to engage all relevant stakeholders in the
evaluation process (Issel, 2009).
All the data collected (primary and secondary) will need to be analyzed by a consultant
as the qualitative data will need to be transcribed and the quantitative data subjected to
statistical analysis. The HSFS Memorandum of Understanding with University of Saskatchewan
may allow for the analysis to be done by volunteer students, in an effort to further the
partnership between the University of Saskatchewan and HSFS (Issel, 2009).
7. Conclusion
7.1 SummaryThe proposed process evaluation for the HSFS has been developed by specifically
applying concepts of evaluation to the C.A.R.E. health promotion program. The process
evaluation will allow HSFS to see how the C.A.R.E. programs are performing, a basis for
30
explanation of differences between communities, and areas that can be improved. To gain the
most accurate results a baseline of data should be collected before or at the beginning of the
programs and be periodically compared to data collected throughout the life of the programs.
The analysis of the data will show if the programs are performing as expected, as well as the
satisfaction of all stakeholders involved, and highlight disparities between communities. The
collection of this data and its analysis is vital to the success of the evaluation. C.A.R.E. has the
potential to change many lives and reduce the public burden of heart and stroke related-disease
on Saskatchewan, when it is implemented and its programs are used to their full potential.
7.2 RecommendationsDuring program development organization resources have been identified that are
integral in implementing C.A.R.E. but to insure the success and sustainability of the programs
HSFS must use the data to hold the programs accountable. In order to maintain accountability,
baselines or preliminary data needs to be collected to evaluate the initial opinions and statistics
before program implementation, thus to truly see the effect of C.A.R.E. In an effort to see
positive changes and evaluate if the programs are doing their intended purpose periodic
evaluations must be done. The data from these periodic evaluations will allow HSFS to see if
the programs need to be changed or if certain programs are excelling, and why. From the
obtained information reports must be formally made to HSFS to be used in evaluations and
program implementation. The regular reports will also allow a format for information to be
passed to stakeholders, volunteers, and members of the community; which is key for continued
support and the sustainable success of C.A.R.E. (Issel, 2009).
31
Appendix A: Summary Tables
Appendix B
Appendix C
References
Bernstein, E., & Wallerstein, N. (1988). Empowerment education: Freire's ideas adapted to health education. Health Educ Behav; 15: 379.
32
Borden, L.M. & Perkins. D.F (1999). Assessing Your Collaboration: A Self Evaluation Tool. Journal of Extension. 37(2).
Bryman. A. & Duncan, C. (1990). Quantitative Data Analysis: For Social Scientist. New York: Routledge.
Dillman, D. A. (1978). Mail and telephone surveys: The total design method. New York: John Wiley and Sons.
Ebbesen, L. S., Woodard, G. B., McLean, S., Butler-Jones, D., Green, K., Reeder, B. A., Steer, S., & Feather, J. (2001). The saskatchewan dissemination story. Promotion & Education, Suppl 1, 35-39.
Elder, J.P., Schmid, T.L., Dower, P. & Hedlund, S. (1993). Community Heart Health Programs: Components, Rationale, and Strategies for Effective Interventions. Journal of Public Health Policy, 14(4): 463-479.
Haalboom, B. J., Robinson, K. L., Elliott, S. J., Cameron, R., & Eyles, J. D. (2006). Research as intervention in heart promotion. Canadian Journal of Public Health, 97(4), 291-295.
Heart and Stroke Foundation of Canada. (2009). Knowledge transfer and exchange in research: A guide for applicants and reviewers Canada:
Issel, L. M. (2008). Health program planning and evaluation: a practical and systematic approach for community health. 2nd ed. Toronto: Jones and Bartlet.
Klein, G.L. & Adelman, L. (2005). A Collaboration Evaluation Framework. Submitted to: International Conference on Intelligence Analysis. Retrieved from: http://abbyxmix.com/abby/ATK%20Collab%20Framework.pdf
Kouzes, J., & Posner, B. (2005). The leadership challenge. San Francisco, CA: John Wiley & Sons Ltd.
McKillip, J. (1987). Need analysis: Tools for the human services and education. Newbury Park, CA: Sage.
Neudorf, C. (2008). Sources of health data and health status monitoring (Powerpoint slides). Saskatoon Health Region.
Riley, B. L., Stachenko, S., Wilson, E., Harvey, D., Cameron, R., Farquharson, J., Donovan, C., & Taylor, G. (2009). Can the Canadian heart health initiative inform the population health intervention research initiative for Canada? Canadian Journal of Public Health, 100(1)
Sperling, J., Von Sychowski, S., & Zarinpoush, F. (2007). Effective knowledge transfer & exchange for nonprofit organizations: A framework Canada: Imagine Canada.
Schwartz, R., Smith, C., Speers, M.A., Dusenbury, L. J., Bright, F., Hedlund, S., Wheeler, F., Schmid, T.L. (1993). Capacity Building and Resource Needs of State Health Agencies to
33
Implement Community-Based Cardiovascular Disease Programs. Journal of Public Health Policy. 14(4):480-494.
Zapka, J., Goins, K. V., Pbert, L., & Ockene, J. K. (2004). Translating efficacy research to effectiveness studies in practice: Lessons from research to promote smoking cessation in community health centers. Health Promotion Practice, 5(3), 245-255.
34