Post on 18-Jul-2020
61The Canadian Journal of Program Evaluation Vol. 26 No. 2 Pages 61–88ISSN 0834-1516 Copyright © 2012 Canadian Evaluation Society
Corresponding author: Rich Janzen, Centre for Community Based Research, 73 King St. W, Kitchener, ON, Canada N2G 1A7; <rich@communitybasedresearch.ca>
USiNG EvAlUAtiON tO ShApE ANd diRECt COmpREhENSivE COmmUNity iNitiAtivES: EvAlUAtiON, REflECtivE pRACtiCE, ANd iNtERvENtiONS dEAliNG With COmplExity
Rich JanzenCentre for Community Based Research and Wilfrid laurier University
daniela Seskar-hencicRegional municipality of Waterloo
yasir dildarCentre for Community Based Research, Kitchener, Ontario
peter mcfaddenWaterloo Region immigrant Employment Network
Abstract: the purpose of this article is to discuss how evaluation can be used to shape and direct Comprehensive Community initia-tives in an ongoing way. it does so by offering a case example of the Waterloo Region immigrant Employment Network (WR-iEN). the article begins by reviewing how participatory action research can encourage reflective practice. After the WRiEN case example is presented, we consider this example in light of collaborative evaluation literature and five facilitators of reflec-tive practice: (a) be location-based, (b) value experiential and practical knowledge, (c) provide ongoing feedback, (d) facilitate democratic dialogue, and (e) focus on a vision for the common good. the article ends by discussing contributions to the broader evaluation knowledge base, particularly developmental evalua-tion.
Résumé : le but de cet article est de discuter comment l’évaluation peut servir à former et diriger d’une façon continue les initiatives communautaires intégrées. On vous offre, par exemple, le cas du réseau d’emplois des immigrants à la région de Waterloo (WRiEN). l’article commence par examiner comment la recher-che-action participative peut encourager la pratique réflexive. Après la présentation du cas d’exemple du WRiEN, on considère cet exemple par rapport à la littérature sur l’évaluation collabo-rative et cinq facilitateurs de la pratique réflexive : (a) se baser sur la localisation, (b) valoriser les connaissances expérientielles
62 The Canadian Journal of Program evaluaTion62
et pratiques, (c) fournir la rétroaction continue, (d) faciliter le dialogue démocratique, et (e) concentrer sur une vision de l’inté-rêt commun. Cet article se termine en discutant les contributions à la base plus large de connaissances en évaluation, en particu-lier, l’évaluation développementale.
Since the 1990s, Comprehensive Community initiatives (CCis) have emerged as a common means of addressing social con-cerns within local communities. CCis are guided by the principles of comprehensiveness and community-building (Kubisch, fulbright-Anderson, & Connell, 1998). that is, CCis are designed to address social problems holistically, at multiple levels, and with a view to pro-moting sustainable systems change (comprehensiveness), which is best achieved through the broad-based, cross-sectoral, and equitable engagement of citizens who act as the architects of their own solu-tions (community-building) (Baker, 2003; foster-fishman, Nowell, & yang, 2007). the underlying assumption is that significant com-munity improvements will not occur unless the surrounding system adjusts to accommodate the desired goals. As such, CCis not only work for changes for individuals and groups, but also seek to improve the conditions in which they are more likely to thrive (Baum, 2001; Natasi & hitchcock, 2009). this implies that community stakehold-ers (people who have a stake in the issue under consideration) are challenged to create new and different ways of working together (foster-fishman et al., 2007). it also means that CCis are designed to evolve over time so as to respond to ever-changing community needs and priorities (lafferty & mahoney, 2003).
Given increasing demands for accountability on investment, there are growing expectations that CCis be evaluated. Evaluation is seen to be important because CCis (a) benefit from the support of a wide range of constituents who need to be kept informed of progress and outcomes; (b) need to generate useful feedback to guide action within a dynamic context; and (c) hold the opportunity to contribute to the broader, longer-term “social learning” that can be transferred to other community settings (Kubisch et al., 1998). it has been widely recognized that it is difficult to evaluate CCis (foster-fishman & long, 2009; Natasi & hitchcock, 2009). With summative evaluations the challenge lies in measuring and analyzing broad multisectoral outcomes within a complex socio-political context where the inter-vention is only one factor (Kubisch et al., 1998). however, and more central to this article, formative evaluation of CCis can also be chal-lenging. formative evaluations are conducted as the intervention unfolds for the purpose of learning, shaping, and directing it in an
6363la revue Canadienne d’évaluaTion de Programme
ongoing way (Shadish, Cook, & leviton, 1991). here the challenge lies in the dynamic nature of CCis, whose flexible and developmental nature in response to shifting local dynamics adds ambiguity and uncertainty to evaluation design and implementation (Baum, 2001).
A program theory approach to evaluation is typically seen as a means of beginning to address these challenges (lafferty & mahoney, 2003). A program theory attempts to explain how an intervention is sup-posed to work, rationally linking its resources and activities with its short- and long-term intended outcomes (Chen, 2005). A program theory approach can also be useful in sharpening program plan-ning—isolating those efforts that lead to desired outcomes and those that do not (mackenzie, & Blamey, 2005). Nevertheless, a CCi’s the-ory of change is not necessarily unified or stable, but rather evolves over time in response to complex and fluid environments (Baum, 2001; lafferty & mahoney, 2003). Evaluations of CCis therefore require mechanisms to enable ongoing reflective practice—mecha-nisms that enable stakeholders to collectively reflect on what they have done (practice) and what they have learned about what was effective (theory), all for the sake of adapting and informing their future practice and deepening their program theory (Natasi & hitch-cock, 2009). Such mechanisms of reflective practice are particularly needed within evaluations of innovative interventions (as CCis often are), where previous models of practice are lacking and where con-texts are complex and fluid (as is often the case with CCis). patton (2011) recently coined the term “developmental evaluation” to dis-tinguish evaluations of these complex systems-change interventions from traditional formative evaluations that seek to improve fixed-models of practice in more stable environments.
practical examples of evaluations of CCis that use a theory of change approach are beginning to emerge in the evaluation literature. how-ever, these typically involve summative evaluations that report on outcomes and related methodological challenges (e.g., foster-fish-man & long, 2009; mackenzie & Blamey, 2005). there are limited examples demonstrating how evaluation can be used to shape and direct CCi practice in an ongoing way. the purpose of this article is to speak to this gap by discussing the processes and mechanisms of how evaluation can be an avenue to spur reflective practice within CCis. it does so by offering a case example of an evaluation of one CCi (the Waterloo Region immigrant Employment Network—WRiEN) that adopted a participatory action research approach. the article begins by first reviewing the broader literature of how a participatory ac-tion research approach can facilitate reflective practice. following
64 The Canadian Journal of Program evaluaTion64
the case example, we bring evaluation literature into the conversa-tion and link WRiEN’s reflective practice to this literature. in this way, the article bridges the gap between two long-standing bodies of literature (collaborative evaluative inquiry and participatory ac-tion research) that often remain in parallel. We conclude by briefly discussing how this article contributes to the broader evaluation knowledge base, including that of developmental evaluation. We note how the case study illuminates that the formative-developmental distinction may be less categorical than patton (2011) presents, but can rather be seen as a continuum in how loosely or tightly a par-ticular program theory is held.
pARtiCipAtORy ACtiON RESEARCh ANd REflECtivE pRACtiCE
As one type within the broader typology of research, evaluation can draw on the growing literature that discusses how research can stimulate reflective practice. Such approaches to research operate under a variety of names, including action research (e.g., Stringer, 2007), community-based research (e.g., travers et al., 2008), commu-nity-based participatory research (e.g., israel, Eng, Schulz, & parker, 2005; minkler & Wallertsein, 2008), or—the preferred term in this article—participatory action research (e.g., Kemmis & mctaggart, 2005). Regardless of label, this approach fuses the “northern” utiliza-tion-focused action research tradition initiated by Kurt lewin (1948), with the “southern” emancipatory participatory tradition initiated by paulo friere (1970) (for fuller discussions of these historical roots, see Cargo & mercer, 2008; Wallerstein & duran, 2003). Common to these traditions is a commitment to interweave theory and practice within a research process in a way that leads to knowledge-driven collective action. indeed, “learning as we go” is a key value of partici-patory action research (Nelson, Ochocka, Griffen, & lord, 1998)—a value fitting well with the evolving and responsive nature of CCis.
Within participatory action research, theory and practice are inter-connected through a process known as the reflective action cycle. the cycle generally includes some combination of planning, action, and reflection in successive spirals over time (lewin, 1948; Stringer, 2007; Wallerstein & duran, 2003). Within the process of research (including evaluation), the reflective action cycle could be conceived as four nonlinear and repeated phases (laying the foundations, plan-ning, information gathering/analysis, and acting on findings) that are ever-attuned and adaptive to an emerging context and ongoing learning (see figure 1). While we elaborate on these phases in more
6565la revue Canadienne d’évaluaTion de Programme
detail elsewhere (Janzen, hatzipantelis, vinograd, & Kellerman, 2004; taylor & Botschner, 1998), it is worth highlighting here that the phases emphasize not only traditional technical elements of research, but also foreground the relational aspects of collaborative research. they do so because of a belief that a collaborative process of inquiry is as important as the outcomes or findings of the research (Reason, 2006). Research produces not only a vision for future collec-tive action, but also builds a sense of community and inspires people to work together toward a common goal (Stringer, 2007).
Figure 1The Four Phases of Participatory Action Research
Adapted from CCBR, 1998; 2004.
the broader research literature also provides insights into how reflec-tive practice can be facilitated through research. Below we outline five facilitators that we gleaned from the literature and that represent our theoretical contribution to the participatory action research dis-course. While we do not presume these facilitators to be exhaustive, they do provide a means of connecting participatory action research to the knowledge base of evaluation (discussed later in the article).
66 The Canadian Journal of Program evaluaTion66
Be location-Based
the first facilitator of reflective practice is to focus the scope of research to a particular community setting. Being location-based requires a deep ecological understanding of the local context (New-brough, 1995). it also implies that design changes to respond to a context, which is dynamic in nature (Reason, 2006). this is because the emphasis is on solving local problems for a specific group of people—local theory for local practice (Stringer, 2007). Reflecting on local practice is pre-eminent, as generalizing theory to benefit other settings is of secondary importance. Within the evaluation of CCis, being location-based implies a clear bounding of what constitutes the intervention under study and what constitutes external context (foster-fishman et al., 2007). it also implies that depictions of a pro-gram’s theory of change, such as program logic models (mclaughlin & Jordan, 2004), are not viewed statically but change and evolve in response to the deepened understanding of what kind of intervention is needed with a specific location.
value Experiential and practical Knowledge
traditional research has privileged knowledge that is produced by synthesizing data contributed by others, typically experts. Such a view keeps theory and practice separate. But knowledge can also be understood through the critical reflection of personal and collective experiences, whether these experiences are recent (Clare, 2006) or historical (fals Borda, 1987). in addition, knowledge can emerge as people do something in a critically reflective way. heron and Reason (1997) call these two other ways of knowing “experiential” and “prac-tical” respectively. valuing experiential and practical knowledge as-sumes that people can create new understanding that is grounded in their social involvements, which in turn creates a better informed practice that is guided by newfound insights (israel, Schulz, parker, & Becker, 1998).Evaluation of CCis can value such knowledge by trian-gulating the opinions of all stakeholders, encouraging them to critical-ly reflect on both their experience of, and their efforts within, the CCi.
provide Ongoing feedback
to ensure that the experiential and practical ways of ‘knowing’ de-scribed above are a collective exercise, research should include cycles of feedback and exchange among people (heron & Reason, 1997). Such co-operative inquiry implies that communities involved in re-
6767la revue Canadienne d’évaluaTion de Programme
search own their knowledge (fals Borda, 1987). Community members are therefore in the best position to link what they collectively know to what they collectively do. Evaluation of CCis can incorporate regu-lar feedback of findings into the evaluation and intervention design, using multiple formats and forums that speak most clearly to respec-tive stakeholders (Chavez, duran, Baker, Avila, & Wallerstein, 2008).
facilitate democratic dialogue
following the critical theorist Jurgen habermas, Kemmis and mctag-gart (2005) call for researchers to create “communicative spaces.” Research participants are not seen as forced subjects of a research process or passive recipients of its findings. Rather, participants free-ly interact within research and with the knowledge it produces in or-der to reach inter-subjective agreement, mutual understanding, and unforced consensus. they are engaged in dialogue because they are encouraged to speak about what is important to them in ways that promote liberty, fraternity, and equality (Newbrough, 1995). Such a view implies that researchers are not only technicians, but also adopt the role of facilitator (Conde-frazier, 2006). Researchers become col-laborators rather than controllers, and research quality rests on the ability of researchers to stimulate, use, and interpret open discussion (heron & Reason, 1997). Such democratic dialogue provides the re-flective space in which theory and practice can be brought together. Evaluation of CCis can nurture this reflective space by incorporating methods that encourage interaction (e.g., focus groups and commu-nity forums), and by using feedback to facilitate further discussion of what existing findings mean for future action.
focus on a vision for the Common Good
A final facilitator for reflective practice is carrying out the research collaboration in such a way that people gain the collective capacity to imagine how the circumstances of their lives could be improved (Kemmis & mctaggart, 2005). indeed, the research process itself could be conceived of as a negotiation among stakeholders about the type of future they wish to co-create. if such a vision-focus is fostered within a research process, participants will gain licence to reflec-tively link their collective knowledge with their desired collective action. Within CCis there are often diverse stakeholder perspectives and interests, which are often in competition and at odds. the re-search process can act as a forum in which diverse perspectives and interests are aired and negotiated in a transparent and systematic way (Ochocka, moorlag, & Janzen, 2010).
68 The Canadian Journal of Program evaluaTion68
thE WRiEN CASE ExAmplE
description of Waterloo Region immigrant Employment Network
the Waterloo Region immigrant Employment Network (WRiEN—pronounced “Ryan”) was launched in 2006 in Waterloo Region, a community of half a million in southern Ontario (see www.wrien.com). WRiEN was created to act as a local strategy to help ensure that the skills of immigrants would be more optimally used to the benefit of immigrants and their families, the region’s economy, and the community as a whole. it was developed in response to the well-documented under-utilization of immigrant skills across Canada (Al-boim, 2009; Wayland, 2006), including considerably higher rates of unemployment and under-employment, as well as lower earnings of immigrants in Waterloo Region relative to Canadian-born—this de-spite the higher education levels of immigrants (miedema & vande-belt, 2006). WRiEN was a comprehensive community initiative that provided a forum to explore, develop, and test innovative system change responses to immigrant employment (comprehensive). it did this through a broad-based and cross-sectoral engagement of com-munity stakeholders who identified their own solutions and priorities (community-building) (Janzen,Walton-Roberts, & Ochocka, in press).
WRiEN emerged as a result of extensive community consultation. led by the Centre for Community Based Research (CCBR), diverse community stakeholders were brought together to identify local needs, resources, and strategies in facilitating immigrants’ access to employment. Stakeholder groups included immigrants, business, local government, community organizations, education, and non-government funders, many of which had not been previously engaged on this issue. WRiEN was launched following an immigrant Skills Summit at which leaders from all stakeholder groups prioritized and committed to a comprehensive set of community action strate-gies (Janzen, hatzipantelis, & hogarth, 2005). WRiEN subsequently functioned as a body that convened and facilitated the collective ac-tion of these various local stakeholders in implementing the identi-fied strategies. WRiEN was among the first immigrant employment comprehensive community initiatives to be developed in a mid-size urban centre in Canada, if not beyond (mcfadden & Janzen, 2007).
Winning awards from both nonprofit and private sectors, WRiEN’s innovation lay in leveraging diverse community resources toward the common goal of immigrant employment, with few implementation models to draw on. WRiEN was hosted by the Greater Kitchener-
6969la revue Canadienne d’évaluaTion de Programme
Waterloo Chamber of Commerce and consisted of a multi-stakeholder steering committee (privileging the immigrant and business perspec-tives), four active work groups (investments, employment initiatives, immigrant support, and qualification recognition and enhancement) and was supported by four staff. WRiEN’s structure was intended to be non-hierarchical with an emphasis on involving all community stakeholders. Core WRiEN funding was provided by local govern-ment and non-government funders. WRiEN was also successful in securing additional project funding to support work group priorities.
it is important to emphasize that WRiEN was not a direct service immigrant employment program. that is to say, WRiEN was not de-signed to directly match individual immigrants with employers (the region has numerous such programs). instead, it was a broad-based network designed to create the community conditions that would make successful immigrant employment more likely (i.e., systems change). it did this by facilitating diverse segments of the local com-munity to work together in new ways. for example, WRiEN emerged as a region-wide voice in advocating to senior levels of government on an array of immigrant employment issues. WRiEN was also in-strumental in launching and supporting a series of community initia-tives: an immigrant mentorship program, an immigrant internship program, an immigrant loan program (to support further education), an immigrant web portal, and an employer connections program. in all these examples WRiEN itself was not the lead organization but was the convener and catalyst in securing the necessary partners and resources to create and deliver “made-in-Waterloo” solutions (Wayland, 2007).
description of the WRiEN Evaluation
An ongoing process and outcome evaluation was built into WRiEN’s initial three-year mandate. Evaluation was intended to determine how and to what extent WRiEN’s multi-stakeholder and region-wide efforts were impacting the lives of immigrants, the local economy, and the broader community of Waterloo Region. the main research questions related to the structure of WRiEN, the implementation of its activities, outcomes observed to date, lessons learned, and recom-mendations for the future. A research team from the CCBR led the evaluation under the guidance of a cross-stakeholder steering group.
the evaluation used a mixed method design (tashakkori & teddlie, 2003) carried out in annual cycles over the three-year evaluation
70 The Canadian Journal of Program evaluaTion70
period. methods that were repeated each year and that considered multiple stakeholder perspectives included key informant interviews (n = 50); focus group interviews (n = 52 in seven groups); participant observation of the WRiEN steering committee, work group meetings, and WRiEN-sponsored community events (total meetings/events = 25; almost 40% of all meetings/events); brief meeting tracking forms completed by participants at the end of each steering committee and work group meeting (n = 263); and an internal and external docu-ment review. in addition, an employers’ web survey (for private, pub-lic, and nonprofit sectors) was conducted in the first and last years of the study (n = 264).
the WRiEN evaluation adopted a participatory action research ap-proach that we defined as a “research approach that involves active participation of stakeholders, those whose lives are affected by the issue being studied, in all phases of research for the purpose of producing useful results to make positive changes” (Nelson et al., 1998, p. 9). three primary mechanisms were used to implement this approach. first, the CCBR evaluation team comprised members bringing content expertise (in immigrant employment and systems change evaluation), experiential knowledge (recent immigrants who had subsequently been unemployed or underemployed), and a com-mitment to advancing immigrant employment as an issue of so-cial justice. Second, a seven-member evaluation committee guided each step of the evaluation process. Committee members gave input into all evaluation tools, sampling criteria, participant recruitment, analysis, interpretation, and the dissemination of findings. mem-bers reflected WRiEN’s diverse stakeholder groups and had varying degrees of involvement within WRiEN. third, the evaluation was characterized by frequent feedback in multiple formats. in addition to the annual evaluation reports, feedback was provided via evalua-tion e-bulletins, summaries and presentations of individual methods, findings posted on the evaluation corner of the WRiEN website, and evaluation updates at all WRiEN steering committee meetings. in addition, a community forum was held each year at which time the year’s annual evaluation findings were reported.
the evaluation repeated the action reflection cycle (figure 1) three times (i.e., one cycle for each of the three years). following each year, the evaluation committee reconsidered how to adapt the evaluation process based on emergent learning. the research questions and methods remained relatively constant from year to year with minor changes. more pronounced changes were made in how the evaluation findings were shared from year to year, and how this subsequently
7171la revue Canadienne d’évaluaTion de Programme
informed new practice within WRiEN (i.e., phase 4 of the action reflection cycle). in addition, the program’s theory of change was reviewed each year using an evolving logic model technique. the baseline and year 3 logic models are found in figures 2 and 3 respec-tively. these logic models served as reflective tools to consider what had been accomplished over the previous year, and looked forward to the emergent theory of what should be done in the upcoming year.
Evaluation findings highlighted the strength of WRiEN’s collabora-tive approach, but also pointed out limitations to date (dildar & Jan-zen, 2009; hogarth, Janzen, & Cushing, 2007; Janzen & dildar, 2008). WRiEN had increased community awareness about the issue of im-migrant employment and was successful in leveraging additional fi-nancial resources from senior levels of government. WRiEN was also successful in creating an increased level of trust among its partners that led to the formation of a series of new interventions dealing with specific aspects of immigrant employment (e.g., internship, mentor-ship, loan programs). however, further community engagement was needed, particularly among employers and recent immigrants, as was more attention on a rapid advocacy response and on marketing. Another significant constraint of WRiEN was its narrow focus on skilled professionals to the exclusion of unskilled workers.
liNKiNG WRiEN’S REflECtivE pRACtiCE tO thEORy
What did we learn about the role of evaluation in helping to shape and direct the evolving WRiEN intervention? Below we use the five facilitators of reflective practice that we previously identified as a frame to discuss our learnings. We also link our learnings to the evaluation literature, particularly that which has appropriated or adapted collaborative modes of inquiry. this is done to demonstrate how previous evaluation discourse has contained elements of partici-patory action research.
Be location-Based
the WRiEN initiative was situated in a community that had a long history of addressing immigrant employment issues. the CCBR and several service providers had provided nearly a decade of leadership on various research and community development projects prior to launching WRiEN (Centre for Community Based Research, 2010). those efforts included (a) research projects that analyzed the local employment and support needs and capacities of immigrants, (b) en-
72 The Canadian Journal of Program evaluaTion72Fi
gure
2W
RIE
N B
ase
Line
Log
ic M
odel
(200
6)
Wor
k G
roup
s (V
ertic
al)
Sam
ple
Prio
rity
Activ
ities
Mai
n C
ompo
nent
s
-Com
pile
and
di
strib
ute
info
rmat
ion
on
imm
igra
nt
empl
oym
ent
issu
es lo
cally
an
d na
tiona
lly
Info
rmat
ion
Cle
arin
g-ho
use
Stee
ring
Com
mitt
ee
(Hor
izon
tal)
Activ
ity
Area
s C
omm
uni-
catio
ns
Adv
ocac
y
Qua
lific
atio
n R
ecog
nitio
n/E
nhan
ce-
men
t
Imm
igra
nt
Sup
port
Imm
igra
nt
Attr
actio
n In
vest
men
ts
Net
wor
k C
oord
inat
ion
Com
mun
ity
Con
sulta
tion
Em
ploy
er
Initi
ativ
es
Dire
ct
Out
com
es
Ove
rarc
hing
O
utco
mes
Long
Ter
m
Visi
onar
y O
utco
mes
-Dev
elop
m
arke
ting
stra
tegy
for
entir
e ne
twor
k
-Con
duct
re
gion
-wid
e ad
voca
cy th
at
is in
form
ed b
y w
ork
grou
ps
and
com
mun
ity
cons
ulta
tions
-Det
erm
ine
linka
ges
and
enco
urag
e flo
w o
f in
form
atio
n ac
ross
the
netw
ork
-Pro
vide
fo
rum
for
com
mun
ity-
wid
e in
put i
nto
WR
IEN
pr
iorit
ies
-Ens
ure
finan
cial
vi
abili
ty o
f W
RIE
N
oper
atio
ns
-Attr
act $
from
se
nior
leve
ls
of g
ov’t
-Mar
ketin
g ca
mpa
ign
-Cas
e st
udie
s -E
mpl
oyer
re
sour
ce
guid
es
-Net
wor
king
op
portu
nitie
s
-Adv
ocat
e/
coor
dina
te
brid
ging
pr
ogra
ms
-Pro
mot
e ac
cess
to
finan
cial
re
sour
ces
-Co-
advo
cate
at
all
leve
ls
-Pro
vide
in
form
atio
n to
im
mig
rant
s -E
ncou
rage
or
gani
zatio
nal
impr
ovem
ent
Bra
nd/m
arke
t W
ater
loo
to
imm
igra
nts
-D
eplo
y w
ebsi
te
-Gat
her
data
em
ploy
men
t re
sour
ces
Mor
e st
rate
gic
recr
uitm
ent
and
reta
inm
ent o
f im
mig
rant
sk
ills
Bet
ter e
quip
ping
of
imm
igra
nts
in
over
com
ing
syst
emic
em
ploy
men
t ba
rrier
s
Incr
ease
d re
sour
ces
leve
red
and
mor
e ef
ficie
nt
allo
catio
ns
Mor
e st
rate
gic
orie
ntat
ion
and
supp
ort o
f em
ploy
ers
in
hirin
g im
mig
rant
s an
d su
cces
sful
ly
inte
grat
ing
them
in
to th
e w
orkp
lace
Bet
ter
reco
gniti
on a
nd
enha
ncem
ent
of th
e pr
ior
lear
ning
and
cr
eden
tials
of
imm
igra
nts
Impr
oved
co
llabo
ratio
n,
partn
ersh
ips
and
syne
rgiz
es
amon
g st
akeh
olde
rs
Impr
oved
regi
on-
wid
e sy
stem
ic
advo
cacy
and
be
tter l
obby
ing
of
seni
or
gove
rnm
ents
re:
po
licy
chan
ges
Incr
ease
d aw
aren
ess
of
imm
igra
nt
empl
oym
ent
issu
es a
nd
impr
oved
acc
ess
to re
leva
nt in
fo
Mor
e im
mig
rant
-fri
endl
y w
orkp
lace
s
Mor
e in
clus
ive
hirin
g pr
actic
es/s
trate
gies
am
ong
empl
oyer
s an
d re
vers
ed tr
end
of
imm
igra
nt e
mpl
oym
ent
barri
ers
Incr
ease
d lik
elih
ood
of
empl
oyer
s w
ith v
acan
cies
hi
ring
imm
igra
nts/
im
mig
rant
s ar
e m
ore
visi
ble
and
succ
essf
ul w
ithin
re
crui
tmen
t and
sel
ectio
n pr
oces
ses
Impr
oved
sen
se
of b
elon
ging
w
ithin
Wat
erlo
o R
egio
n by
im
mig
rant
s, a
nd
imm
igra
nts
mor
e lik
ely
to s
tay
Mor
e an
d m
ore
effe
ctiv
e im
mig
rant
su
ppor
ts, w
ith
decr
ease
d ga
ps/o
verla
ps
Gre
ater
coh
eren
cy
and
effe
ctiv
enes
s of
re
gion
al/p
rovi
ncia
l/ fe
dera
l im
mig
rant
em
ploy
men
t pol
icy
Incr
ease
d #
of
empl
oyer
s an
d go
vern
men
t/ co
mm
unity
age
ncie
s se
tting
ann
ual
prog
ram
pla
ns a
nd
allo
tting
fund
s In
crea
sed
num
ber o
f im
mig
rant
s an
d em
ploy
ers
acce
ssin
g av
aila
ble
supp
orts
Incr
ease
d m
arke
tabi
lity
of
imm
igra
nt s
kills
Bet
ter
prio
ritiz
atio
n an
d be
tter
impl
emen
tatio
n of
a
com
preh
ensi
ve,
regi
on-w
ide
plan
Em
ploy
ers
mor
e aw
are
of
imm
igra
nt
empl
oym
ent
issu
es
Impr
oved
at
tract
ion
of
imm
igra
nt ta
lent
re
lativ
e to
labo
ur
mar
ket t
rend
s
Impr
oved
im
mig
rant
wel
l-be
ing
Stro
ng L
ives
…
Stro
ng E
cono
my…
St
rong
Com
mun
ities
…
Incr
ease
d re
gion
al
econ
omic
pr
ospe
rity
Hea
lthie
r, m
ore
vibr
ant a
nd m
ore
incl
usiv
e co
mm
uniti
es
7373la revue Canadienne d’évaluaTion de Programme
Figu
re 3
WR
IEN
Yea
r 3 L
ogic
Mod
el (2
009)
Prio
rity
Activ
ities
Mai
n C
ompo
nent
s
Activ
ity
Area
s
Foun
datio
nal
Cha
nge
Tang
ible
C
omm
unity
C
hang
e
Long
Ter
m
Cha
nge
Wor
k G
roup
s (W
ithin
issu
e)
-Com
pile
and
di
strib
ute
imm
igra
nt
empl
oym
ent
libra
ry
-Reg
ular
em
ail
upda
tes
-Dis
tribu
te
“Tod
ay’s
W
orkf
orce
” e-
bulle
tin
-Web
de
velo
pmen
t
Info
rmat
ion
Cle
arin
g-ho
use
Ste
erin
g C
omm
ittee
(C
ross
Net
wor
k) C
omm
uni-
catio
ns a
nd
Mar
ketin
g A
dvoc
acy
Qua
lific
atio
n R
ecog
nitio
n/E
nhan
ce-m
ent
Imm
igra
nt
Supp
ort
Inve
stm
ents
C
omm
unity
Pl
anni
ng
Com
mun
ity
Con
sulta
tion
and
Enga
gem
ent
Empl
oym
ent
Initi
ativ
es
-Dev
elop
a
com
preh
ensi
ve
mar
ketin
g /
com
mun
icat
ion
stra
tegy
for e
ntire
ne
twor
k
-Con
duct
regi
on-
wid
e ad
voca
cy
that
is in
form
ed
by w
ork
grou
ps
and
com
mun
ity
cons
ulta
tions
-D
evel
op a
loca
l ra
pid
resp
onse
ad
voca
cy
stra
tegy
-Det
erm
ine
linka
ges
and
enco
urag
e flo
w o
f in
form
atio
n ac
ross
the
netw
ork
- Ini
tiate
and
su
ppor
t new
im
mig
rant
em
ploy
men
t in
terv
entio
ns
-Org
aniz
e pu
blic
fo
rum
for i
nput
an
d in
volv
emen
t w
ithin
WR
IEN
pr
iorit
ies
- Stri
ke a
n im
mig
rant
ou
treac
h ta
sk
forc
e to
invo
lve
imm
igra
nts
-Ens
ure
finan
cial
vi
abili
ty o
f WR
IEN
op
erat
ions
-A
ttrac
t $ fr
om
seni
or le
vels
of
gov’
t - S
treng
then
the
role
of t
he g
roup
to
util
ize
non-
finan
cial
re
sour
ces
-Impl
emen
t and
ex
pand
em
ploy
er
enga
gem
ent
stra
tegy
-P
rovi
de
netw
orki
ng
oppo
rtuni
ties
for
empl
oyer
s an
d im
mig
rant
s
-Adv
ocat
e fo
r cr
eden
tial
asse
ssm
ent
-Lau
nch
Imm
igra
nt L
oan
Prog
ram
-A
dvoc
ate
for
Fede
ral p
oint
s sy
stem
- A
dvoc
ate
for
mor
e br
idgi
ng
prog
ram
s
-Co-
advo
cate
at
all l
evel
s -P
rovi
de
info
rmat
ion
to
imm
igra
nts
-Sup
port
prog
ram
s fo
r Im
mig
rant
in
tegr
atio
n in
the
com
mun
ity a
nd
the
wor
kpla
ce
Few
er s
yste
mic
bar
riers
face
d by
im
mig
rant
s in
find
ing
and
mai
ntai
ning
mea
ning
ful e
mpl
oym
ent
Incr
ease
d nu
mbe
r of
imm
igra
nts
and
empl
oyer
s ac
cess
ing
avai
labl
e su
ppor
ts
Incr
ease
d m
arke
tabi
lity
of
imm
igra
nt s
kills
Impr
oved
im
mig
rant
wel
l-be
ing
Stro
ng L
ives
…
Stro
ng E
cono
my…
S
trong
Com
mun
ities
…
Incr
ease
d re
gion
al
econ
omic
pr
ospe
rity
Hea
lthie
r, m
ore
vibr
ant a
nd m
ore
incl
usiv
e co
mm
uniti
es
Incr
ease
d re
sour
ces
leve
rage
d an
d m
ore
effic
ient
allo
catio
ns
Mor
e re
spon
sive
regi
on-w
ide
advo
cacy
to e
mer
ging
loca
l iss
ues
and
bette
r lob
byin
g of
sen
ior
gove
rnm
ents
re: p
olic
y ch
ange
s
Incr
ease
d #
of e
mpl
oyer
s an
d go
vern
men
t/ co
mm
unity
age
ncie
s se
tting
an
nual
pro
gram
pla
ns a
nd
allo
tting
fund
s
Incr
ease
d aw
aren
ess
of im
mig
rant
em
ploy
men
t is
sues
and
impr
oved
acc
ess
to re
leva
nt
info
rmat
ion
(incl
udin
g su
cces
s st
orie
s)
Bet
ter p
riorit
izat
ion,
com
mun
icat
ion
and
publ
icity
of a
com
preh
ensi
ve, r
egio
n-w
ide
plan
Incr
ease
d co
mm
unity
rea
dine
ss fo
r cha
nge:
Pre
-WR
IEN
act
iviti
es c
ause
d pe
ople
see
the
pote
ntia
l of c
olla
bora
tion
and
mot
ivat
ed p
eopl
e to
w
ant t
o co
ntrib
ute
to th
is u
niqu
e co
mm
unity
opp
ortu
nity
(en
ergi
zed
visi
on).
Incr
ease
d st
akeh
olde
r (pa
rticu
larly
em
ploy
er
and
imm
igra
nt) e
ngag
emen
t Im
prov
ed e
nviro
nmen
t for
col
labo
ratio
n,
partn
ersh
ips
and
syne
rgiz
es a
mon
g st
akeh
olde
rs
(sha
red
owne
rshi
p)
Info
rm a
nd m
ake
requ
ests
of
each
oth
er
Dec
reas
ed g
aps
and
over
laps
in
loca
l im
mig
rant
sup
port
prog
ram
s (n
ew c
olla
bora
tive
inte
rven
tions
)
Impr
oved
sen
se o
f bel
ongi
ng w
ithin
W
ater
loo
Reg
ion
by im
mig
rant
s,
and
imm
igra
nts
mor
e lik
ely
to s
tay
Mor
e im
mig
rant
-frie
ndly
wor
kpla
ces
with
suc
cess
ful e
mpl
oym
ent
outc
omes
Incr
ease
d op
portu
nitie
s fo
r im
mig
rant
s to
in
tera
ct w
ith b
usin
ess
com
mun
ity
Incr
ease
d ac
cess
to
educ
atio
nal/s
kill
upgr
adin
g fo
r im
mig
rant
s
Mor
e in
clus
ive
hirin
g pr
actic
es
amon
g em
ploy
ers
and
incr
ease
d tre
nd o
f im
mig
rant
s fin
ding
wor
k in
thei
r fie
ld
Gre
ater
coh
eren
cy a
nd
effe
ctiv
enes
s of
re
gion
al/p
rovi
ncia
l/fed
eral
im
mig
rant
em
ploy
men
t pol
icy
Mor
e su
ppor
t for
em
ploy
ers
to h
ire
imm
igra
nts
and
succ
essf
ully
in
tegr
ate
them
into
the
wor
kpla
ce
(lang
uage
& a
ccul
tura
tion
supp
orts
)
Bet
ter r
ecog
nitio
n an
d en
hanc
emen
t of t
he p
rior
lear
ning
and
cre
dent
ials
of
imm
igra
nts
74 The Canadian Journal of Program evaluaTion74
gagement of diverse stakeholders in gathering ideas and sugges-tions for action, (c) monitoring of relevant initiatives in Ontario and beyond, and (d) extensive work on building a community of activists around immigrant employment issues. these initiatives helped build a foundation of local knowledge, a strong sense of local community need, and an engaged local stakeholder base. But, perhaps most im-portantly, these initiatives also laid the groundwork to identify the priorities for local action that WRiEN could champion.
the WRiEN evaluation built on this local momentum. to begin, in-volving the CCBR to head evaluation efforts ensured that the evalu-ation process would be carried out with a deep understanding of local context and history. Evaluations of system change initiatives such as WRiEN require that a clear bounding of the system is established where social problems, the multiple levels of the system, and the set-tings, organizations, and actors are clearly understood and defined (foster-fishman et al., 2007; Stake, 1978). in addition, the CCBR had become a trusted entity in bringing new stakeholders together, and could now leverage that trust toward cross-stakeholder participation within the evaluation. A lack of trust has been identified as a leading challenge in promoting successful community research partnerships (Becker, israel, & Allen, 2005), including between evaluators and community members (Nelson, Janzen, Ochocka, & trainor, 2010). the WRiEN evaluation benefited from its past history of trust as evaluators facilitated stakeholders to conceptualize and agree upon their unique local theory of change. in this way the evaluation team became an active member of the intervention team (patton, 2011). in the words of one WRiEN staff, “the evaluation was not leading, but helping the leaders to lead.”
value Experiential and practical Knowledge
CCis such as WRiEN are typically conceived by diverse stakehold-ers and grounded in their lived experiences and perspectives. the program theory, goals, and objectives are driven and negotiated by stakeholders (Kubisch et al., 1998). this reflected the efforts lead-ing up to WRiEN and its subsequent implementation. the WRiEN evaluation built on this tradition by inviting stakeholders to (a) rec-reate objectives and actions in each of the stages of the intervention development, (b) critically reflect on the WRiEN’s contributions and achievements from year to year, and (c) develop solutions for improve-ments and options for meeting the overall goal of the intervention. the WRiEN evaluation therefore mirrored patton’s (2008) “theory of
7575la revue Canadienne d’évaluaTion de Programme
action framework” by encouraging practitioners to examine, reflect on, and deal with the discrepancies between their “espoused theory” (how practitioners explain what they are attempting to do) and their “theory in use” (what their behaviour reveals about what actually guides what they do).
valuing the experiential and practical knowledge of evaluation par-ticipants is a key component of reflective practice (Nelson, Ochocka, Janzen, trainor, & lauzon, 2004). Within WRiEN this was concretely achieved through three means. first, many of the questions asked within the evaluation encouraged participants to reflect both on their experience of being involved in WRiEN (e.g., “to what extent was your opinion respected within this WRiEN meeting?”), as well as what they practically accomplished through the involvement (e.g., “to what extent did this meeting move ahead the agenda of this work group/steering committee?”). in addition, the design of the evalua-tion emphasized triangulation of these opinions through multiple methods (e.g., surveys, key informant interviews, observations, focus group, meeting tracking) and from multiple stakeholder perspectives (immigrants, business, government, service providers, education, non-governmental funders). finally, the process of analysis encour-aged collective critical reflection on emerging evaluation themes. data from the multiple mixed-methods were initially analyzed in parallel by the research team. preliminary evaluation findings were then verified and shaped through discussions among WRiEN partici-pants who sat on the evaluation and steering committees.
theory that is grounded in social experiences can lead to better in-formed and more effective practice (Cargo & mercer, 2008; israel et al., 1998). the WRiEN evaluation validated the experiences (whether positive or negative) of the various stakeholders involved in WRiEN, and leveraged that validation in moving participants from a place of opinion and critique toward constructive new action. that is not to say that valuing experiential and practical evidence was without challenges. for example, expectations were raised among evaluation participants that their opinions actually mattered in reshaping the future of the intervention.
the stated openness to forgo a typical fixed theory of change in favour of “learning as we go” (Nelson et al., 1998) certainly intro-duced the challenge of raised expectations that WRiEN would be responsive to evaluation findings. While the goals, objectives, and actions of WRiEN were initially agreed upon, the evaluation was a built-in mechanism to ensure that the numerous and diverse objec-
76 The Canadian Journal of Program evaluaTion76
tives would be continuously negotiated over time, based on the ac-knowledgement and documentation of the experiential knowledge. in this way, the evaluation promoted an organizational learning system (Cousins, Goh, Clark, & lee, 2004). to illustrate, year 3 evaluation findings obtained from multiple stakeholders’ experiences revealed a need for much stronger promotion, communication, and marketing of WRiEN as one of the main priorities for action even though this was not a prominent part of the original set of activities. the challenge was then to deliver on this new urgent set of actions.
provide Ongoing feedback
Knowledge utilization within evaluation is maximized when efforts are made to share evaluation findings in clear, useful, and respectful language targeted at multiple stakeholder perspectives (Nelson et al., 2010; patton, 2008). A number of feedback strategies were used in the WRiEN evaluation to maximize reflective practice. the first dealt with the timeliness of feedback. Whether giving feedback of indi-vidual evaluation methods or providing an annual overall summary, an emphasis was placed on providing feedback as quickly as possible. Closing the gap between data collection and analysis was important in order for those involved in WRiEN (i.e., staff, work groups, steer-ing committee) to make evidence-based decisions that were informed by current information. “Real-time evaluation” was how one member of the business community consequently dubbed the evaluation. the timeliness of generating feedback was essential, particularly in mo-ments when dissatisfaction was detected and when new solutions were emerging. Such a situation occurred between years 2 and 3, when the evaluation results provided grounds for action that led to significant reshaping of its original ambitious goals and priorities.
A second strategy was to use multiple stages and forms of feedback. Regular feedback of evaluation processes and preliminary findings was provided to evaluation committee members in e-news format and during meetings. Summary bulletins for selected methods (e.g., employer web survey and meeting tracking forms) were also distrib-uted and presented to both the evaluation and steering committees for verification and to begin the discussion of implications for future action. At year end, the final report went through multiple stages of feedback. it began with the evaluation committee verifying main themes, providing interpretive comments about how best to frame these themes, and identifying options for action (i.e., recommenda-tions). A presentation to the WRiEN steering committee again served
7777la revue Canadienne d’évaluaTion de Programme
to verify and frame themes, but, more importantly, to stimulate decision-making about how evaluation recommendations could be translated into concrete action steps. finally, the year-end evalua-tion findings were presented to the broader community via a report posted on the WRiEN website and at an open forum.
this leads to the third feedback strategy. Evaluation can be seen as a servant in support of a broader social change movement, particularly for those with limited power and opportunity (mertens, 2009; Nelson et al., 2004). in this light, the feedback loops within WRiEN’s evalu-ation findings could be viewed as an opportunity to constructively advance an immigrant employment agenda. to this end, WRiEN findings were carefully positioned to provide constructive rather than destructive criticism, pointing to the value of learning and evo-lution rather than simply examining and judging what worked and what did not. Such a tack enabled participants to provide not only reflections and comments but also options and scenarios for improv-ing the advancement of immigrant employment.
in summary, a commitment to utilization was foregrounded with researchers adopting feedback formats that tried to connect with di-verse stakeholder audiences (patton, 2008). providing such feedback served to engage stakeholders within the WRiEN intervention: (a) it helped stakeholders continuously reflect and stay on track with their own goals, (b) it provided a venue for timely suggestions for improvements, and (c) it served as a tool for synthesizing knowledge and translating it into desired actions. in other words, evaluation feedback was used to embed continuous reflection into the “WRiEN culture” (Cousins et al., 2004).
facilitate democratic dialogue
in collaborative approaches to evaluations the evaluator may act as a facilitator and co-agent of change (Janzen, 2011; Nelson et al., 2010). the evaluator’s role is not only to enable the generation and synthesizing of new knowledge, but also to lead participants into a democratic dialogue that helps them to co-create knowledge and use it as a catalyst for future action. the facilitation of democratic dialogue resonates with the concept of “process use” that foregrounds the learning that occurs during the evaluation process itself (patton, 2008). learning, including through evaluation, is viewed as a social process (Amo & Cousins, 2007). therefore, how an evaluation is con-ducted (and not simply its findings) may lead evaluation stakehold-
78 The Canadian Journal of Program evaluaTion78
ers to acquire new knowledge and to learn new behaviours and skills (Alkin & taut, 2003).
Within CCis, facilitating democratic dialogue is encouraged by an openness to an evolving program theory that is ever responsive in the emergent local setting (lafferty & mahoney, 2003). for example, WRiEN’s first-year evaluation revealed that despite considerable preparatory work, WRiEN’s role as a systems change intervention was still not understood across participants. Without tangible and visible accomplishments in the first year, some community members began to question the intervention, seeing it as being too broad and removed from their day-to-day realities. the steering committee dis-cussions about these preliminary evaluation findings, however, led to the reframing of findings by recognizing the importance of “founda-tional” change (e.g., increased awareness of immigrant employment issues and improved environment for synergistic partnerships) as precursors to more tangible outcomes (e.g., new collaborative inter-ventions, improved hiring practices). A bamboo analogy was used to explain that considerable root development is needed (foundational change) prior to bamboo shoots breaking through the earth (tangible change). Rather than being simply frustrated with the slow pace of tangible change, stakeholders came to articulate a local logic of how observed foundational changes could lead to desired longer-term impacts.
As the above example illustrates, democratic dialogue was stimu-lated within the WRiEN evaluation primarily in two phases of the reflective action cycle: information gathering (phase 2) and informa-tion sharing (phase 4). With regards to information gathering, a mix of qualitative and quantitative as well as individual and group methods provided variety in dialogue formats. Similarly, the multiple feedback sessions in phase 4 allowed multiple opportunities for ex-change of opinion. in both cases a range of stakeholder perspectives was included, but always emphasizing the voice of WRiEN’s two central stakeholder groups (immigrants and employers). following the tradition of participatory evaluation, evaluation utilization was therefore maximized by actively engaging these multiple stakehold-ers throughout the research process, recognizing that their per-spectives were critical given that knowledge is socially constructed (Cousins & Earl, 1992).
facilitating democratic dialogue was particularly important when transformational changes were desired (such as in year 2 of WRiEN’s operation). At this time the stakeholders were clearly prepared to
7979la revue Canadienne d’évaluaTion de Programme
look at whether the intervention was reaching its main goals and were much clearer in their judgement of successes and challenges. during this stage, the researchers took additional steps to clarify the importance of carefully interpreting results and exploring proactive solutions. this process ensured that the participants felt heard while their issues were addressed in a constructive and mutually respect-ful fashion. moreover, those involved in the interpretation of findings and the generation of recommendations (i.e., evaluation and steering committees) felt empowered to offer solutions that led to improve-ments and desired changes (fetterman & Wandersman, 2005).
facilitating democratic dialogue was not without its challenges. to begin, facilitating democratic facilitation can be resource-intensive. WRiEN staff in particular acknowledged the strain that additional information gathering placed on their busy workloads. Consistent with the experience of staff within other collaborative system change evaluations (e.g., Janzen, Nelson, hausfather, & Ochocka, 2007), WRiEN staff saw the direct benefit of evaluation while also not-ing resource challenges in supporting its activities. Second was the challenge of engaging the full range of stakeholders in dialogue. As with other efforts to engage employers on immigrant matters (public policy forum, 2004), the business sector in particular proved most challenging to engage. Consequently, in the first year of the evaluation few employers were included in the key informant method (contrary to the sampling frame). however, participation in dialogue increased over time as the business sector became more involved in WRiEN and its evaluation and offered its unique insights into im-migrant hiring and retention.
focus on a vision for the Common Good
it is not uncommon for community coalitions to be composed of partners with disparate, if overlapping, agendas, expectations, and interests (Baum, 2001). Such was the case with WRiEN. for example, partners came into the intervention with differing histories—some having grappled with the issue of immigrant employment for many years if not decades, while others being relatively new to the is-sue. Some partners were ready for a systems change intervention (recognizing the limitations of individual-level efforts in addressing systemic barriers to immigrant employment), while others were en-trenched in a mindset of the primacy of individual-level efforts and outcomes. in addition, some partners were primarily motivated in joining WRiEN as a means of promoting social justice: working to
80 The Canadian Journal of Program evaluaTion80
give immigrants equitable opportunity to participate in the labour market and consequently in community life. Others were primarily attracted to an economic imperative agenda, recognizing that an aging population meant that the local (as well as national) economy was entering a time of labour skill shortages that immigrants poten-tially could fill. finally, partners held significantly different social norms for pursuing healthy partnerships. Even the seemingly trivial expectation of what constituted a fruitful meeting exposed deep dif-ferences among partners (i.e., the brief early morning meetings with rational decision-making and a bias for action expected by the busi-ness community, contrasted with the constant connection to lived experience and ample discussion to understand the depth of issues expected by immigrant service providers).
it was within this context that the evaluation was able to help focus the disparate partnership on a common vision. the evaluation acted as a forum in which these diverse perspectives and interests were aired and negotiated in a transparent and systematic way. All stake-holder perspectives were sought as evaluation data. All stakeholders were also invited to re-examine and collectively recommit to their collective vision within the various multi-stakeholder feedback and discussion formats. Still, keeping the initial common vision alive across many stakeholder groups was difficult.
What proved helpful in this regard was to continually focus stake-holders, despite their different motivations, on what they considered to be the “common good.” in evaluation language this translates to reaching a shared understanding of desired outcomes. the slo-gan that stakeholders had crafted for the original immigrant Skills Summit (“strong lives, strong economy, strong community”) was subsequently reframed as long-term goals that remained constant throughout the evaluation (see bottom of the logic models in figures 2 and 3). Such a three-pronged “common good” satisfied both social justice and economic imperative agendas, while framing them as interdependent outcomes within WRiEN’s “official” program theo-ry. Consequently, these long-term outcomes served as a launching point for the ongoing discussion of what intermediary outcomes were expected and what priority activities were needed to reach them. Stakeholders could therefore see how their specific efforts fit into the broader (and shared) system change goals.
Using evaluations within system change interventions as a platform to bring clarity around theory of change is certainly not unique to
8181la revue Canadienne d’évaluaTion de Programme
WRiEN (e.g., foster-fishman & long, 2009; Janzen et al., 2007; Ku-bisch et al., 1998). And using evaluation data as a means of facilitat-ing partnership commitment and activities is also well documented (e.g., Natasi & hitchcock, 2009). the important point to make here is that evaluation that focuses on a vision of the common good offers participants an opportunity to reflectively link their collective knowl-edge with their desired collective action. in other words, evaluation can build a new sense of community among previously separated stakeholder interests—a community that inspires people to work together toward a common goal (Stringer, 2007).
CONClUSiON
in this article we discussed how evaluation can help shape and direct CCis. this was done by offering the Waterloo Region immigrant Em-ployment Network (WRiEN) as a case example to illustrate reflec-tive practice—how theory and practice can be intertwined through evaluation. By framing our reflections around five facilitators that interconnect theory and practice, and by referring to literature on collaborative evaluation, we have placed our discussion within a broader body of knowledge. this body of knowledge represents two parallel and complementary knowledge bases; on the one hand what is known about reflective practice through multidisciplinary writ-ings on participatory action research, and on the other hand what is known about reflective practice through collaborative inquiry within the trans-discipline of evaluation. Although obviously not exhaus-tive, our intent was to bridge the gap that often exists between these two knowledge bases.
the WRiEN case study outlined in this article contributes to the emerging discourse distinguishing formative from developmental evaluation. proponents of developmental evaluation (e.g., Gamble, 2008; patton, 2008, 2011) emphasize that developmental evaluation develops a newly emerged intervention model, while formative evalu-ation improves an existing intervention model. intervention models need to be developed (rather than tested and tweaked) when their contexts are complex with low stakeholder agreement and certainty about outcomes. We agree that this distinction is an important one, and view the WRiEN example as being clearly developmental. the emphasis on emerging program theory within WRiEN explicitly in-vited ongoing reflective practice and allowed evaluators to position themselves as part of what patton calls the “adaptive management team.” Although WRiEN had developed a baseline program theory,
82 The Canadian Journal of Program evaluaTion82
this theory was significantly renegotiated over time. the WRiEN experience therefore suggests that perhaps it is less a matter of whether an intervention model does or does not pre-exist (can an intervention ever be entirely theory-free?) than it is a matter of how loosely or tightly a model is held. from this angle the formative-de-velopmental distinction could be conceived of as a continuum rather than as two discrete categories.
there is a second contribution to the discussion about developmen-tal evaluation. in this article we reflected on WRiEN’s evaluation through five facilitators of reflective practice that were gleaned from past literature. While in his book patton (2011) introduces develop-mental evaluation as a new approach to evaluation (as distinct from formative and summative evaluation), the WRiEN case example demonstrates that using research as a means of facilitating ongo-ing reflective practice within complex environments is neither new nor novel. Although the term “developmental evaluation” was not yet coined in the literature at the beginning of WRiEN’s evaluation, the evaluation can be seen to stand on the shoulders of a deep par-ticipatory action research tradition that spans more than 60 years. it also draws on the growing interest among evaluators pursuing collaborative forms of inquiry that adopt constructivist and other postmodern paradigms of research. these paradigms of research are comfortable with moving beyond the objectivity, linear cause-and-effect assumptions, and controlled environments of positivistic science, particularly when evaluating innovation (Janzen & Wiebe, 2010; Nelson et al., 2010). patton’s developmental evaluation does add helpful connections to complexity theory and the emerging body of knowledge related to social innovation. it may yet prove to become a consensus label useful in articulating and promoting a postmodern approach to evaluation. time will tell. Still, regardless of which label we use, we believe that CCis provide fertile ground to further explore the link between evaluation, reflective practice, and interventions that deal with complexity.
ACKNOWlEdGEmENtS
the authors would like to acknowledge the WRiEN evaluation and steering committees for their input and support during the evalua-tion process, and drs. Geoffrey Nelson and terry mitchell for com-ments on previous section drafts.
8383la revue Canadienne d’évaluaTion de Programme
REfERENCES
Alboim, N. (2009). Adjusting the balance: Fixing Canada’s economic immi-gration policies. toronto, ON: maytree foundation.
Alkin, m. C., & taut, S. m. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1–19.
Amo, C., & Cousins, J. B. (2007). Going through the process: An examina-tion of the operationalization of process use in empirical research on evaluation. New Directions for Evaluation, 116, 5–26.
Baker, t. E. (2003). Evaluating community collaborations: An overview. in t. E. Baker (Ed.), Evaluating community collaborations (pp. 1–18). New york, Ny: Springer.
Baum, h. S. (2001). how should we evaluate community initiatives? Jour-nal of the American Planning Association, 67(2), 147–158.
Becker, A. J., israel, B. A., & Allen, A. J. (2005). Strategies and techniques for effective group processes in CBpR partnerships. in B. A. israel, E. Eng, A. J. Schulz, & E. A. parker (Eds.), Methods in community-based participatory research for health (pp. 52–72). San francisco, CA: Jossey-Bass.
Cargo, m., & mercer, S. (2008).the value and challenges of participatory re-search: Strengthening its practice. Annual Review of Public Health, 29, 325–350.
Centre for Community Based Research. (2010). Mobilizing Waterloo Region around immigrant employment. Retrieved from www. communitybasedresearch.ca/page/view/immigrant_Employment.html
Chavez, v., duran, B., Baker, Q. E., Avila, m. m., & Wallerstein, N. (2008). the dance of race and privilege in community based participatory research. in m. minkler & N. Wallerstein (Eds.), Community-based participatory research for health (pp. 91–106). San francisco, CA: Wiley.
Chen, h.-t. (2005). Practical program evaluation: Assessing and improv-ing planning, implementation, and effectiveness. Newbury park, CA: Sage.
Clare, R. C. (2006). putting faith into action: A model for the North Ameri-can middle class. Religious Education, 101(3), 368–389.
84 The Canadian Journal of Program evaluaTion84
Conde-frazier, E. (2006). participatory action research: practical theology for social justice. Religious Education, 101, 321–329.
Cousins, J. B., & Earl, l. m. (1992). the case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397–418.
Cousins, J. B., Goh, S. C., Clark, S., & lee, l. l. (2004). integrating evalua-tive inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.
dildar, y., & Janzen, R. (2009). Waterloo Region Immigrant Employment Network (WRIEN) final evaluation report. Kitchener, ON: Centre for Community Based Research.
fals Borda, O. (1987). the application of participatory action-research in latin America. International Sociology, 2(4), 329–347.
fetterman, d. m., & Wandersman, A. (2005). Empowerment evaluation principles in practice. New york, Ny: Guilford.
foster-fishman, p. G., & long, R. (2009). the challenges of place, capacity and systems change: the story of yes We Can. Foundation Review, 1(1), 69–84.
foster-fishman, p. G., Nowell, B., & yang, h. (2007). putting the system back into system change: A framework for understanding and chang-ing organizational and community systems. American Journal of Community Psychology, 39(3), 197–215.
freire, p. (1970). Pedagogy of the oppressed. New york, Ny: Seabury press.
Gamble, J. A. A. (2008). A developmental evaluation primer. montreal, QC: J.W. mcConnell family foundation.
heron, J., & Reason, p. (1997). A participatory inquiry paradigm. Qualita-tive Inquiry, 3(3), 274–294.
hogarth, K., Janzen, R., & Cushing, S. (2007). Year one evaluation report for Waterloo Region Immigrant Employment Network (WRIEN). Kitch-ener, ON: Centre for Community Based Research.
israel, B., Eng, E., Schulz, A. J., & parker, E. A. (2005). introduction to methods in community-based participatory research for health. in B. A. israel, E. Eng, A. J. Schulz, & E. A. parker (Eds.), Methods in
8585la revue Canadienne d’évaluaTion de Programme
community-based participatory research for health (pp. 3–26). San francisco, CA: Jossey-Bass.
israel, B., Schulz, A., parker, E., & Becker, A. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Public Health, 19, 173–202.
Janzen, R. (2011). Reaching out to multicultural neighbours: Stories that evaluate and encourage innovative church outreach. Unpublished dissertation. Wilfrid laurier University, Waterloo, ON.
Janzen, R., & dildar, y. (2008). Year two evaluation report for the Waterloo Region Immigrant Employment Network (WRIEN). Kitchener, ON: Centre for Community Based Research.
Janzen, R., hatzipantelis, m., & hogarth, K. (2005). Immigrant skills sum-mit: Waterloo Region conference proceedings. Kitchener, ON: Centre for Research and Education in human Services.
Janzen, R., hatzipantelis, m., vinograd, J., & Kellerman, m. (2004). Good practice and resource guide: Community needs assessments and serv-ice evaluations in Military Family Resource Centres. Kitchener, ON: Centre for Community Based Research (formerly Centre for Re-search and Education in human Services).
Janzen, R., Nelson, G., hausfather, N., & Ochocka, J. (2007). Capturing system level activities and impacts of consumer-run organizations: methods that inform future action. American Journal of Community Psychology, 39(Special issue on systems change evaluation), 287–299.
Janzen, R., Walton-Roberts, m., & Ochocka, J. (in press). immigrant integra-tion and inclusion in Waterloo Region. in J. Biles & C. Andrew (Eds.), Immigration, integration and inclusion in Ontario cities. montreal, QC: mcGill-Queen’s University press.
Janzen, R., & Wiebe, d. (2010). putting God in the logic model: developing a national framework for the evaluation of faith-based organizations. Canadian Journal of Program Evaluation, 25(1), 1–26.
Kemmis, S., & mctaggart, R. (2005). participatory action research. in N. K. denzen & y. S. lincoln (Eds.), Handbook of qualitative research (3rd ed.; pp. 559–603). thousand Oaks, CA: Sage.
Kubisch, A. C., fulbright-Anderson, K., & Connell, J. p. (1998). Evalu-ating comprehensive community initiatives: A progress report. in
86 The Canadian Journal of Program evaluaTion86
K. fulbright-Anderson, A. C. Kubisch, & J. p. Connell (Eds.), New approaches to evaluating community initiatives: Volume 2–Theory, measurement, and analysis (pp. 1–14). Washington, dC: Aspen in-stitute.
lafferty, C. K., & mahoney, C. A. (2003). A framework for evaluating com-prehensive community initiatives. Health Promotion Practice, 4(31), 31–44.
lewin, K. (1948). Action research and minority problems. in G. W. lewin (Ed.), Resolving social conflicts (pp. 201–216). New york, Ny: harper & Row.
mackenzie, m., & Blamey, A. (2005). the practice and the theory: lessons from the application of a theory of change approach. International Journal of Theory, Research and Practice, 11(2), 151–168.
mcfadden, p., & Janzen, R. (2007). the importance of immigrants to Water-loo Region’s prosperity: A dynamic collaborative community process. Our Diverse Cities, 4(fall), 104–107. Retrieved from http://canada.metropolis.net/publications/index_e.htm
mclaughlin, J. A., & Jordan, G. B. (2004). Using logic models. in J. S. Wholey, h. p. hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 7–32). thousand Oaks, CA: Sage.
mertens, d. m. (2009). Transformative research and evaluation. New york, Ny: Guilford press.
miedema, J. m., & vandebelt, d. (2006). Immigrants and growth: A look at health and employment in Waterloo Region. Waterloo, ON: Region of Waterloo public health.
minkler, m., & Wallerstein, N. (2008). introduction to community based participatory research. in m. minkler & N. Wallerstein (Eds.), Com-munity-based participatory research for health (pp. 5–24). San fran-cisco, CA: Wiley.
Nastasi, B. K., & hitchcock, J. (2009). Challenges of evaluating multilevel interventions. American Journal of Community Psychology, 34(4), 360–376.
Nelson, G., Janzen, R., Ochocka, J., & trainor, J. (2010). participatory action research and evaluation with mental health self-help groups and or-ganizations: A theoretical framework. in l. Brown & S. Wituk (Eds.),
8787la revue Canadienne d’évaluaTion de Programme
Mental health self-help: Consumer and family driven initiatives (pp. 39–58). New york, Ny: Springer.
Nelson, G., Ochocka, J., Griffen, K., & lord, J. (1998). “Nothing about me, without me”: participatory action research with self-help/mutual aid organizations for psychiatric consumer/survivors. American Journal of Community Psychology, 26, 881–912.
Nelson, G., Ochocka, J., Janzen, R., trainor, J., & lauzon, S. (2004). A comprehensive evaluation framework for mental health consumer/survivor organizations: values, conceptualization, design, and action. Canadian Journal of Program Evaluation, 19(3), 29–53.
Newbrough, J. R. (1995). toward community: A third position. American Journal of Community Psychology, 23(1), 9–37.
Ochocka, J., moorlag, E., & Janzen, R. (2010). A framework for entry: pAR values and engagement strategies in community research. Gateways: International Journal of Community Research & Engagement, 3, 1–19.
patton, m. Q. (2008). Utilization-focused evaluation (4th ed.). Beverly hills, CA: Sage.
patton, m. Q. (2011). Developmental evaluation: Applying complexity con-cepts to enhance innovation and use. New york, Ny: Guilford press.
public policy forum. (2004). Bring employers into the immigration debate. Ottawa, ON: Author.
Reason, p. (2006). Choice and quality in action research practice. Journal of Management Inquiry, 151(2), 187–203.
Shadish, W. R., Cook, t. d., & leviton, l. C. (1991). Foundations of program evaluation: Theories of practice. Newbury park, CA: Sage.
Stake, R. (1978). the case study method of social inquiry. Educational Re-searcher, 7(2), 5–8.
Stringer, E. t. (2007). Action research (3rd ed.). thousand Oaks, CA: Sage.
tashakkori, A., &. teddlie, C. (2003). the past and future of mixed methods research: from data triangulation to mixed model designs. in A. tashakkori & C. teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 671–701). thousand Oaks, CA: Sage.
88 The Canadian Journal of Program evaluaTion88
taylor, A., & Botschner, J. (1998). Evaluation handbook. Kitchener, ON: Centre for Community Based Research (formerly Centre for Re-search and Education in human Services).
travers, R., Wilson, m. G., flicker, S., Guta, A., Bereket, t., mcKay, C., … Rourke, S. B. (2008). the greater involvement of people living with AidS principle: theory versus practice in Ontario’s hiv/AidS com-munity-based research sector. AIDS Care, 20(6), 615–624.
Wallerstein, N., & duran, B. (2003). the conceptual, historical, and practice roots of community based participatory research and related partici-patory traditions. in m. minkler & N. Wallerstein (Eds.), Community-based participatory research for health (pp. 25–46). San francisco, CA: Wiley.
Wayland, S. (2006). Unsettled: Legal and policy barriers for newcomers to Canada. Ottawa, ON: Community foundations of Canada and the law Commission of Canada.
Wayland, S. (2007). The Learning Exchange: Disseminating good ideas: Skilled immigrants in the labour market. Report prepared for CiC and the J.W. mcConnell foundation.
Rich Janzen is Research director at the Centre for Community Based Research (www.communitybasedresearch.ca). Rich recently completed his ph.d. (Community psychology) at Wilfrid laurier University.
DanielaSeskar-Hencicis the former manager of planning, Evalu-ation and Epidemiology at the Regional municipality of Waterloo. presently daniela is Associate director, institutional Analysis and planning at the University of Waterloo.
Yasir Dildar is Senior Researcher at the Centre for Community Based Research. yasir has m.A.s in sociology and development stud-ies.
PeterMcFaddenis the former Executive director of the Waterloo Region immigrant Employment Network (WRiEN).