Post on 04-Apr-2018
7/29/2019 Evaluating Community Organizing (2)
1/16
Grassroots Actionand Learning for Social Change:
2010 Center for Evaluation Innovation. All rights reserved.
May not be reproduced whole or in part without written permission from Center for Evaluation Innovation.
Evaluating CommunityOrganizing
CatherineCrystal Foster
Justin Louie
Blueprint Research
& Design, Inc.
March 2010
7/29/2019 Evaluating Community Organizing (2)
2/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 1
Catherine Crystal Foster and Justin Louie
Blueprint Research & Design, Inc.
March 2010
Evaluating Community Organizing
Grassroots Actionand Learning for Social Change:
Community
organizing shares
characteristics with
policy advocacy,
but it differs in
important ways.
Approaches for
evaluating the two
also differ.
In recent years, the practice o community organizing has matured and evolved,
gaining visibility as a vibrant and potent orce or social change. This has coincided
with a growing ocus on systemic reorms throughout the social sector, reorms
that eective community organizing can help advance. As this work has gained moreprominence, and its potential has been more broadly understood, interest in assess-
ing the contributions and impact o organizing activities has surged.
This interest is part o a larger trend seeking metrics to drive program improvement
and innovation in the social sector. Growing numbers o unders, evaluators, and
nonprots are striving to develop meaningul program assessments, particularly or
systemic reorms such as organizing, advocacy, and other policy change work.
While it was once common to hear questions about whether advocacy eorts
could be meaningully evaluated, those have long since given way to discussions
o how to take advocacy evaluation to the next level. Development o rameworks
and an array o tools to measure progress along the multiple dimensions o policychange and advocacy have emerged. And now, ater implementing and vetting these
approaches, practitioners have been adapting them to address the particular needs
and circumstances o community organizing, as distinct rom other orms o social
change or advocacy work.
Community organizing or social change shares many characteristics with policy
advocacy, but it diers in signicant ways, and the approaches to evaluating the two
also dier. As evaluators, we have partnered with organizers, advocates, and their
unders over the last ve years, and we have seen these dierences rst-hand. Since
developing our initial ramework or assessing advocacy and policy change activities
in 2005,1
we have worked closely with seasoned and emerging community organizerson a range o evaluation projects, covering multiple issue areas in a variety o
locations. We have provided technical assistance and tools to organizers, helped
systematize their existing assessment processes, and led external evaluation eorts
in partnership with them. Throughout our work, we have continued to learn rom
organizers and rene our own thinking and processes.
1 Guthrie, K., Louie, J., David, T., & Crystal Foster, C. (2005). The challenge of assessing policy and advocacyactivities: Strategies for a prospective evaluation approach. San Francisco, CA: Blueprint Research &Design.
7/29/2019 Evaluating Community Organizing (2)
3/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 2
Community
organizing
catalyzes the power
of individuals to
work collectively
for the changes
they want in their
communities. It
places a strong
emphasis on
developingcommunity leaders.
This brie describes our vision o community organizing evaluation. It is grounded
in a set o principles that have emerged rom our experiences with organizers, as
well as the collective wisdom o others working in this eld. We have learned that
evaluation o organizing should be:
Participatory, rather than purely third-party
Prospective (contemporaneous and orward-looking), rather than retrospective
Learning-based, rather than part o a pass-or-ail mentality
Real-time, rather than delayed and academic
Respectul o the culture o organizing
Attentive to leadership development as well as policy wins
Focused more on evidence than proo.
We begin the brie with an examination o the similarities and dierences between
advocacy and organizingdierences that shape how our approach to evaluationo organizing diers rom evaluation o advocacy. We then present a ramework or
evaluating organizing, and discuss some o the lessons we have learned through the
ongoing process o implementation.
What is Community Organizing?
Community organizing is a democratically-governed, values-driven process thatcatalyzes the power o individuals to work collectively to make the changesthey want to see in their communities. Community organizers honor and develop
the leadership potential in everyday people by helping them identiy problems and
solutions, and then by supporting them as they take action to make those solutionsa reality. In so doing, organizing challenges the existing power structure.
Relationships lie at the heart o organizing, and the one-to-one relational
conversation between an organizer and a community member is the building block
o organizing. As those community members participate in social change work, build
skills, and take on responsibilities, they become leaders within the organizing
group. Developing these leaders and building the base o leaders and other
community members is an ongoing ocus o community organizing.
A given community organizing group may emerge on its own and remain
independent, or be associated with one o several national organizing networks,
such as the PICO National Network, Association o Community Organizations or
Reorm Now (ACORN), Industrial Areas Foundation (IAF), Direct Action and Resource
Training Center (DART), Gamaliel Foundation, and others. Some are aith-based, and
others are institutionally- or place-based. Organizing may also coalesce around an
issue, like immigration, health, or education; or around populations, such as youth,
or a specic ethnic group.
7/29/2019 Evaluating Community Organizing (2)
4/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 3
Organizing Shares Qualities with Policy Advocacy
Like policy advocacy, community organizing eorts requently require longtimerames to come to ruition. The process o growing and mobilizing a base,building power, shiting attitudes, and changing policies or practices does not
happen overnight. Holding those in power accountable ollowing change also takes
time. The work o organizing, like advocacy, is complex and iterative. Strategies
shit, oten rapidly and multiple times, based on the external environment and otheractors. Ultimate outcomes may not be readily quantiable. The work is requently
collaborative, which brings with it benets and challenges. And organizing
campaigns, like advocacy campaigns, oten spring rom a sense o crisis and cause
the participants to approach the work with a high level o emotional intensity.
But Organizing is Distinct rom Advocacy
Organizing and advocacy dier, however, at a core level. Community organizingis emphatically bottom-up. It is the community members who select theissues, proer the solutions, and drive strategy and execution. Most advocacy is
undamentally top-down, even i the work is authentically undertaken on behalo community members. Advocates speak or others, while organizers inspire
community leaderseveryday peopleto speak or themselves. Tellingly, the
so-called Iron Rule o organizing is, Never do or people what they can do or
themselves.
Organizers and leaders also believe that community members can be experts, and
that expertise is not the sole domain o policy proessionals. A low-income mother
with little ormal education can be an expert on local educational needs just like a
senior think tank ellow, through her own experience or by conducting community-
led action research in her neighborhood school.
The leader-ocused lens also points to another dierence rom advocacy. In
organizing, leadership development is a central concern and a key outcome in
addition to policy change objectives. This has major implications or priorities and
goals. It makes capacity development look dierent in organizing than in advocacy,
since the capacities to attract and develop leaders are a top priority in organizing.
Finally, certain logistical aspects o organizing dier rom advocacy in a signicant
way. Organizers operate in a predominantly oral culture, in contrast to the more
archived, written culture o advocacy. Organizing oten places a premium on process
and ritual, particularly as it concerns base-building and direct actions. In addition,
organizing takes place in a more diuse setting: in homes, churches, schools, or
community venues, rather than in a central oce or the corridors o the state house.
Evaluation o Organizing Presents Unique Challenges
In many respects, the similarities between advocacy and organizing presentsimilar evaluation challenges. But or organizing, evaluation requires additionalconsiderations that refect the particular qualities o the work. Most important, the
bottom-up nature o organizingdriven by the community, not by organizational
managers or external proessionalscreates a whole new set o complexities. This
7/29/2019 Evaluating Community Organizing (2)
5/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 4
orientation collides with the inherently top-down nature o traditional third-party
evaluation, in which outside experts ask the questions, set the terms, and make
judgments. As we have noted, organizers have a undamentally dierent view rom
advocates not only o how decisions are made and priorities are set but also in
where expertise resides. That aects how organizers view evaluation generally, and
what role they see or themselves and their leaders in that process.
I the community-dened, bottom-up goals or the work do not align completelywith a unders goals, an evaluator measuring against those goals aces the dicult
task o navigating between the two. When the work unolds as part o a multi-site
initiative in which multiple communities have been unded to work on an issue,
those complications are compounded. Since the goals, strategies, and tactics o
organizing bubble up in ways that are highly context-specic, multi-site evaluation
o an organizing eort is particularly hard. It is quite dicult to standardize
methodologies and roll up results when the work and processes are driven by the
needs and approaches o each community.
As well, the intense ocus on leadership development in organizing, and the
emphasis on process within some schools o organizing lead to identicationo interim benchmarks and goals that oten dier rom those in an advocacy
campaign targeting similar policy change objectives. Organizing requires additional
benchmarks and goals related to the processes o growing leadership and power,
and organizers may prioritize them dierently rom advocates.
Finally, certain practical considerations implicit in organizing work can impact
evaluation. Many organizers value refection quite highly, and incorporate it in
their work more explicitly than some advocates. This is particularly true or aith-
based organizers. As a result, evaluation may be more about systematizing inormal
refection and helping to ocus it more on impact than process, not about teaching
the value o it. Yet, while they do refect regularly, organizers have very little timeor ormal evaluation and the rigorous, uniorm, and documented processes o data
collection and analysis that ormal evaluation can imply. They pride themselves on
never being in the oce, instead spending their time in the community. Leaders who
carry out the work are community members who may have entirely separate day
jobs, making systematic evaluation ar more challenging than when partnering with
advocates working in a more traditional oce environment.
How an Evaluator Can Address the Unique Challenges o EvaluatingOrganizing
When conronted with these complicating actors, how can an evaluator
respond? In our work, we have started by setting an overall approach to
the evaluation that refects key attributes o community organizing. First and
oremost, a community organizing evaluation is most eective when it involves
the participants in an ongoing and meaningul way, and rames the evaluation as a
learning opportunity or everyone. Embracing the bottom-up approach o organizing
and the organizers highly democratic notion o expertise leads to an evaluation
that leverages the strengths o organizing rather than ghting its values. This is a
stark contrast to accountability-ocused third-party analysis. We have ound that
the participatory approach is the most eective way to capture useul and reliable
Organizing requires
benchmarks
to capture the
process of growing
leadership and
power.
7/29/2019 Evaluating Community Organizing (2)
6/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 5
inormation, interpret that inormation, and ensure that data collected are actually
used to improve the work. Intentional relationship building with organizers and
leaders is crucial to success in such an evaluation.
The dynamic nature o most organizing work and the intensity o organizing
campaigns make real-time data collection and eedback particularly important.
Organizers need eedback today to move their campaigns orward tomorrow.
They are not as interested in academic examinations o long-past work or detailedcompilations o proo. Experienced organizers know what power looks like and
they know where they want to go. What they oten welcome is a process or more
systematically examining the multitude o interrelated orces and activities that will
take them there.
Evaluation eedback or organizers becomes most useul when it is oriented
toward learning rather than judgment or a pass-or-ail accountability. A ocus on
inormation or learning and growth has particular resonance or organizers who
devote themselves to developing individual leaders and the process o building
power. Faith-based organizers and those adopting an action-refection method o
organizing naturally gravitate toward a more ormative, developmental approach toevaluation. 2
Given this receptivity, and our interest in promoting evaluation or organizational
learning, we have ound it productive to help organizers build their own internal
evaluation capacities. This enables them to work in greater partnership with us and
to carry on authentic assessment activities in the long-term, both with and without
external evaluation assistance.
We have also ound that in contrast to program or advocacy evaluation, organizing
evaluation must measure and credit achievement o leadership development
and capacity development goals as much as policy goals, since that mirrors the
organizers work.
The evaluation also works best when it accommodates organizing style and culture.
This requires attention to the language and philosophical underpinnings o a given
organizing groups approach, recognition o the volunteer-based stang structure,
and the more oral culture implicit in organizing work.
Basic Scaolding or Evaluation o Community Organizing
Understanding the dierences between community organizing evaluation andadvocacy evaluation at the conceptual level we have described is crucial, buthow does an evaluator use that understanding to implement a real evaluation onthe ground?
2 Formative evaluation, in contrast to summative evaluation, assesses a program or activities whilethey are forming or unfolding. It looks at the context, inputs, and processes of implementation to helpimprove and refine the program or activities. Developmental evaluation, a term coined by MichaelQuinn Patton, is a team effort that supports program, project, personnel and/or organizational devel-opment through a process of continuous improvement and adaptation. The evaluator in a develop-mental evaluation facilitates data-based decision-making by the team.
Evaluation for
organizers is most
useful when it is
oriented toward
learning rather
than judgment
or a pass-or-fail
accountability.
7/29/2019 Evaluating Community Organizing (2)
7/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 6
We have ound in our work that to structure an evaluation o community organizing
that yields timely inormation that organizers and their unders can use, we need to
start by erecting some basic scaolding with the organizers. There is little new about
this part o the process, particularly to those amiliar with advocacy evaluation, but
it is important to establish what the supporting beams look like. As we describe, it is
the next stages o construction that add the particular approaches and perspective
that eective organizing evaluation requires.
To begin, in partnership with organizers:
Articulate a theory o change,3 by whatever name and in whatever orm is most
accessible to the organizers and their leaders, making it clear that the theory
does not need to be linear or static.
Articulate evaluation questions, thinking collaboratively about what organizers,
leaders, unders, and other stakeholders want to learn.
Dene benchmarks and indicators, paying particular attention to mid-term
benchmarks and to the range o outcomes that are important to reaching
organizing goals, recognizing that they may shit over time.
Establish multiple data collection methods and collect data by involving
evaluators, organizers, and leaders in the process.
Provide real-time eedback, to the extent easible.
Document progress toward benchmarks and goals, and re-evaluate and rene the
goals and benchmarks as strategies shit.
Engage with unders on ndings and processes.
Next, based on what has been learned:
Organizers rene their strategy.
Organizers and evaluators rene evaluation practices and maintain on-going
evaluation.
Building on the ScaoldingWhat We Have Learned
T he basic elements o community organizing evaluation are relativelystraightorward, but how does an evaluator build on them in practice to create alearning partnership that eels authentic to organizers and yields useul results?
Categorize Organizing Goals Systematically
We have learned that when determining evaluation questions, setting benchmarks,
and selecting data collection methods, it helps to categorize the work in a way that
incorporates the values and orientation o organizing. One useul ramework has
3 An organizers theory of change describes, in broad terms, the ultimate goals of the organizing effortand the logical chain of activities and interim outcomes that the organizers believe will get them tothose goals.
7/29/2019 Evaluating Community Organizing (2)
8/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 7
been developed by the Alliance or Justice (AFJ).4 We adapted the AFJ rame, using the
ollowing core components o organizing:
Participation and membership (the base): This includes, but is not limited to,
the numbers o members o an organizing group, their attendance at organizing
events, and the extent to which they represent the community being organized.
It encompasses their engagement in issue and strategy selection and their
identication with the goals o the organizing group.
Constituent leadership and power: Flowing rom participation, real leadership
is the distinguishing hallmark o community organizing, as opposed to advocacy
or other orms o social change work. As members gain skills, take ownership
o change processes, and begin to lead others in their community, they become
leaders. They then use that leadership capacity to gain power and make change.
Organizational power: The collective power o a community organizing group is
what gives it the ability to name problems, demand specic solutions, and hold
those in power accountable. It gives the organization visibility and leverage.
Organizing wins: Wins are the most valued currency in community
organizing. When organizers and leaders succeed in a campaign, show power
in a demonstrable way, or get what they seek in an ask, they chalk up a win.
A win may not be as obvious as passage o a desired policy. It may also be a
key step along the way, the prevention o a less-desired outcome, or a shit in
relationships.
Meaningul impact ollowing wins: Implementation and accountability ater a
win are as important as the win itsel. Having the power to realize the promise o
a win is critical in organizing.
Organizational capacity: As with any successul social change entity, acommunity organizing group requires structures, resources, knowledge, and
vision to achieve its goals. Given the way community organizing operates, the
ability to orge and strengthen relationships, create strategic alliances, and
manage a base are vital capacities in community organizing.
Refection and innovation: An organizing groups process or sel-assessment,
renement o strategy, and adaptability in a rapidly shiting environment are key
to separating skilled organizing rom mere activism (direct action that may not
be grounded in strategy or oriented toward sustained and eective ollow-up or
realignment o power relationships). Clearly, this aspect o organizing has strong
resonance in the evaluation context.
4 This Core Components of Community Organizing framework and additional useful information onevaluating organizing can be found at AFJs Resources for Evaluating Community Organizing website,http://www.afj.org/for-nonprofits-foundations/reco/.
When developing
the evaluation, the
core components of
organizing can be a
guide.
7/29/2019 Evaluating Community Organizing (2)
9/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 8
Crat Evaluation Questions Meaningul to Organizers
We have engaged both unders and grantees in the development o evaluationquestions at the very start. This keeps the evaluation relevant and useul, andpromotes greater buy-in and participation by the organizers. As we continuously
look to organizers or cooperation in data collection and or engagement on
interpretations and ndings, it has been quite helpul that many o the questions we
are asking are the ones they want answered.
Evaluation questions are highly individualized to the needs o the evaluation audience
and sponsors. We ound, however, that in our organizing evaluations, they generally
all within the core organizing component categories listed above. By reerencing
the components when developing questions, it may be easier to see what is missing
and to surace any unintended biases in proposed areas o inquiry. Training attention
on organizational capacity and constituent leadership and power, or example, can
balance what might otherwise be a more exclusive ocus on policy wins.
Set Clear Interim Benchmarks, and Be Willing to Reine Them
Data collection and analysis may seem like the hardest work o an evaluation.Yet setting meaningul, sucient, and achievable interim benchmarks that areexpected to occur between activities and longer-term social change goals can oten
be a major struggle in organizing work. With varying degrees o success, we have
encouraged hard thinking about interim benchmarks ocused more on substantive
outcomes. We have ound it can be helpul to look explicitly at the ull range o
organizing components and the ways in which they interrelate when setting
meaningul interim benchmarks. Similarly, we have relied on regular, in-depth
dialogue with organizers about the way progress within each component develops
and what it looks like, since organizing benchmarks rarely remain static over time.
Reerring back to the relatively high-level components o organizing we set orthearlier adds a consistency to the work when the shiting environment in which
organizers operate causes strategies and tactics to evolve or quickly change.
The table on page 15 illustrates examples o some o the benchmarks we have used
and the corresponding data collection methods used to track progress toward those
benchmarks.
Use Data Collection Methods that Fit with Organizing Culture
We have used a wide array o data collection methods in our organizingevaluations. We ound that we obtain the most useul inormation, with themost productive organizer participation, by using methods that t the style and
existing practices o the organizing group and are tailored to capture the dierent
components o organizing. When possible, we support organizers in creating their
own data collection tools.
For example, we worked with a multi-issue, aith-based community organizing
group in Caliornia on evaluating their eorts to expand access to health care in
their county. They wanted to look deeply at the strength and breadth o the local
organizing committees o their member congregations (aspects o Participation
7/29/2019 Evaluating Community Organizing (2)
10/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 9
and Membership and Constituent Leadership and Power). These committees
were the core teams working to move the organizing groups health access agenda.
The organizers led an inclusive, thorough process to select the key participation
measures they wished to track. They then worked with a consultant to customize
ree sotware (Zoho) that would allow each organizer to track their leaders by
committee (even when out in the eld, using their iPhones), and to provide clear
charts and graphs that highlighted progress and areas or growth.
Looking at a very dierent aspect o growth, we helped education organizers in
Pennsylvania think about how to document their progress in building external
champions. These champions are key indicators o development o organizational
power in order to leverage wins. The organizers created an online survey tool that they
update regularly, which tracks relationships and interactions with specic targets over
time. We can reer to the results as we assess progress, while the organizers can use
the data to rene their own strategies and activities. We have shared the templates
or this and other organizer data collection tools with other organizing groups in the
multi-site collaborative in which the Pennsylvania organizers participate. We plan to
post them on a private website so that organizers across sites can adapt each others
methods (taking care not to publicize matters that reveal too much about strategy inways that could jeopardize the organizing work).
One other methodology that we have ound particularly productive across almost
all aspects o organizing work is what we call the critical incident debrie.5
Immediately ater a win, loss, or infection point in an organizing drive, we conduct a
series o 360-degree interviews with individuals and groups both inside and outside
o the campaign, including opponents, allies, observers, and decision makers. We
analyze what took place, what went well, what could be improved, what growth
occurred, and what can be learned as the work moves orward. We write up our
ndings in a narrative or the organizers to use internally and share with their
unders. Our ndings are generally revealing, sometimes raw, sometimes validating,and close enough in time to be actionable. In one example, our debrie ollowing a
coalitions painul legislative loss in Colorado helped provide insight on leadership
and strategy that guided the coalition in a renewed eort to move the bill during the
next legislative cycle.
Develop Buy-In and Trust to Ensure Successul Implementation
Successul implementation o an organizing evaluation hinges on the relationshipbetween the organizers and the evaluators, and on the extent to which theevaluation meets the organizers where they are.
Our evaluation eorts have sparked the most learning and have worked best when
we have had internal evaluation champions within the organizing group. The
champions help ensure engaged participation, which we rely upon or data collection,
interpretation, and use. For example, we have worked with executive directors
o organizing groups whose enthusiasm or evaluation has created a culture o
5 Our methodology was inspired by the Intense Period Debrief focus group protocol used byInnovation Network. http://www.innonet.org/?section_id=101&content_id=581. See also Bagnell Stuart,J. (2007). Necessity leads to innovative evaluation approach and practice. The Evaluation Exchange,13(1), 10-11.
Evaluations tend
to work best when
internal evaluation
champions areengaged.
7/29/2019 Evaluating Community Organizing (2)
11/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 10
learning and inquiry within their organizations. This has led them to inuse data and
documentation into the organizations refection practices and to institutionalize
evaluation. An organizing group in Denver, or example, used one o our debrieng
narratives to call a meeting with their unders to start a dialogue on some o the
provocative questions we raised. This led to a productive exchange and noteworthy
shit in power relationships and credibility or the organizing grantees. Our work with
a Caliornia group gave the executive director and a key sta person the opportunity
to develop and implement a new suite o planning, tracking, and refection protocols
that the organizers now use routinely.
But we have also seen that it is important that the evaluation is widely held within an
organization, not merely by the champion. Absent broader buy-in, evaluators may not
receive consistent, complete, and sustainable fows o inormation, particularly given
how busy organizers can be and the usual rate o turnover in organizing groups.
Our evaluation champions all were naturally attuned to learning, but our ability
to cultivate them as evaluation champions depended on our eorts to develop
relationships with them. This relationship-building takes time, and requires evaluators
to work to earn the organizers trust. Honesty, preservation o the condentialityo strategic inormation, using the unique language o organizing, and showing
understanding o the groups organizing philosophy all make a real dierence.
Similarly, demonstrated interest in digging deep into the policy and political issues
the organizers are conronting, and understanding their political environment and
the players in it, yields major dividends. For some organizers, that means having the
kind o one-to-one relational meetings that organizers have with their leaders. For
others, it means poring over blog posts and news accounts to get a detailed grounding
in the political realities o a campaign so that we can speak more as partners and use
organizers time respectully. Ultimately, the relationship and the evaluation need to
have enough value to the organizers to justiy the expense o time and eort. And the
relationship needs to be strong enough that evaluators can maintain the ongoing datacollection and eedback activities that prospective, real-time evaluation requires, even
when the eedback delivered may not be what the organizer wants to hear.
Beyond the basics o interpersonal relationship-building, in our work with organizers
we have secured additional stipends or the organizers to assist with local data
collection. While the stipends do not ully compensate the organizers or the time they
spend on evaluation, the act that we sought the stipends is a gesture o respect and
helps solidiy our partnership.
We have also ound it important to respect the culture and existing organizational
systems and approaches o our organizing partners. In some cases, that means using
more rudimentary data collection approaches than desired, or relying on existing
systems. For others, it means waiting until the organization can more organically
arrive at systems that have meaning to them and are sustainable. For example, we
created an online media tracking tool or one organization, which they modied and
piloted. They determined it was not a process they could maintain, nor was it yielding
the inormation they really needed. We gave them the space to create what they
needed and they now are ocusing on tracking the growth o their relationships with
journalists in targeted markets rather than logging coverage. They have ound this
useul and so have we.
When developing
data collection
systems, it is
important to respect
the culture of
organizing groups
and their existing
evaluation capacity
or systems.
7/29/2019 Evaluating Community Organizing (2)
12/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 11
Set Achievable Expectations to Maximize Learning and Sustainability
For organizations that were building internal evaluation mechanisms rather thanrelying on external evaluation, we have encouraged them to go slow and startsmall. That may require ocusing on a single aspect o the work. The organization
that examined its local organizing committees using Zoho sotware, or example,
was initially going to evaluate its health policy work more directly. Instead, the
leadership development work provided a comortable and helpul entry point.Starting with activities that oer useul, timely insights into ongoing work has
been most eective in our organizing evaluations to date. In act, despite our
recommendations, some organizers with whom we worked have been so excited
about implementing new tracking and evaluation systems that they have orged
ahead, only to become overwhelmed part-way through. Our eagerness to see these
organizations embrace evaluation probably inhibited us rom orcing the issue and
requiring them to scale down their expectations and not bite o more than their
organization could handle. As approaches have been tested, they have learned and
retooled the approaches to work or them in the longer term.
Use Evaluation Results to Create Impact, But Recognize theChallenges o Getting Results in Real-Time
We rmly believe that evaluation has value only i the results are used. Wehave been gratied by the extent to which organizers and their unders havelearned rom our evaluation eorts and acted on what they have learned. We have
seen that ownership o the data and willingness to learn (even rom hard truths)
can lead to powerul impact. The Colorado organizers who have used evaluation to
rene campaign strategy and explore issues with unders in a new way are just some
examples. One coalition o organizing partners in Chicago is using evaluation data
on a win to make certain that implementation realizes its ull potential. Another
Chicago organizer coalition is learning rom our debrieng on a win to ensure thateorts to repeat the success are executed most eectively. Several organizers
elsewhere are rening their base-building eorts ater examining areas or growth
revealed through data.
Using results well, however, is not always easy. Timeliness matters, and providing
real-time or close-in-time data is oten challenging. Collecting enough data rom
a sucient number and range o sources is not quick, particularly when the
inormation sources are external to the organizing eort. Organizers assisting with
data collection may be too busy to collect data regularly enough or it to be useul.
While we attempt to provide eedback as rapidly as possible, we have realized over
time that we needed to be realistic with organizers about how long it can take to getsucient and accurate inormation back to them. Nevertheless, we continue to try
to make our eedback happen as quickly as possible.
It can also be dicult to propagate the learning in ongoing, ast-paced, and sensitive
campaigns. Similarly, the politics and complex communications paths in coalitions
and networks can slow down or complicate inormation sharing, particularly when
there are questions o attribution or responsibility. We have ound it helpul to be
as patient, ocused, and transparent as possible and to provide our analysis in an
accessible way. Liting up examples rom peer organizing groups has also eectively
7/29/2019 Evaluating Community Organizing (2)
13/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 12
motivated organizers to ocus on and use evaluation themselves.
Clariy What Success Means
Dening and measuring success plays a critical role in evaluation, whether thatmeans a policy win, the achievement o learning goals, or achieving growth inorganizational capacity. Through our work in a variety o social change contexts
and with a range o unders, it has become clear how important it is to suraceassumptions about what success looks like rom the perspective o an organizer,
versus an advocate, versus a under (and even among dierent types o unders).
As we have noted, success or organizers encompasses leadership development
and positive shits in power dynamics as well as desired policy change. Success or
others may be more limited. In addition, our evaluation work with organizers and
advocates has reinorced that or many organizers who work in a values-based
context, how you win is oten as important as winning itsel. Passing a bill may be an
important goal, but it may be equally important or the individual leaders pushing
or that passage to have their role and identity ramed in a particular way as the bill
is considered, or or the compromises to evince a particular power alignment. Thismay not be as true or some allied advocates or unders. Evaluation o an organizing
campaign needs to account or such nuances when measuring success.
Moreover, unders o organizing may dier about the role that organizing can and
should play in a policy success, depending on whether they come at the grant as
issue-oriented unders or as community or civic engagement unders. Issue-based
unders (health care, education, environment, etc.) may see organizing as simply one
tool to be used in reaching a policy change or social change goal, while community
or civic engagement unders may see organizing as the central component o
the work and an end in and o itsel. When raming the evaluation questions and
analyzing the results, evaluators must ensure that there is clarity on the undersorientation and how unders articulate success.
Conclusion
We have learned rom our work with organizers that expertise comes in manyorms and rom a variety o perspectives. This brie presents our perspectiveon evaluating community organizing based on the evaluations we have conducted
to date. We are still learning, and there is much we would like to develop urther.
How can we provide more useul, usable tools that t with organizing systems and
culture? How can we nd ways to get valid eedback to organizers more quickly?
How can we balance the needs and expectations o unders, organizers, and
community leaders? We see this brie as part o a continuing conversation on how to
evaluate community organizing in a way that benets all involved. We look orward
to more discussions that validate, challenge, or rene our thinking, and we welcome
new voices to the conversation.
As we continue our evaluation eorts, we will work to uncover and communicate
new lessons or the eld, and bring the voices o organizers into the conversation. As
part o our recent evaluation capacity- building eorts with community organizing
grantees o The Caliornia Endowment, we developed a series o videos in which
Success for
organizers
encompasses
leadership
development and
positive shifts in
power dynamics
as well as desired
policy change.
7/29/2019 Evaluating Community Organizing (2)
14/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 13
organizers discuss their experiences developing and conducting evaluations o their
eorts. These can be ound at http://www.blueprintrd.com/do-it-yoursel-community-
organizing-evaluation. We invite you to take a look and to join our conversation.
About the Authors
Catherine Crystal Foster is an independent strategist, advocate, and evaluator
with over 20 years experience in the nonprot community. She serves a strategicpartner o the San Francisco-based philanthropy strategy rm Blueprint Research
& Design, where she co-leads evaluations o policy, advocacy, and community
organizing initiatives. Catherine also partners with nonprot organizations and
unders to advance their policy and advocacy activities. These partnerships are
grounded in Catherines experience directing child advocacy and welare policy
initiatives beore opening her consulting practice. She has also served on numerous
nonprot boards o directors and is an active participant in civic aairs in her home
city o Palo Alto, Caliornia. Catherine can be reached at catherine@blueprintrd.com.
Justin Louie is a senior analyst at Blueprint Research & Design, and has spent the
last 13 years developing and conducting evaluations or nonprots, or-protcorporations, oundations and government agencies. He leads Blueprints practice
area in organizational learning and evaluation. Early on in his career he was a
community and youth organizer ocused on HIV prevention, and he has served
on the boards o a number o grassroots, state, and national policy and political
organizations. He has degrees in sociology (BA) and public policy (MPP) rom the
University o Caliornia, Berkeley. Justin can be reached at justin@blueprintrd.com.
About Blueprint Research & Design, Inc.
Blueprint Research & Design helps grantmaking oundations, individual andamily donors, and philanthropic networks achieve their missions. Blueprint
believes that philanthropy is a critical part o our global society and that research
and systems thinking help individual organizations best achieve their goals. Services
include program strategy and design; organizational learning and evaluation;
and philanthropic industry analysis. Blueprint has been a leader in the advocacy
evaluation eld, and with support rom The Caliornia Endowment, produced a
seminal report on the eld in 2005 titled The Challenge of Assessing Policy and
Advocacy Activities: Strategies for a Prospective Evaluation Approach. That report
can be ound at www.blueprintrd.com.
About the Center or Evaluation Innovation
The Center or Evaluation Innovation is pushing evaluation practice in newdirections and into new arenas. The Center specializes in areas that are hard tomeasure and where resh thinking and new approaches are required. This includes
advocacy and policy change, communications, and systems change eorts. The
Center works with other organizations to develop and then share new ideas and
solutions to evaluation challenges through research, communications, training
development, and convening. www.evaluationinnovation.org.
This brief was developed
with generous supportfrom The California
Endowment, with
special thanks to Astrid
Hendricks and Gigi
Barsoum.
Acknowledgements
7/29/2019 Evaluating Community Organizing (2)
15/16
Grassroots Action and Learning or Social Change: Evaluating Community Organizing 14
About Advocacy Evaluation Advances
In 2009, 120 advocates, evaluators and unders gathered at The CaliorniaEndowments Center or Healthy Communities or two days o thought-provokingpresentations and discussions on recent advocacy evaluation advances. The
convening, sponsored by The Caliornia Endowment with support rom The Atlantic
Philanthropies and Annie E. Casey Foundation, ocused on real-lie experiences
with advocacy evaluation and what has been learned rom testing dierent toolsand approaches in this emerging eld over the last several years. It also ocused on
challenges that still must be addressed, and identied priorities or the eld moving
orward. Some o the content and ndings eatured in this brie were presented and
discussed during the convening. To access other convening and presenter resources,
visit theAdvocacy Evaluation Advances web page at www.calendow.org/article.
aspx?id=3774.
7/29/2019 Evaluating Community Organizing (2)
16/16
1 This is a form of participatory monitoring and evaluation developed by Rick Davies and Jess Dart that involves the participants collectionand discussion of stories about the most significant changes resulting from a program or action. See http://www.mande.co.uk/docs/MSCGuide.pdf.
Sample Benchmarks and Data Collection Methods or the Core Components o Organizing
O Compo Bcmk D Coc Mod
PartiCiPatiOn anDMeMBershiP
COnstituent leaDershiPanD POwer
OrganizatiOnal POwer
Organizing wins
Meaningful iMPaCtfOllOwing wins
OrganizatiOnalCaPaCity
refleCtiOn anDinnOvatiOn
Changes in numbers, demographicsor location o members
Changes in attendance (numbers, types
o events, who attends)
Changes in attitudes, skills, andknowledge
Changes in sel-esteem and sel-eicacy
Changes in stature within communityor among decision makers
Development o relationships withdecision makers, media, and inluentialigures
Changes in stature within communityor among decision makers
Changes in membership
Changes in turnout to events
Policy wins
Policy wins
Shits in norms or content o debate
Holding the line against negativeactions
Implementation o policies
Changes in practices
Public accountability or action orinaction
Sustained shits in norms or contento debate
Impact on community
Changes staing
Changes in inrastructure
Changes in skills
Changes in resources
Building on and systematizing internalprocesses
Inusing data and documentation intorelection
Use o data in reinement o strategyor tactics
Membership tracking (includingdemographic and geographic ino)
Attendance tracking
Tracking elements o leadership growth
Organizer check-ins and debries
Documenting 1-to-1s
Journaling/portolios
Focus groups
Power analysis
Relationship/champion tracking
Base-building/mobilization tracking
Media tracking
Policy developments tracking
Interviews
Critical incident debries or case studies
Policy tracking
Collection o archival documents
Media tracking
Critical incident debries or case studies
Policy implementation tracking
Community indicators tracking
Action research (accountability surveys,interviews, ocus groups)
Interviews
Critical incident debries or case studies
Organizational capacity assessments
Most Signiicant Change1
Interviews and check-in calls ormeetings
Interviews and check-in calls ormeetings
Collection o assessment documents orexamination o systems