Stanford University Press, April 2017

Theory, models, and

instruments to conduct

effective advocacy and policy

change evaluation

Annette L. Gardner, PhD, UCSF

Claire D. Brindis, DrPH, UCSF

Aspen Institute

Washington, DC

June 14, 2017

‘TURBULENT POLICY LANDSCAPE’

OUR PROJECT

• Curate the field of advocacy and policy change (APC)

evaluation:

2014 Aspen/UCSF APC Evaluation Survey (N = AEA 106

evaluators, 7% AEA response rate, 9% APC TIG)

Six evaluation case studies

Review of tool-kits, models, instruments

• Strengthen the link between the scholarship on advocacy,

public policy, nonprofit management, and evaluation practice

DESIRED OUTCOMES

• Advance individual evaluation practice, particularly

Substantive knowledge of ‘advocacy’ and ‘policy’

Appropriate and rigorous designs

Partnerships with advocacy, funders, decision-makers

• Advance the field of advocacy and policy change evaluation

“Bringing to bear scholarship, models, resources, and examples of useful evaluation designs and tools so that advocacy can fulfill its vision more

effectively.”

THEORY: POLICY AND POLICYMAKING

• Achieving systems-level change to improve social and

environmental conditions

• Complex, uncertain policymaking process and political context.

• Evaluators have robust foundation—theories of change, body

of APC evaluation practice— on which they can draw

THEORY: ADVOCAY

• A broad swath of society—individuals, organizations,

policymakers, groups, social movement. But not all voices are

equal.

• A vast cachet of advocacy tactics with which to mobilize a

community, advance a policy.

“Championing or supporting a cause or a policy goal.”

(Obar, Zube, and Lampe 2012)

CURRENT EVALUATION PRACTICE:

ASPEN/UCSF APC EVALUATION SURVEY

• Early adopters of new instruments, emerging

models

• Low use of experimental designs—6 percent

• Gaps in content expertise—policy areas, political

context

• Practice dominated by health policy but APC

evaluators work in multiple policy arenas

• Funding—a perennial concern

Advocacy Activities Percent of Aspen/UCSF

Survey Respondents that

Focus on Tactic

Mobilizing Citizens and Organizing Advocacy Allies

Coalition building 59%

Community organizing (also referred to as

community mobilization)

52%

Protests or demonstration 6%

Expanding Public and Policymaker Awareness

Public awareness campaigns 59%

Public will campaigns 36%

Media advocacy 48%

Voter outreach 7%

Influencing Policymaker Support

Influencer/influential education 48%

Policymaker education 46%

Champion development 36%

Political will campaigns 32%

Lobbying 25%

Researching and Monitoring Policy

Policy analysis and research 55%

Regulatory feedback 21%

Budget monitoring 20%

Model legislation 18%

Litigation 2%

CURRENT EVALUATION PRACTICE:

SIX EVALUATION CASES

• The Initiative to Promote Equitable and Sustainable Transportation (2008 –2013) Adoption of policies for equitable and sustainable transportation options in the U.S.

• The Let Girls Lead program (2009 – present). Creation of a global movement of leaders and organizations advocating for adolescent girls’ rights.

• The GROW Campaign (2012 – present). A multi-national campaign to tacklefood injustice and build a better food system that sustainably feeds a growing population.

• The International Lands Conservation Program (1999 – present). Conserve old-growth forests and extend wilderness areas in Canada and Australia;

• Project Health Colorado (2011-2013), a public will building campaign that engaged individuals and organizations in a statewide discussion abouthealth care; and

• The Tribal Tobacco Education and Policy (TTEP) Initiative (2008-2013). Pass or expand formal and informal smoke-free policies at five Tribes.

SAMPLE CASE EVALUATION:

THE TRIBAL TOBACCO EDUCATION AND POLICY (TTEP) INITIATIVE

• Advocacy strategy and tactics: local-level advocacy capacity building,

community education, and passage of smoke-free policies

• Design:

Formative and summative. Included a logic model

Participatory approach to facilitate strategic learning (e.g., “sharing

sessions”) sensitive to cultural context and historical antecedents

Mixed-methods: web-based monthly data collection, media analysis,

technical assistance survey, dialoguing sessions, spider diagrams, Most

Significant Change stores, Aspen Policy Mapping tool

• Dissemination: Site reports, 2-page Community Change stories, annual

meetings at sites to reflect and celebrate. Shared locally and nationally.

APC EVALUATION DESIGN GUIDING PRINCIPLES

Partner with funders, advocates early and often

Determine the evaluator role

Know the initiative and its context—clarify terms and definitions

Embrace initiative complexity and consider different evaluation models

Use a theory of change and/or logic model to advance stakeholder understanding

Refer to the repository of advocacy and policy change outcomes

Balance attribution and contribution, prove vs. improve

Address challenges to design and issues of rigor

Engage in effective, continuous communications

FRAMING THE APC EVALUATION

Strengthening advocacy,

grant-making, and

policymaking.

Traditional and unique evaluation methods

Experimental design

Quasi-experimental design

Non-experimental designs

Evaluation Frameworks and

strategies, e.g.,

Developmental Evaluation

Theory of change

Systems thinking

Empowerment evaluation

Etc.

Tailored APC

inputs,

outputs,

outcomes,

impacts and

indicators

AEA Guiding

Principles for

Evaluators

TRADITIONAL AND UNIQUE EVALUATION

INSTRUMENTS

Advocacy capacity:

Expertise

Partnerships

Leadership

Resources

Problem

recognition:

Public and

policymaker

awareness

Agenda setting:

Issue saliency and

policymaker support

support

Policy formulation:

Analysis of policy

solutions

Policy adoption:

Policymaker support

Selection of policy

proposal

Policy

implementation:

Developing the

blueprint for action

Policy evaluation:

Systems change

Changing Lives

Expanding Democracy

Social Norms

Surveys

Interviews

Observations

Case Studies

Advocacy capacity

assessments

Policymaker instruments

Media analyses

Financial analyses

Policy analysis

Document review

Policymaking

Stage Model

ROLE OF THE EVALUATOR

• Informing advocate, funder strategic learning to increase

advocacy effectiveness

• Building an evaluation culture and strengthening advocate

evaluation capacity

• Informing funder grant-making

• Educating decision-makers

EVALUATOR AS ADVOCATE?

ADVANCING INDIVIDUAL EVALUATION

PRACTICE, ADVANCING THE FIELD

• ‘Mindful’ evaluation practice – honest self-appraisal of

advocacy, political, and policymaking acumen

• Bridge-building to advance the field

• Creating vehicles and venues for sharing

• Link evaluation practice and findings to the scholarship on

advocacy effectiveness, models of change

THANK YOU!

• Aspen Planning and Evaluation Program

• Center for Evaluation Innovation

• Community of APC evaluators

• Evaluation ‘pioneers’

• Advocates and funders

For more information:

Annette L Gardner, PhD

Annette.gardner@ucsf.edu

Claire D. Brindis, DrPH

Claire.brindis@ucsf.edu