“There’s nothing so practical as good theory”

Evaluating ‘Wicked’ Problems

Bob GardnerJuly 2011

Clear Starting Points• English evaluation leader Ray Pawson quotes sociologist Kurt Lewin that

“there’s nothing so practical as good theory”• Pawson wasn’t arguing for abstract theory, but for ensuring we are

always clear about • the assumptions we are making that underpin our work – whether planning a

specific service initiative or developing a broad multi-sectoral strategy• the pathways and factors that we assume will lead from the planning through

service delivery to the hoped-for impact• how we think all of this will vary depending upon the organizational, social or

policy context• there are many approaches and as many terms – theory of change,

program theory, framework for change• but the basic idea is to be really clear about starting points and premises

when planning any initiative • and it’s this set of assumptions, pathways and objectives in our theory of

change that we evaluate

2April 18, 2023 | www.wellesleyinstitute.com

The Problem to Solve = Systemic Disparities in Ontario

•there is a clear gradient in health in which people with lower income, education or other indicators of social inequality and exclusion tend to have poorer health + major differences between women and men•the gap between the health of the best off and most disadvantaged can be huge – and damaging•in addition, there are systemic disparities in access to and quality of care within the healthcare system

3April 18, 2023

‘Wicked’ Policy Problems• health inequities and their underlying social determinants of health are classic

‘wicked’ policy problems:• shaped by many inter-related and inter-dependent factors • in constantly changing social, economic, community and policy environments• action has to be taken at multiple levels -- by many levels of government,

service providers, other stakeholders and communities• solutions are not always clear and policy agreement can be difficult to achieve• effects take years to show up – far beyond any electoral cycle

• have to be able to understand and navigate this complexity to develop solutions • need comprehensive strategy to tackle the roots of health inequality in the

wider social determinants of health• from high-level national social and policy change to reduce inequality through

community-based innovation, cross-sectoral collaboration and mobilization

4April 18, 2023

…Need Theory• for complex social problems – from health inequities, through poverty

reduction, to neighbourhood renewal and comprehensive community initiatives -- we need to be able to:• identify the connections and causal pathways between multiple factors • articulate the mechanisms that we assume drive change in these factors and

the problem as a whole• within that, identify the crucial leverage points that will drive significant

change in the problem• develop strategies for moving these leverage points• specify the short, intermediate and long-term outcomes expected and the

preconditions for achieving them.

that is our theory of change

5April 18, 2023

… And Here’s Howhow to develop theory of change:

• basic principle is to pull group working on the issue together to answer those questions

• to clearly articulate our starting points – what we think are the key causal pathways, mechanisms for change and crucial leverage points

• these are our assumptions and goals for how we think program, community initiative or policy direction will work

• based upon best available evidence, learning and experienceand some wrinkles

• can’t have all the answers and can’t be too prescriptive – don’t wait for ‘perfect’ theory

• be as participatory as possible – especially if community-based

evaluation then tests these assumptions about change mechanisms and outcomes → contributes to further learning and development of the initiative

6April 18, 2023

Theories of Change for ‘Wicked’ Problem I: Health Inequities

complex multi-level strategies to tackle health gaps:

• international frameworks such as WHO Commission on Social Determents

• European Union and other international initiatives

• many individual countrieskey features in all:

• focus on inclusive labour market, anti-discrimination, childcare, affordable housing, social security and other macro policies

• targets and deliverables for relevant departments

• equitable access to quality health care is just one part of this broader package

• emphasized partnerships with community service providers and organizations – in both policy development and service delivery

Theory of change = all about reducing structured inequality :

• health inequities arise out of wider social determinants of health

• won’t solve through health reform alone• underlying structures of social and

economic inequality need to be addressed by new policies

• policies need to be aligned with the incentives and processes that drive government

• key levers will vary – e.g. could be reducing prevalence and impact of precarious employment

• has to involve collaboration and coordination across governments and with many community and non-government stakeholders

7April 18, 2023

Theories of Change for ‘Wicked’ Problem II: Community Level Action

Comprehensive community initiatives:• broad partnerships of local residents,

community organizations, governments, business, labour and other stakeholders

• coming together to address deep-rooted local problems – poverty, neighbourhood deterioration, health inequities, racism

• collaborative cross-sectoral efforts – not just for better coordinated service provision but for community development and capacity building

• Vibrant Communities – 14 communities across the country working to build individual and community capacities to reduce poverty

Theory of change:• poverty, social exclusion, etc are not

individual problems but are deeply rooted in fundamental social and economic inequality → even the best programs cannot get at these underlying structures

• underlying structures of inequality need to be addressed by new policy directions

• building community capacities is another direction to address structural inequalities:• drilling down – Vibrant Communities

works to build up individual skills and other assets as building block for community change

• broad collaborations are key to both enhancing community capacities and implementing policy changes locally

• broad coalitions also can highlight the deep-seated nature of problems → build community and public awareness and focus political advocacy

8April 18, 2023

Evaluating Complex Policy and Community Initiatives

• far too complex to pick apart all the causal relations and patterns of influence:• very difficult to attribute particular changes to particular components of the overall

initiative• will never meet RCT gold standard of proof – that approach can’t capture complexity• but that doesn't mean initiative is ineffective

• impact can take many years to show up• and that doesn’t mean nothing is happening

• traditional evaluation of one program in isolation or of a particular population among many will not capture this complexity

• so we evaluate our theory of change• Tamarack talks of framework for change – to highlight the breadth of activities and

stakeholders involved in comprehensive initiatives• we identified levers in our strategy – did they prove to be important in practice?• looking for indications that the change mechanisms unfold as we expected, that the

direction of causal influence and impact is as we expected ,etc• looking for evidence that outcomes anticipated are being achieved, and the change

mechanisms we anticipated have been important

9April 18, 2023

Build Use and Users Into Evaluation

• crucial to identify users of evaluation • what they need • the context in which they will interpret and use the results of evaluation

• policy makers need to:• solve social problems as they see them• determine where to invest scarce resources for maximum impact• avoid mixing up accountability requirements with evaluation• get beyond the assumption that randomly controlled trials are the ‘gold

standard’ of evidence• community leaders and members are interested in:

• the same – how to solve social problems as they see them – and live them• how to make case for reform/spending to policy makers• how to ensure programs and services meet the needs of community members

– and how to continually improve service delivery• both need to be able to determine – and demonstrate – what types of

initiatives work, for whom and in what contexts

10April 18, 2023

Case Study: Health Equity Strategy Into Action

• goal is to ensure equitable access to high quality healthcare regardless of social position

• can do this through a multi-pronged strategy:1. building health equity into all health care planning and delivery

• doesn’t mean all programs are all about equity• but all take equity into account in planning their services and outreach

2. aligning equity with system drivers and priorities3. embedding equity in provider organizations’ deliverables, incentives

and performance management 4. targeting some resources or programs specifically to addressing

disadvantaged populations or key access barriers• looking for investments and interventions that will have the highest impact

on reducing health disparities or enhancing the opportunities for good health of the most vulnerable

5. while thinking up-stream to health promotion and addressing the underlying determinants of health

11April 18, 2023

Equity-Focused Planning• a critical component of all these strategic directions is good planning• addressing health inequities requires a solid understanding of:

• key barriers to equitable access to high quality care• the specific needs of health-disadvantaged populations• gaps in available services for these populations

• great deal of emphasis on tools such as Health Equity Impact Assessment

• but can’t see particular tool as ‘magic bullet• to develop effective planning, we need:

• clear strategy• a coherent approach• a repertoire of effective tools and techniques• support for planners and practitioners to effectively use them• good actionable information

• and drilling down: what is our ‘theory’ of how equity-focused planning works?

12April 18, 2023

13April 18, 2023

taking account of social

constraints & conditions

not just individual

programs but coordination,

partnerships & collaboration

14April 18, 2023

enhanced access to

health promotion for

most disadvantaged

up-stream heath conditions & opportunities

improve fastest for those in

greatest need

Complexities At All Stages• processes and constraints are complex, and outcomes

uncertain and unpredictable, at each of these junctures• and all of this varies by context:

• particular communities or neighbourhoods – with their different health challenges and needs

• particular population health and service landscape in specific areas• community resilience, connectedness, organizing and traditions• further specified by health condition or concern (e.g. the determinants

and dynamics of mental health are different)• we need to know what works best at each of these junctures

(and cumulatively), what programs and frameworks work for particular populations, and what works in varying contexts

→ need to build evaluation in from the start to learn

15April 18, 2023