Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and...

63
© The Wellesley Institute www.wellesleyinstitute.com Bob Gardner The ACHIEVE Research Partnership: Action for Health Equity Interventions Dec 2, 2009

Transcript of Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and...

Page 1: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

© The Wellesley Institutewww.wellesleyinstitute.com

Bob Gardner

The ACHIEVE Research Partnership: Action for Health Equity Interventions

Dec 2, 2009

Page 2: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• understanding the scale, impact and roots of existing health disparities in Ontario

• why we need a comprehensive health equity strategy to address these pervasive disparities:• macro level social and economic policy• within the health system• through specific service interventions

• analyzing where evaluation fits:• driving more effective equity-focused planning• underpinning more effective program and service interventions • grounding more effective and innovative collaborations• supporting strategic coherence across all these levels

2

Page 3: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• there is a clear gradient in health in which people with lower income, education or other indicators of social inequality and exclusion tend to have poorer health + major differences between women and men

• the gap between the health status of the best off and most disadvantaged can be huge – and damaging• difference btwn life expectancy of top and bottom income decile in

Canada = 7.4 years for men and 4.5 for women• more sophisticated analyses add the pronounced gradient in morbidity

to mortality → taking account of quality of life and developing data on health adjusted life expectancy

• even higher disparities btwn top and bottom = 11.4 years for men and 9.7 for women (Statistics Canada Health Reports Dec 09)

• in addition, there are systemic disparities in access to and quality of care within the healthcare system

3

Page 4: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

4

Page 5: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

5

Page 6: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

6

Page 7: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

7

Page 8: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

8© The Wellesley Institute

www.welleseyinstitute.com

• clear research consensus that roots of health disparities lie in broader social and economic inequality and exclusion

• impact of inadequate early childhood development, poverty, precarious employment, social exclusion, inadequate housing and decaying social safety nets on health outcomes is well established here and internationally

• real problem is differential access to these determinants – many analysts are focusing more specifically on social determinants of health inequalities

Page 9: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

9

Page 10: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

10

Page 11: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

Determinants interact and intersect with each other -- in a constantly changing and dynamic system

In fact, through multiple interacting and inter-dependent economic, social and health systems

Determinants have a reinforcing and cumulative effect on individual and population health

11

Page 12: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• Health disparities or inequities are differences in health outcomes that are avoidable, unfair and systematically related to social inequality and disadvantage

• This concept:• is clear, understandable and actionable

• identifies the problem that policies will try to solve

• is tied to widely accepted notions of fairness and social justice

• The goal of health equity strategy is to reduce or eliminate socially and institutionally structured health inequalities and differential outcomes

12

Page 13: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• a positive and forward-looking definition = equal opportunities for good health

• health equity is a broad concept that also prioritizes diversity:• reflecting the increasing diversity of Ontario society and the fact that

racism and ethno-cultural differences are important determinants of health disparities

• recognizing that services that reflect and speak to the diversity of cultures -- cultural competence – are essential to an equitable system

• and can encompass equity-focused health promotion• recognizing that vulnerable populations face more complex and

serious barriers to good health• recognizing that programs and plans need to always take this social

context and constraints into account

• achieving health equity would extend far beyond enhancing individual and collective well-being

13

Page 14: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

14

Page 15: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

15

• health disparities can seem so overwhelming and their underlying social determinants so intractable → can be paralyzing

• think big and think strategically, but get going• make best judgment from evidence and experience• experiment and innovate • learn lessons and adjust – why evaluation is so crucial • gradually build up coherent sets of policy and program

actions – and keep evaluating

• need to start somewhere – and we’re in health systems

Page 16: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• need action addressing health disparities at multiple levels:• macro social and economic policy• broad strategic frameworks within health and other

systems• policies and strategies for particular issues – chronic

conditions, primary care, e health• specific service interventions geared to specific outcomes

or places

• interventions at every one of these levels are complex• impact is inter-dependent and contingent → need

coherent overall strategy

16

Page 17: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• we want policy and program interventions to have:• the most impact in reducing disparities

• that sustain that impact over the long-term

• reducing overall social and economic inequality may be the most significant single way to reduce health disparities

→need to reduce unequal distribution of social and economic resources – of SDoH

→ requires a significant commitment and re-orientation of social and economic policy and fundamental institutional arrangements

17

Page 18: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• there is always much to be learned from policies, programs and initiatives in other jurisdictions

• a number of countries have made lessening health disparities a top national priority and have developed cross-sectoral policy frameworks and/or action plans:• England, Scotland, Australia, New Zealand

• many European countries

• also increasing international and high-level attention:• WHO Commission on Social Determinants of Health

• European Union, with its Closing the Gap project to tackle health disparities

• look broadly for policy solutions, and adapt flexibly to local/provincial circumstances

18

Page 19: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• coordinated national policy to reduce health disparities by reducing the number of people at risk of social and economic vulnerability

• national public health strategy has 12 key objectives –five of which, defined as fundamental to all the others, are about improving social and economic determinants • also focus on inclusive labour market, anti-discrimination, childcare,

affordable housing and other policies• equitable access to improved health care was seen to be just one part

of this broader package

• emphasized partnerships with community service providers and organizations – in both policy development and service delivery

19

Page 20: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• reducing health disparities – and underlying social and economic inequality – are complex challenges

• developing and mobilizing health policy is a classic ‘wicked problem’:• the issue crosses policy fields and jurisdictional boundaries

• involves many govt and external stakeholders

• is shaped by forces beyond govt control

• is necessarily long-term

• won’t align neatly with party/govt interests and electoral cycles

20

Page 21: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• but goal is reducing health disparities – is that ultimate evaluation criteria?• disparities do not appear to be lessening in countries with good

strategies…. but these effects are necessarily long-term?• no evidence that comprehensive policies work is the wrong conclusion …. but have not been evaluating properly to conclude interventions don't

work

• Britain has reviewed its comprehensive policies:• no decline in disparities – perhaps worsening• mixed results on specific objectives – some on target, some not• can't conclude that interventions don’t work – too soon to tell• but did conclude that evaluation was not properly built into policy and

program design

21

Page 22: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• might need more modest immediate to mid-term evaluation approaches and objectives – more process than outcomes?

• is there coherence in strategies and objectives across policy fields and departments?

• more specifically, comprehensive strategies are generally accompanied by efforts for greater ‘joined-up’ government coordination and collaboration

• can evaluate processes

• are priorities and directions aligned towards common goals and vision?

• were the right stakeholders involved – in the right ways – in developing policy?

• is strategy based on best available theory, evidence and experience?

• is strategy comprehensive – are critical components missing or under-developed?

22

Page 23: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• even though roots of health disparities lie in far wider social and economic inequality

• how the health system is organized and how services and care are delivered is still crucial to tackling health disparities

• many countries have developed comprehensive multi-sectoral strategies to reduce health disparities

• in all of them, transforming the health system is an indispensable element, including:• reducing barriers to equitable access to high quality care• targeted interventions to improve the health of the poorest, fastest• up-stream investments in primary and preventative care directed to

most vulnerable• delivering these services in coordinated way at community/local level

23

Page 24: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

1. it’s in the health system that the most disadvantaged in SDoH terms end up sicker and needing care

• equitable healthcare and proactive health promotion can help to mediate the harshest impact of the wider social determinants of health on health disadvantaged populations and communities

2. in addition, there are systemic disparities in access and quality of healthcare that need to be addressed

• people lower down the social hierarchy tend to have poorer access to health services, even though they may have more complex needs and require more care

• unless we address inequitable access and quality, healthcare and health promotion could make overall disparities even worse

• at the least, the goal is to ensure equitable access to care/support for all who need it, regardless of their social position

24

Page 25: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

while health disparities are pervasive and deep-rooted, they can be changed through policy and program action

comprehensive strategy developed in 2008 for Toronto Central LHIN

many recommendations have been acted on

other LHINs are also prioritizing and moving to address health disparities

25

Page 26: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• given impact of wider SDoH, cannot expect even the most progressive and equitable health system to lead to major reductions in disparities of outcomes

• goal is to ensure equitable access to high quality healthcare regardless of social position

• can do this through a two pronged strategy :1. building health equity into all health planning and delivery

• doesn’t mean all programs are all about equity• but all take equity into account in planning their services and

outreach2. targeting some resources or programs specifically to addressing

disadvantaged populations or key access barriers• looking for investments and interventions that will have the highest

impact on reducing health disparities or enhancing the opportunities for good health of the most vulnerable

26

Page 27: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• one critical component of this strategic approach is good planning

• to develop effective planning, we need:• clear strategy• coherent approach• repertoire of effective tools and techniques• with support for planning authorities and

practitioners to effectively use them• good actionable information

• and then drilling down: what is our ‘theory’ of how equity-focused planning works?

27

Page 28: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

taking account

of social constraints &

conditions

not just individual

programs but coordination,

partnerships & collaboration

28

Page 29: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

enhanced access to

health promotion for

most disadvantaged

up-stream heath conditions & opportunities

improve fastest for those in

greatest need

29

Page 30: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• processes and constraints are complex, and outcomes uncertain and unpredictable, at each of these junctures

• and all of this varies by context:• particular communities or neighbourhoods – with their different

health challenges and needs• particular population health and service landscape in specific areas• further specified by health condition or concern (e.g. mental health)• existing municipal and local polices and traditions• community resilience, connectedness, organizing and traditions

• we don’t really know what works best at each of these junctures (let alone cumulatively) or in varying contexts

→ need to build evaluation in from the start to learn

30

Page 31: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

1. quick check to ensure equity is considered in all service delivery/planning

2. take account of disadvantaged populations, access barriers and related equity issues in program planning and service delivery

3. assess current state of provider organization

4. determine needs of communities facing health disparities

5. assess impact of programs/interventions on health disparities and disadvantaged populations

1. simple equity lens

2. Health Equity Impact Assessment – has been piloted in Toronto and MOHLTC is considering wider roll-out

3. equity audits and/or HEIA

4. equity-focused needs assessment

5. equity-focused evaluation

31

Page 32: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

1. screening – projects where HEIA would be useful

2. scoping – which pop’n and health effects to consider

3. assessing potential equity risks and benefits – specifying particular pop’n

4. developing recommendations –to promote positive or mitigate negative effects

5. report results to decision makers

6. monitoring and evaluation – of effectiveness of recommendations

while HEIA is sometimes promoted as easy-to-use ‘first-pass’ planning tool

does not mean it is only about 1 -- 3

experts argue core of HEIA is in fact 4 – assessing & developing recommendations to address equity implications

32

Page 33: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• a premise of the draft Ontario HEIA – and many others – is that:• assessing the potential impact of initiatives on particular populations

requires solid understanding of that population's health status, needs and context

• this can benefit from ongoing community engagement with the population and/or specific needs assessment

• analyzing possible mitigation strategies will also benefit from engaging the affected population in designing the necessary service changes

• similarly, the stage of monitoring and assessing the impact of the initiative – and how HEIA contributed -- also needs:• research and input from the affected population on impact• outcome data stratified by population and determinants

33

Page 34: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• comprehensive policies on health equity from other countries all include:• setting clear strategic objectives and targets • defining indicators – that build on available reliable data

and make the most sense in the particular context• closely monitoring progress against the indicators or

targets• disseminating the results widely for public scrutiny

• key driver of system change = building equity targets and objectives into routine performance management and provider planning

34

Page 35: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• we know there will be broad targets for priorities such as diabetes and mental health → build equity into these targets:• several LHINs have identified areas where diabetes incidence is

highest → equity target = reduce differences in incidence, complications and rates of hospitalization among areas across Central

• similarly, systemic inequities in depression → equity target = reduce those differences by gender, income, region

• looking up-stream → equity target = ensuring take-up of health promotion programs does not vary inequitably by income level, neigbourhood, gender, race, etc.

• many programs assess their services through client satisfaction surveys and similar methods • providers look for high and improving satisfaction → equity target =

reduce any differences in satisfaction by gender, income, ethno-cultural background, etc.

35

Page 36: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• MOHLTC flow funds to LHINS, who in turn allocate funds to health service providers

• these funding programs, and the accountability agreements that go with them with providers, provide an opportunity to build in specific equity expectations

• expectations will vary by community and provider, but could include:• undertaking appropriate equity-focused planning• providing sufficient services in languages of community and

appropriate interpretation• identifying areas where access to services is inequitable and

developing plans to address barriers and gaps• ensuring service utilization matches appropriately with demography

and needs of their catchment profile• developing specific services or outreach to particular disadvantaged

populations – homeless, isolated seniors, etc.

36

Page 37: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• greater chance of success for equity strategy if aligned with provincial priorities

• MOHLTC and MHP priorities such as mental health and diabetes are particularly sensitive to social conditions• chronic disease prevention and management programs cannot be successful

unless they take account of social conditions and constraints• various supports designed to enable people with mental health challenges to

live in the community also need to take into account their social conditions

• Wellesley and Canadian Mental Health Association–Ontario partnered on input to current discussions about mental health strategy:• stressed that programs had to take account of SDoH in ways discussed here• highlighted healthy communities approach • highlighted the potential of specific planning tools such as Mental Health

Impact Assessment

37

Page 38: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• identify key priorities and imperatives for wider health system transformation

• equity also supports other system drivers

• better access to primary care is key to reducing pressure on ER wait times and ALC

• reducing language barriers to good care through better interpretation can reduce mis-diagnoses and over-prescriptions → enhanced quality and cost effectiveness

• reducing higher expenditures on vulnerable populations due to health disparities → can contribute to overall and sustainability

38

Page 39: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• a promising direction several LHINs have taken up is to have providers undertake specific equity planning exercises designed to:• identify access barriers, disadvantaged populations, service gaps and

opportunities in their catchement areas and spheres• develop programs and services to address those gaps and better meet

healthcare needs of disadvantaged communities

• these provider plans have the potential to:• raise awareness of equity within the organizations• build equity into planning, resource allocation and routine delivery• pull their many existing initiatives together into a coherent overall

equity strategy• build connections among providers for addressing common equity

issues

39

Page 40: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

http://www.torontoevaluation.ca/tclhin/index.html

40

Page 41: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

41

• target services to specific areas or populations:

• those facing the harshest disparities – to improve the health of the worst off fastest

• or those most in need of specific services

• or to the worst barriers to equitable access to high-quality services

• this requires sophisticated analyses of the bases of disparities:

• i.e. is the main problem language barriers, lack of coordination among providers, sheer lack of services in particular neighbourhoods, etc.

• which requires good local research and detailed information – speaks to great potential of community-based research to provide rich local needs assessments and evaluation data

• involvement of local communities and stakeholders in planning and priority setting is critical to understanding the real local problems

Page 42: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• language was a major theme in Toronto hospital equity plans• we identified acting on this barrier as potential ‘quick win’ in

analysis of plans• TC LHIN is funding project to explore innovative ways to

streamlining and enhancing access to interpretation• build in evaluation:

• can track improvements in volume of interpretation provided• but need to match that against need• need to also assess quality• need to assess impact on improving access to (which?) services

for (which?) populations

42

Page 43: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

Diabetes Incidence, TC LHIN 2004/05

5.8

13.3

0

2

4

6

8

10

12

14

16

Low Income High Income

New

Cases/1

,000

Two fold difference in diabetes incidence between lowest and highest neighbourhoods.

Age Standardized Rates. Data Source: Ontario Diabetes Database, 2004/05 www.ices.on.ca/intool

43

Page 44: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

44

• investing in better chronic care prevention and management are vital elements of health reform and major provincial priority

• up-stream initiatives need to be planned and implemented through an equity lens• very clear gradient in incidence – and impact – of chronic conditions• some populations and communities need greater support to prevent

and manage chronic conditions– poor, Aboriginal and other vulnerable communities face greater incidence

and greater challenges in managing diabetes– at the same, time these communities tend to have less access to safe

open space and recreational facilities to encourage exercise– the Toronto diabetes atlas produced by ICES found that only 25% of in

low-income neighbourhoods participated in weekly sports – versus 75% form high-income

– built environment is also key -- Atlas found that people -n low-income areas walked more for transportation purposes but less for exercise

Page 45: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• a very interesting example is the integrated diabetes program developed out of the London InterCommunity Health Centre:– far greater incidence and impact in local Hispanic community

– CHC, community groups and others worked closely together to concrete services in these areas of greatest need

– language specific and culturally sensitive services

– preventative and promotion services offered where people went –e.g. shopping malls

– also saw that social conditions had to be addressed → referrals to social service support, advocacy around employment and other problems

45

Page 46: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• more emphasis on health promotion is vital to long-term sustainability of system and individual health• consistent data on variations of risk factors along the social

gradient• anti-smoking, exercise and other health promotion programs

need to explicitly consider the particular social, cultural and economic factors that shape risky behaviour in poorer communities– not just the usual focus on individual behaviour and lifestyle

• need to customize and concentrate health promotion programs for most disadvantaged

• if this isn’t done → universal programs can unintentionally widen disparities as better off take up programs more

• need to also build local community needs and a priority for disadvantaged into decisions on where to locate new programs

46

Page 47: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• hub-style multi-service centres in which a range of health and employment, child care, language, literacy, training and social services are provided out of single ‘one stop' locations

• Winnipeg Regional Health Authority and Manitoba Family Services and Housing have partnered on a new model to integrate health and social service delivery – one-stop access models in various communities to deliver a broad range of health and social services directly and to refer on to other agencies when services aren’t available

• Ontario provincial associations representing CHCs, mental health and community service agencies have been promoting idea -- including to LHIN CEO provincial planning table

47

Page 48: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

48

• British example of comprehensive policy: Health Action Zones and other models were designed to combine community economic development with targeted healthcare and social service improvements

• in Canada, some Regional Health Authorities have developed operational and planning links with local social services or emphasized community capacity building:• Saskatoon is developing cross-sectoral action on health equity:

• began from local research documenting shocking disparities among neighbourhoods

• focusing interventions in the poorest neighbourhoods – locating services in schools, relying on First Nations elders to guide programming, etc.

• wide collaboration among public health, municipality, business, community, Aboriginal and other leaders

Page 49: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• increasing international, Cdn and Ont interest in more strategic and realist evaluation

• figuring our what interventions and approaches work, for which populations, in what contexts and why

• also drawing on more community-based and participatory approaches

• this is absolutely crucial for equity strategy:• to identify ‘successful’ policies and programs that most effectively

reduce access barriers or support the most health disadvantaged populations

• to guide investment in directions that will have the most equity impact

49

Page 50: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• at immediate levels:• service and client objectives were built into program plan – were they met?• does service provide good quality care?• does it work for those with most complex needs or from most disadvantaged

circumstances?

• more equity-orientated:• if service is targeted to equity objectives -- does it support the particular

population or address the particular access barrier as anticipated• standard analysis of impact against objectives

• more generally:• can’t just measure activity – number or % of pop’n that participated in a

program• does this delivery help to lessen disparities in access for disadvantaged

populations or reduce access/quality barriers• need to measure health outcomes – even when impact only shows up in long-

term

50

Page 51: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• does the overall strategy deliver the right mix of:

• quick wins to build momentum and address pressing immediate problems

• longer-term and/or wider scale interventions

• are the many initiatives coordinated effectively?

• is it the particular service – better primary care – or the context/combination – delivered out of comprehensive community centres – that makes the difference?

• are interventions aligned coherently to build towards common strategic directions?

• how can we evaluate the cumulative impact of many equity-focused interventions – let alone of their degree of coherence and coordination?

51

Page 52: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• can’t go for simplistic indicators• e.g. lowering the overall incidence of diabetes• a number of researchers and LHINs have identified areas where diabetes

incidence is highest

→ equity target = reduce differences in incidence between populations or areas• need to evaluate if and how programs have met these equity objectives• in fact, we should expect a successful strategy to lead to incidence getting

worse in the short tem as more disadvantaged people at greater risk of diabetes are being identified and brought into the system to get care

• want to identify specific interventions or directions that have high equity impact:• ensuring access and use of primary health care does not vary inequitably by

income level, immigration status, neigbourhood, gender, race, etc. • in fact, concentrating services in most disadvantaged communities with

greatest needs

52

Page 53: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• a good service target has been proposed for diabetes = high/increasing % of people who get best standard care • there is clear clinical consensus on what this best standard care

constitutes and how to measure it→ equity target = reduce inequitable differences in quality care

• so we need to evaluate % who get the highest standard care by gender, income, ethno-cultural background

• and assess any inequitable variations in outcomes in terms of complications and rates of hospitalization

• many programs assess their services through client satisfaction surveys and similar methods • providers look for high and improving satisfaction → equity

target = reduce any differences in satisfaction by gender, income, ethno-cultural background, etc.

53

Page 54: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• if goal is to increase the % of kids who exercise regularly• equity target is to reduce the differentials in % of kids who

exercise by neighbourhood, gender, ethno-cultural background, etc.

• and that won’t be just a question of education and awareness but facilities and proactive empowerment of kids and communities in which they live

• if goal is increasing overall exercise rates and decreasing smoking → equity target = reduce those differences by gender, income, region

• drilling down: to achieve these equity targets, we need to: • ensure take-up of health promotion programs does not vary

inequitably by income level, neigbourhood, gender, race, etc. • in fact, take-up – and sticking with the programs – needs to be

highest in most disadvantaged communities with greatest needs

54

Page 55: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• don’t just want to evaluate results for those who make it through the doors of service providers

• but also disadvantaged populations who are not being reached

• need to assess who isn’t signing up?

• and once signed up for particular program, are there inequitable variations in who sticks with the program?

• need to differentiate those with greatest need = who programs most need to reach and keep to have an impact

→ develop funding and evaluation weighting that recognizes more complex needs and challenges of most disadvantaged, and builds this into incentive system

→ again, to guide investment in policy directions and program interventions that will have the strongest equity impact

55

Page 56: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• huge number of community and front-line initiatives already addressing equity and health promotion across province

• + equity focused planning will yield useful information on existing system barriers and the needs of disadvantaged populations, and on promising and successful program interventions

• we need to be able to:• collate and analyze all the useful intelligence gained from equity-focused

planning

• capture and share information on local initiatives, and build on local front-line insights

• share the resulting knowledge across regions – and beyond

• assess the most promising initiatives or directions

• scale up promising initiatives across the province where appropriate

• central role of evaluation in all this

56

Page 57: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

1. Where would you start and what would be your most important challenges in developing a comprehensive cross-sectoral strategy to reduce health disparities in ten years? How would you build in an effective evaluation framework?

2. Outline a strategy to reduce the incidence and impact of diabetes in Toronto and set up an evaluation framework for it.

3. Develop a plan and set up an evaluation framework for community-based programs in Parkdale to reduce the incidence and impact of diabetes.

57

Page 58: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• back to bigger picture

• following is a roadmap for comprehensive integrated policy action on determinants of health and health inequality

58

Page 59: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

1. look widely for ideas and inspiration from jurisdictions with comprehensive health equity policies, and adapt flexibly to Canadian, provincial and local needs and opportunities;

2. address the fundamental social determinants of health inequality – macro policy is crucial, reducing overall social and economic inequality and enhancing social mobility are the pre-conditions for reducing health disparities over the long-term;

3. develop a coherent overall strategy, but split it into actionable and manageable components that can be moved on;

4. act across silos – inter-sectoral and cross-government collaboration and coordination are vital;

5. set and monitor targets and incentives – cascading through all levels of government and program action;

59

Page 60: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

6 rigorously evaluate the outcomes and potential of program initiatives and investments – to build on successes and scale up what is working;

7 act on equity within the health system:

• making equity a core objective and driver of health system reform – every bit as important as quality and sustainability;

• eliminating unfair and inefficient barriers to access to the care people need;

• targeting interventions and enhanced services to the most health disadvantaged populations;

8 invest in those levers and spheres that have the most impact on health disparities such as:

• enhanced primary care for the most under-served or disadvantaged populations;

• integrated health, child development, language, settlement, employment, and other community-based social services;

60

Page 61: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

9 act locally – through well-focussed regional, local or neighbourhood cross-sectoral collaborations and integrated initiatives;

10 invest up-stream through an equity lens – in health promotion, chronic care prevention and management, and tackling the roots of health disparities;

11 build on the enormous amount of local imagination and innovation going on among service providers and communities across the country;

12 pull all this innovation, experience and learning together into a continually evolving repertoire of effective program and policy instruments, and into a coherent and coordinated overall strategy for health equity.

61

Page 62: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

• these speaking notes and further resources on policy directions to enhance health equity, health reform and the social determinants of health are available on our site at http://wellesleyinstitute.com

• my email is [email protected]

• I would be interested in any comments on the ideas in this presentation and any information or analysis on initiatives or experience that address health equity

62

Page 63: Making Evaluations Matter for 'Wicked' Policy Problems; Supporting Strategy, Policy and Interventions to Drive Health Equity

The Wellesley Institute advances urban health through rigorous research,

pragmatic policy solutions, social innovation, and community action.

63