Vibrant Communities Canada: Measuring Impact

39
Vibrant Communities Canada: Measuring Impact Donna Jean Forster-Gill Tamarack-An Institute for Community Engagement

description

By: Donna Jean Forster-Gill, Tamarack-An Institute for Community Engagement

Transcript of Vibrant Communities Canada: Measuring Impact

Page 1: Vibrant Communities Canada: Measuring Impact

Vibrant Communities Canada: Measuring Impact

Donna Jean Forster-Gill Tamarack-An Institute for Community Engagement

Page 2: Vibrant Communities Canada: Measuring Impact

Why Vibrant Communities?

Page 3: Vibrant Communities Canada: Measuring Impact

The Impetus

Page 4: Vibrant Communities Canada: Measuring Impact

The Complex Nature of Poverty

“Poverty is a complex issue. There is no single cause and no one solution. Its successful reduction, and ideally its eradication, require a set of linked interventions undertaken by all orders of government working in collaboration with communities.”

Poverty Policy Sherri Torjman, Caledon Institute of Social Policy October 2008

Page 5: Vibrant Communities Canada: Measuring Impact

What is Vibrant Communities?

Page 6: Vibrant Communities Canada: Measuring Impact

A Pan-Canadian initiative exploring comprehensive, multi-sector approaches to poverty reduction

Launched in 2002 by three national partners

•  Tamarack: An Institute for Community Engagement •  The Caledon Institute of Social Policy •  The J.W. McConnell Family Foundation

Page 7: Vibrant Communities Canada: Measuring Impact

Vibrant Communities An experiment designed to test a specific

way to address the complex realities of poverty through local level action.

Theory of Change: Guided by 5 principles & assisted by extra supports provided by national sponsors –

local organizations and leaders could revitalize poverty reduction efforts in their

communities and generate significantly improved outcomes.

Page 8: Vibrant Communities Canada: Measuring Impact

We believed that using a comprehensive, multi-sector approach communities can …

•  Raise the local and national profile of poverty

•  Build a constituency for change

•  Encourage collaborative ways of working

•  Begin to shift the systems underlying poverty

•  Generate substantial changes for a large number of people living in poverty.

Page 9: Vibrant Communities Canada: Measuring Impact

Who are the Vibrant Communities?

Page 10: Vibrant Communities Canada: Measuring Impact

Part One – Exploring Principles

13 Vibrant Communities

+ Niagara Region

Page 11: Vibrant Communities Canada: Measuring Impact

How does Vibrant Communities work?

Page 12: Vibrant Communities Canada: Measuring Impact

What are the 5 components of Vibrant Communities?

Page 13: Vibrant Communities Canada: Measuring Impact

Multi-Sector Collaboration

Business  •  Exper(se,  credibility  and  

voice,  connec(ons,  funding  and  other  resources,  leadership    

Nonprofit  Organiza2ons  •  Exper(se,  experience  on  

the  ground,  service  delivery,  ability  to  ramp  up  change  efforts      

Government  •  Exper(se,  connec(ons  to  

elected  officials,  funding  and  other  resources,  policy  change,  leadership    

Ci2zens  with  Lived  Experience    •  Exper(se  about  the  issues,  

prac(cal  and  relevant  solu(on,  leadership,  connec(ons  to  other  ci(zens    

Page 14: Vibrant Communities Canada: Measuring Impact

Comprehensive Thinking & Action

address the interrelated root causes of poverty rather than its various symptoms

Create Community Knowledge

Page 15: Vibrant Communities Canada: Measuring Impact

Community Asset Building

building on community strengths rather than focusing on deficits

Develop a Community Aspiration

Page 16: Vibrant Communities Canada: Measuring Impact

Community Learning and Change

embracing a long-term process of learning and change rather than simply undertaking a series

of specific interventions

Create a Framework for Change

Page 17: Vibrant Communities Canada: Measuring Impact

Poverty Reduction

a focus on reducing poverty as opposed to alleviating the hardships of living in poverty

Move towards Systemic Change

Page 18: Vibrant Communities Canada: Measuring Impact

Impact of Vibrant Communities?

Page 19: Vibrant Communities Canada: Measuring Impact

Collective Impact •  Common agenda •  Shared measurement

systems •  Mutually reinforcing

activities •  Continuous

communication •  Backbone support

organization

–  John Kania and Mark Kramer, Winter 2011

Page 20: Vibrant Communities Canada: Measuring Impact
Page 21: Vibrant Communities Canada: Measuring Impact

Measuring the impact of Vibrant Communities –

Shared Measurement

Page 22: Vibrant Communities Canada: Measuring Impact

Original Approach •  Logic Models, Outcomes and Outcome

Tracking. (Traditional Evaluation methods) •  Used for first 2-3 years of project

SPEED BUMP: •  Approach did not fit with what was

happening on the local level •  Shifted to developmental evaluation

approach

Page 23: Vibrant Communities Canada: Measuring Impact

Developmental Evaluation in Vibrant Communities Stream One –each local community to articulate the ‘theory of change’ guiding its work and to reflect annually on how those ideas were playing out in practice. Stream Two –communities to prepare brief stories describing their specific poverty reduction strategies: the challenge addressed, the strategy employed and the outcomes anticipated. These stories helped guide their outcome tracking efforts. Stream Three –communities complete semi-annual statistical reports focussed on two main targets: number of partners participating in their work and number of low-income households benefitting from their efforts. It also asked for an annual narrative report elaborating on the overall development of each community’s initiative, including its efforts to build community capacity and impact wider systems that contribute to poverty.

Page 24: Vibrant Communities Canada: Measuring Impact

What was captured: •  Stories and Lessons Learned

•  Expected and unexpected outcomes

•  Most significant change

•  Statistical Reporting and Narrative reporting focused on 5 types of outcomes initially.

•  Used sustainable livelihoods model with 5 outcomes areas, 13 indicators and many sub-indicators.

•  After 5 years, this was revised to add outcomes around community capacity building and policy and systems change

Page 25: Vibrant Communities Canada: Measuring Impact

Indicators of Community Change

• Changes in public policy • Changes in service and support systems • Changes in material resources • Changes in community-level assets

Policy and Systems Change

• Convening capacity • Multisectoral leadership • Collaboration • Community awareness

Community Capacity Building

• Personal assets • Physical assets • Social assets • Human assets • Financial assets

Individual and Household

Assets

Page 26: Vibrant Communities Canada: Measuring Impact

Social Assets

Human Assets

Financial Assets

Inner resources

• Self-awareness • Self-esteem and self-confidence • Hope and motivation

Basic material goods & services

•  Emergency supports •  Food, Housing •  Transportation •  Dependent care

Relationships and Networks

•  Civic participation •  Support networks

Income, Savings

•  Employment income •  Non-employment

income •  Savings and financial

assets •  Reduced debt/costs

Skills, knowledge, education & health

•  Health, Life skills •  Financial literacy •  Education •  Employment Skills

Page 27: Vibrant Communities Canada: Measuring Impact

Why Developmental Evaluation Worked for VC At the local level: required reflecting on the theory of change and upgrading it as required to better achieve desired outcomes, rapid response to a changing environment, and ability to capture the emerging insights and questions of participants. At the national level: it is about mining the on-the-ground experience of communities for patterns and themes that helped us understand the value of this approach to reducing poverty.

Page 28: Vibrant Communities Canada: Measuring Impact

Challenges in measuring the impact of the

Vibrant Communities Initiative

Page 29: Vibrant Communities Canada: Measuring Impact

Overall Challenges: •  The sheer scope of the work: Lots of time and energy

required

•  ‘attribution issues’: whose efforts are responsible for outcomes achieved or roles that different partners play.

•  The guiding ideas are often left implicit making it hard to assess their validity.

•  Evaluation processes need to be highly flexible due to emergent nature of the process.

•  Outcomes are integral, but are often difficult to define and measure.

•  The long-term focus must be reconciled with the need to track progress in the near- and mid-term.

•  appropriate adjustments must be made along the way

Page 30: Vibrant Communities Canada: Measuring Impact

Most significant challenge: The evaluation needed to accommodate substantial differences across the sites while continuing to meet collaborative objectives. National Level: •  Managing the volume of data •  Getting materials submitted on time •  Lack of evaluation advisory group to guide the

work •  More collaboration needed between evaluators

and local level •  Takes lots of time and concentrated effort

Page 31: Vibrant Communities Canada: Measuring Impact

Local Level Challenges: •  Keeping up with evaluation demands

•  Balance between hard numbers and story

•  National Level questions did not always work well at the community level

•  Staff turnover, inadequate evaluation training for the communities

•  Often off the side of the desk rather than designated staff person

•  Takes lots of time and concentrated effort

•  Not single organization but collaborations

Page 32: Vibrant Communities Canada: Measuring Impact

What have we learned?

Page 33: Vibrant Communities Canada: Measuring Impact

How much more information do we need to know that a hungry child will not do well in school? Stop admiring the problem and get on with the work.

Mark Chamberlain CEO, Trivaris

Page 34: Vibrant Communities Canada: Measuring Impact

Evaluating a comprehensive, multi-sector approach to poverty reduction is …

1.  Hard work, messy and time consuming

2.  Fluid, flexible and requires adapting to constantly changing conditions

3.  Requires outside eyes to hold the evaluation pieces.

4.  Orgs can benefit from training in evaluation

5.  Everyone has to develop the evaluation framework

Page 35: Vibrant Communities Canada: Measuring Impact

Common Success Factors •  Influential and credible convener(s)

•  Cross-sector, connected leadership table •  Challenging community aspiration

•  Clearly articulated purpose and approach

•  High degree of resident mobilization

•  Research which informs the work and captures shared impact

Page 36: Vibrant Communities Canada: Measuring Impact

Reporting to the Community – Continuous

Communication

Page 37: Vibrant Communities Canada: Measuring Impact

Vibrant Communities (2002 - 2010) Evaluation Report

Reflecting on Vibrant Communities: 2002-2006

Understanding the Potential & Practice of Comprehensive, Multi-sector Efforts to Reduce Poverty - The Preliminary Experiences of the Vibrant Communities Trail Builders

In From the Field - Exploring the First Poverty Reduction Strategies Undertaken by Trail Builders in the Vibrant Communities Initiative

To learn about the background of collecting the VC by the Numbers reports: http://tamarackcommunity.ca/downloads/vc/VC_By_the_Numbers_FAQs_032511.pdf

Page 38: Vibrant Communities Canada: Measuring Impact

Where are we headed?

The Landscape has Changed…since 13 cities began to experiment…

•  Municipally: 84 collaborative poverty reduction roundtables have

connected to Vibrant Communities •  Provincially: 11 provinces and territories have or are developing

poverty reduction strategies •  Federally: A new all-party Roundtable has been formed to focus on

poverty, the Government of Canada – HUMA committee, Senate Roundtable on Cities and Federation of Canadian Municipalities have identified poverty as a critical issue

Page 39: Vibrant Communities Canada: Measuring Impact

Our Aspiration:

Imagine…100 cities reducing poverty

TOGETHER