Outcome Mapping

 

Team MeetingFebruary

2010

2

Key evaluation challenges

• measuring development results

• establishing cause & effect in an open system

• timing

• encouraging iterative learning

• clarifying values

3

problem with « impact »

Impact Implies:

• Cause & effect• Positive, intended results• Focus on ultimate effects• Credits a single contributor• Story ends when program

obtains success

Development Implies:

• Open system• Unexpected positive &

negative results occur• Upstream effects are

important• Multiple actors create

results & need credit• Change process never ends

4

Recognizing that change is…

• Continuous

• Complex

• Non-linear

• Multidirectional

• Not controllable

5

OM - a flexible, multiple-use tool

• Planning

• Monitoring

• Evaluation

Thanks to Paul Crawford

Boundary Partners

Ultimate beneficiaries

RESOURCES

OUTPUTS

OUTCOMES

IMPACT

GeSCI

Strategic partners

Strategic partners

7

focus on direct partners

• Key concept is « boundary partners »

• The individuals, groups, and organizations you work with directly and anticipate opportunities for influence

Focus of Outcome Mapping

Behavioural Changes

Local partners / beneficiaries ownership

increases

Program influence decreases

Inputs Activities Outputs Outcomes Impacts

9

Why behaviour changes?

• To stress that development is done by and for people

• To illustrate that although a program can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected

10

Contribution not attribution

You can influence but not control change in your partners

11

• PLANNING: articulate goals & define activities

• MONITORING:assess program performance & partners’ outcomes

• EVALUTION:design & conduct a use-oriented evaluation

Primary uses

The Questions that Outcome Mapping will The Questions that Outcome Mapping will help to answer :help to answer :

• In designing and articulating the program’s logic

- What are our development goals?- How can our programme contribute to those development goals?- Who are our beneficiaries?- How can we help our beneficiaries contribute to the broader

development goals?

• Recording internal and external monitoring data

- How far have our beneficiaries progressed towards achieving outcomes?- What are we doing to support the achievement of those outcomes?- How well have we performed?

• Indicating cases of positive performance and areas of improvement

- What worked well? Why?- How can we maximise our contributions?

Kiely, Team Meeting, December 2007

The Questions that Outcome Mapping will The Questions that Outcome Mapping will help to answer help to answer (continued)(continued)::

• Evaluating intended or unexpected results

- Who changed / what changed? How?- If they did not change as expected, do we need to do something different or

reorient our expectations?

• Gathering data on the contribution that a program made to bringing about changes in its partners

- What activities / strategies were used?- How did the activities influence the individuals, groups, or institutions to

change?

• Establishing evaluation priorities and an evaluation plan

- What strategies, relationships, or issues need to be studied in depth?- How, and from where, can we gather relevant data?

Components of Outcome MappingComponents of Outcome Mapping

• A description of the macro-level changes to which program intends to contribute

vision, mission, boundary partners and outcome challenges, progress markers

• Strategy maps

• Organisational Practices Journal

• A monitoring framework outcome journal, strategy journal, performance journal

• Evaluation Plan

15

boundary partners

Those individuals, groups, and organizations with whom the program

• interacts directly to effect change• anticipates opportunities for influence

• engages in mutual learning

16

Boundary partners have boundary partners

program program’s bp bp’s bp

17

describe:

• the ideal behavioural changes (relationships, activities & actions) of a boundary partner

• how they will contribute ideally to the vision.

Outcome challenges

18

Example: Outcome Challenge Statement (IDRC)

Vibrant and dynamic regional and national knowledge building and sharing communities which will produce and share high quality evidence based research and experiences on ICT4E and KS. These communities will work with GeSCI in a pro-active and committed way to identify critical issues, develop appropriate solutions and support one another on a peer basis. They will make optimal use of the GeSCI resources, tools and expertise. They will commit themselves to participate in leadership capacity building for ICT4E and KS and acquire and apply critical identified skills in their respective fields for system transformation and innovation.

19

Progress markers

✓ A graduated set of statements describing a progression of changed behaviours in the boundary partner

✓ Describe changes in actions, activities and relationships leading to the ideal outcome; shows story of change

✓ Articulate the complexity of the change process

✓ Can be monitored & observed

✓ Permit on-going assessment of partner’s progress (including unintended results)

ExampleExampleBoundary partner Outcome challenge

Government officials and policymakers

Ministries of Education(s) Endorsement of the ICT initiatives, approval of the implementation details and establishing guidelines for future implementation.

Progress markers

Expect to see1. Negotiating with SchoolNet and other service providers on details of implementation2. Setting a priority list for implementation3. Taking control of the initiative

Like to see1. Efficient coordination of partnerships for delivery of ICT services to schools.2. Establishing guidelines for integrating ICT effectively into learning and teaching 3. Setting operational objectives for implementation based on the ICT policy for education4. Direct budgeting for expenditure by the Ministries of Education on the programme to be

specified in the Policy Implementation Plan

Love to see1. Creating the critical mass of ICT skills at school level to be able to realise the country’s

vision for the future.2. Setting specific guidelines on cost-effectiveness for ICT solutions for schools or putting

implementation out to tender to see who offers the best option3. Using the ICT initiative to facilitate a change in emphasis to student-centred learning and

teaching.

21

Taken together, progress markers

facilitate mid-course corrections and improvements

stimulate the program to consider how it can contribute to the most profound transformation possible

articulate the complexity of change

22

X

23

Strategy Maps

• Outlines the program`s approach in working with the boundary partner

• Breaks strategies down into different types so you can be as strategic and influential as possible

24

6 types of strategies

Aimed at the

Boundary Parnter

Aimed at the Boundary Partner`s

Environment

Strategy SupportivePersuasiveCausal

I-1

• Direct Output

I-2

• Arouse New Skills/ Thinking

I-3

•Supporter who guides change over

time

E-2

•Modify the information system

E-3

•Create / Strengthen a Peer

Network

E-1

•Alter physical or regulatory environment

25

Practice: Facilitation Questions

I-3I-2I-1

E-1 E-3E-2

What will bedone to producean “immediate”

output?

What networks/relationships willbe established

or utilized?

How willsustainedsupport,

guidance, ormentoring be

provided?

How will you usethe media orpublications?

What will bedone to build

capacity?

What will bedone to changethe physical or

policyenvironment?

M&E Planning

27

Use(s) and User(s) – Who will take action(s) or make decisions with the help of the M&E process or findings? What actions and/or decisions will be taken?

Questions Timing Methods & Data

Sources

Responsibility & Roles

Dissemination & Communication

Costs

What do you want to know about our programmes?

When are the findings needed in order to be useful to the user(s)?

When will the data be collected?

How will data be collected? (I.e., instruments) From what sources?

Who is going to lead the process?

Who needs to play what role to accomplish the work? (eg., who will manage collection? analysis? writing/reporting?)

How will the findings be communicated to the intended user(s)?

What should the final products look like in order to be useful?

Who else will be interested in the findings and should be informed of them?

What human and financial resources are needed and who contributes?

What resources are available or not?

28

different tools you can use to collect data

• Outcome/ strategy mapping journals

• most significant change methodology

• project cycle analysis

• activity system analysis

• any social science tool or method

• ….

30

outcome journal

To understand the change process in boundary partners. Collects information about:

• Story of change and reasons for change

• Unexpected changes

• The actors and factors that contributed to that change

• How we know the change occurred

• Learnings

31

outcome journal

Work Dating From/To:

Contributors to Monitoring Update:

Low =

Medium =

High =

Outcome Challenge:

LMH

Expect to see: Who?

3

1

2

32

set values for low, medium & high

• Can capture either quantity or quality of change(s):

– Quantity (number of boundary partners)– Quality (depth of change)

34

strategy journal

To understand how the programme is contributing to changes in boundary partners. Collects information on:

• Resources that have been invested• Strategies and Activities• Products• Effectiveness• Changes that need follow up• Learnings

35

strategy journal

Working Dating From/To:

Contributors to Monitoring Update:

Strategy to be Monitored:

Effectiveness?

(How did it help the boundary

partner?)

Outputs

36

how to use the journals

• alone or combined with other tools and methods

• focus groups, interviews, e-surveys, conference calls, revision of documents

• can incorporate quantitative and qualitative methods

• face-to-face workshops

• other…

37

new technologies for data collection

• discussion forums • handhelds• participatory video• survey monkey• blogs• chat• skype• what else…?

38

Expanding accountabilities

OM helps you:

• manage multiple accountabilities: up, down, horizontal by:– being able to document & communicate outcomes

(traditional understanding of accountability)– engage in social learning with your boundary partners

making you more accountable to yourself, your organization, and them

39

Taking up M&E findings

• What should we keep doing?

• What do we need to change in order to improve?

• What strategies/practices do we need to add?

• What strategies/practices do we need to drop? (i.e., they produced no results, they require too much efforts or resources to produce results?)

• Has any issue come up that we need to evaluate in greater depth? What? When? Why? How?

40

Reviewing programme design

1. Read the vision statement

Does this still reflect the program's dream?

2. Read the mission statement

Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away?

3. Review boundary partners

Is this who we are working with directly? Do we need to work with anyone else?

4. Review outcomes Do these accurately describe the ideals way that our boundary partners could act to contribute to the achieving of the vision?

5. Review progress markers

Was the change process we set out accurate and useful? What now needs to be added or taken out?

6. Review strategies What did we plan to do? Have we implemented these activities? Why? Why not?

7. Review organizational practices

Are we doing everything we can to maintain our capacity to support our partners?

41

Discussion questions

• What elements of OM would we like to integrate that we think would be helpful? Why?

• What do we need to do to make this happen? • What spaces, methodologies & tools do we already use

that OM could complement? (how can we integrate OM into what we are already doing?)

• What challenges will we face? What opportunities are there to take advantage of?

• How will we communicate to others? • What will be our next steps?

OM 2009Review of Pilot OM Implementation within Culture for Organizational Learning

STRENGTHSIdentify areas where OM was used to provide a benefit to organization, teams and partners in 2009

WEAKNESSESIdentify areas of organizational practice which needed improvement in order to achieve desired outcome of OM integration in 2009

OPPORTUNITIESIdentify areas of OM that could support organizational learning and development in 2010 - that are not currently being utilized

THREATSIdentify possible obstacles or problems to the integration of OM horizontally across our programmes in 2010