Outcome Mapping Planning, Reflecting & Learning Shalini Kala July 2011.
Challenging Evaluation: An Introduction to Outcome Mapping
description
Transcript of Challenging Evaluation: An Introduction to Outcome Mapping
Challenging Evaluation: Challenging Evaluation: An Introduction to Outcome MappingAn Introduction to Outcome Mapping
Amy Etherington & Rebecca Lee
Mini-training for IDRC Interns & PDAs
December 7 & 8, 2005
Session overview:Session overview:
Evaluation context – where does OM fit?
Introduction to OM methodology – main concepts and tools
OM book – it’s useful!
How is OM being used – when is it appropriate?
Warm-up question:Warm-up question:
What words come to your mind when you think of “evaluation”?
IDRC’s Evaluation SystemIDRC’s Evaluation System
Promotes ownership & use of findings at all levels
Decentralized Focus on evaluation processes Monitors quality
Accountability for results at program & project levels
IDRC’s Evaluation UnitIDRC’s Evaluation Unit
Strategic evaluations Cross-cutting issues
Corporate reporting To BoG on performance and results Information systems
Capacity building With Southern partners Promoting M&E with programs
Tools & methods Organizational Assessment Outcome Mapping
Monitoring & evaluation challenges:Monitoring & evaluation challenges:
1. Establishing cause & effect in open systems
2. Sharing ownership & participation
3. Recognizing the contributions of others
4. Tracking progress
5. Encouraging iterative learning
6. Measuring development results of research
7. Timing
Shifting influence over timeShifting influence over time
Low
High
ProgramBeneficiaries/Indigenous Actors
Infl
uen
ce
Time
Are we having an
impact yet ?
A framework that allows researchers to plot human behaviour and actions and assess their contribution to the aims of research projects and programs
Outcomes as changes in behaviour, relationships, activities or actions of the people, groups and organizations with whom a program works directly
IDRC’s Evaluation Unit (1998-2000)
OM bumper-stickers:OM bumper-stickers:
Be prepared for surprises along the way
The map is not the territory!
Being attentive along the journey… This is as important as the destination
Keep your eyes wide open!
OM t-shirtsOM t-shirts
ATTRIBUTIONCONTRIBU
TION
OM t-shirtsOM t-shirts
IMPACTINFLUE
NCE
OM recognizes that in a OM recognizes that in a development context, change is:development context, change is:
Complex (multiple actors and factors)
Continuous (not limited to the life of the project)
Non-linear (unexpected results occur)
Beyond the control of the project (but subject to its influence)
Two-way (program also changes)
Principles of usePrinciples of use
Flexible: modular - use adapted to circumstances
Participatory: seeks dialogue and collaboration with boundary partners in P,M,&E
Evaluative thinking; culture of reflection: promotes social and organizational learning
Step 1: VisionStep 1: Vision
I have a dream!
Martin Luther King, Jr.August 28, 1963
Vision is a guide:Vision is a guide:
NorthLight house
Distant star
Vision statement:Vision statement:
Reflects the broad human, social, and environmental betterment in which the program is engaged and to which it is contributing
Written in present tense - as if the vision were already a reality
“Those who dream make a difference!”
Ms. Kalpana Pant
-Chaitanya
Women and girls in rural India enjoy full access to appropriate health care, education, food and water security and freedom from violence. They earn their own livelihoods with free access to markets and to banking and municipal services. Women knowledgeably use drudgery reduction technologies and agricultural practices that support human and ecological well-being. Villages are served by public transport, are well lit at night and have police forces that enforce laws fully and equitably. Girls attend school full time and, with their families, have the information and resources to make informed decisions regarding their personal health, safety and social needs. Gender equity is evident in the allocation of household labor and household decision-making; and men in the community support gender-responsive initiatives. Schools reinforce the role of women as educated, valuable and empowered community members. Through community-based organizations, women participate in and influence decisions, policies and programs affecting their well-being and share their experiences with others.
What is the human, social and/or environmental condition that you hope to see in the region in which the program is being carried out?
Who is contributing to this situation - and how?
Vision: facilitation questionsVision: facilitation questions
Dream boldly !
Step 2: MissionStep 2: Mission
The mission is that “bite” of the vision statement on which the program is going to focus.
Written in future tense - as something the program will do
Your mission is your Your mission is your “business”“business”
What do you do?
Who do you do it with?
Why do you do it?
How can the program best support or contribute to the achievement of the vision?
What areas do you need to work in?
Where do you have credibility?
Who can you work with?
Mission: facilitation questionsMission: facilitation questions
Vision
Mission
Step 3: Boundary PartnersStep 3: Boundary Partners
Boundary Partners: Boundary Partners: definitiondefinition
Those individuals, groups, and organizations with whom the program:
interacts directly to effect change anticipates opportunities for influence
engages in mutual learning
Program
The rest of the world
= partners
Sphere of influenceSphere of influence
Program
The rest of the world
= partners
Boundary Partners have Boundary PartnersBoundary Partners have Boundary Partners
program program’s bp bp’s bp
CIDA
IDRC
BAIF
StateNGO
StateNGO
StateNGO
StateNGO
StateNGO
StateNGO
SHG Police CommunityLeaders
Families Banks PHCs
SwayamsiddhaSwayamsiddha
In which individuals, groups, or organizations is our program trying to encourage change so that they can better contribute to the vision?
With whom will we work directly?
Are we choosing X BP because we want to influence their behaviour and actions, or because they will influence others? Or both?
What behavioural changes do we (the project and BPs, collectively) want to see in the BP that will contribute to the vision?
Boundary Partners: Boundary Partners: Facilitation QuestionsFacilitation Questions
Vision
BOUNDARY PARTNER 2
Mission
BOUNDARY PARTNER 3
BOUNDARY PARTNER 1
Step 4 & 5: Step 4 & 5: Outcome Challenges and Outcome Challenges and
Progress MarkersProgress Markers
One OC is about a single boundary partner
Describes the ideal behavioural changes, relationships, actions and interactions in this partner
Describes how these changes will contribute to the vision.
Outcome Challenge: Outcome Challenge: definitiondefinition
One for each boundary partner
Does NOT describe program strategies
Written like this: “The program intends to see [boundary partner] who [description of behaviours in the active present tense]”
Outcome Challenge: characteristicsOutcome Challenge: characteristics
Ideally, how would your boundary partner be acting?
With whom would they be interacting?
What would they be doing?
Outcome Challenges: facilitation Outcome Challenges: facilitation questionsquestions
Progress Markers: Progress Markers: descriptiondescription
A graduated set of statements describing a progression of changed behaviours in the boundary partner
Describe changes in actions, activities and relationships leading to the ideal outcome
Articulate the complexity of the change process
Can be monitored & observed
Permit on-going assessment of partners’ progress (including unintended results)
3 Levels of Progress Markers3 Levels of Progress Markers
The program sets out what it would:
Expect to see the boundary partner doing?
Like to see the boundary partner doing? Love to see the boundary partner
doing?
Progress markers = ladder of changeProgress markers = ladder of change
Truly transformativeSet quite high
More active learning, engagement
Early response to program’s basic activities
Love to see
Like to see
Expect to see
X
Why graduated progress markers?Why graduated progress markers?
Taken as a set, the progress markers: are graduated from easier to more difficult to achieve
changes in behaviour
describe the change process of a single boundary partner
are more complete than a single indicator
help the program think about how it can intentionally
contribute to the most profound transformation possible
facilitate mid-course corrections and improvement
Vision
OUTCOMES
Behaviors, Actions, Relat
ionships of
Boundary Partners
Progress Markers Love to See
Like to See
Expect to See
BOUNDARY PARTNER 2 Outcome Challenge
Mission
Progress Markers Love to See
Like to See
Expect to See
Progress Markers Love to See
Like to See
Expect to See
BOUNDARY PARTNER 3 Outcome Challenge
BOUNDARY PARTNER 1 Outcome Challenge
Influencing OutcomesInfluencing Outcomes
BoundaryPartner
Program relevance & viability
Org. Practices
Program Results
Progress Markers
Program Delivery
Strategies
Program
Step 6: Strategy MapStep 6: Strategy Map
Strategy MapStrategy Map
Outlines the program`s approach in working with the boundary partner
Indicates the relative influence the program is likely to have on boundary partner
Helps pinpoint strategic gaps in the approach or if the program is overextended
6 Types of Strategies6 Types of Strategies
Aimed at the
Boundary Parnter
Aimed at the Boundary Partner`s
Environment
Strategy SupportivePersuasiveCausal
I-1
• Direct Output
E-1
•Alter physical or regulatory
environment
I-2
• Arouse New Skills/ Thinking
I-3
•Supporter who guides change
over time
E-2
•Modify the information system
E-3
•Create / Strengthen a Peer Network
Facilitation Questions:Facilitation Questions:
I-3I-2I-1
E-1 E-3E-2
What will bedone to producean “immediate”
output?
What networks/relationships willbe established
or utilized?
How willsustainedsupport,
guidance, ormentoring be
provided?
How will you usethe media orpublications?
What will bedone to build
capacity?
What will bedone to changethe physical or
policyenvironment?
Strategy imagesStrategy images
Vision
ORGANIZATIONAL
PRACTICES
STRATEGIES
Undertaken to Bring
about Outcomes
OUTCOMES
Behaviors, Actions, Relationships of
Boundary Partners
Progress Markers Love to See
Like to See
Expect to See
BOUNDARY PARTNER 2 Outcome Challenge
Mission
Progress Markers Love to See
Like to See
Expect to See
Progress Markers Love to See
Like to See
Expect to See
Strategy Map 3 Strategy Map 2 Strategy Map 1
BOUNDARY PARTNER 3 Outcome Challenge
BOUNDARY PARTNER 1 Outcome Challenge
Step 7: Organizational Step 7: Organizational PracticesPractices
What the program does to:
look within
stay fresh, sharp, effective, healthy
better serve its partners
learn and change
Organizational Practices: definitionOrganizational Practices: definition
Why Organizational Practices?Why Organizational Practices?
Important to how the program is going to function to effectively fulfill its mission
Supporting change in its boundary partners requires that the program be able to change and adapt
8 Organizational Practices8 Organizational Practices
1. Prospecting for new ideas, opportunities, and resources
2. Seeking feedback from key informants
3. Obtaining the support of your next highest power
4. Assessing and (re)designing products, services, systems, and procedures
8 Organizational Practices8 Organizational Practices
5. Checking up on those already served to add value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
Vision
ORGANIZATIONAL
PRACTICES
STRATEGIES
Undertaken to Bring
about Outcomes
OUTCOMES
Behaviors, Actions, Relationships of
Boundary Partners
Progress Markers Love to See
Like to See
Expect to See
BOUNDARY PARTNER 2 Outcome Challenge
Mission
Progress Markers Love to See
Like to See
Expect to See
Progress Markers Love to See
Like to See
Expect to See
Strategy Map 3 Strategy Map 2 Strategy Map 1
BOUNDARY PARTNER 3 Outcome Challenge
BOUNDARY PARTNER 1 Outcome Challenge
Step 8-11: Monitoring Step 8-11: Monitoring priorities and Journalspriorities and Journals
Count every " F " in the following text:Count every " F " in the following text:
FINISHED FILES ARE THE RE
SULT OF YEARS OF SCIENTI
FIC STUDY COMBINED WITH
THE EXPERIENCE OF YEARS...
M & E involves making choicesM & E involves making choices
“You can’t always get what you want…But if you try sometimes, you might find
You get what you need… ahhhhhhh, yeah…”
The Rolling Stones
Choosing WHAT to monitorChoosing WHAT to monitor
Who will use the monitoring information?
What will it be used for?
When is it needed?
Choosing HOW to monitorChoosing HOW to monitor
Which components will be monitored?
How and when will data be collected?
Who will collect it?
Who will analyze, collate, package data?
Where and when will it be discussed and used?
Monitoring needs to be:Monitoring needs to be:
Useful
Affordable
Light in work
Understandable
Outcome JournalOutcome Journal
Work Dating From/To:
Contributors to Monitoring Update:
Low =
Medium =
High =
Outcome Challenge:
LMH
Expect to see: Who?
3
1
2
Strategy JournalStrategy Journal
Working Dating From/To:
Contributors to Monitoring Update:
Strategy to be Monitored:
Effectiveness?
(How did it help the boundary
partner?)
Outputs
Performance JournalPerformance Journal
Working Dating From/To:
Contributors to Monitoring Update:
Practice 1: Prospecting for New Ideas, Opportunities, & Resources
Practice 2: Seeking Feedback from Key Informants
Example or Indicators:
Sources of Evidence:
Lessons:
Example or Indicators:
Sources of Evidence:
Lessons:
Step 12: Evaluation PlanStep 12: Evaluation Plan
Evaluation Plan:Evaluation Plan:
A short description of the main planning elements of an evaluation study
For Whom? What? Where? How? By Whom? When? How Much?
Elements of an Evaluation PlanElements of an Evaluation Plan
Who Will UseEvaluation,
How, & When?
Questions to be
Answered
Info. Sources
EvaluationMethods
Who WillConduct &
Manage theEvaluation?
Date(Start &Finish)
Cost
Evaluation Issue:
Setting Evaluation PrioritiesSetting Evaluation Priorities
Criteria with which to prioritize include:
Learning needs
Reporting requirements
Partners’ information needs
How can OM be used?How can OM be used? Designing and articulating the program’s logic
Recording internal and external monitoring data
Indicating cases of positive performance and areas for improvement
Evaluating intended and unexpected results
Gathering data on the contributions that a program made to bringing about changes in its partners
Establishing evaluation priorities and an evaluation plan
When is OM best used?When is OM best used?
Once strategic direction or primary program areas are established
Particularly effective for larger projects
Best used at the start, but can also be used as a midway or final assessment tool
Activities must be sufficiently specific to identify key groups who will be influenced
Is OM appropriate? Not always!Is OM appropriate? Not always!
May require changes that are not possible Able to focus on “OM” outcomes Commitment to change and self-assessment Not intended for technical evaluations
(assessing relevance of a programming area or a cost-effectiveness comparison)
Credible, and compatible with donor reporting requirements
Team consensus Resource commitment