Monitoring, Evaluation, Reporting, and Learning (MERL) for ... · UNDP United Nations Development...
Transcript of Monitoring, Evaluation, Reporting, and Learning (MERL) for ... · UNDP United Nations Development...
Monitoring,
Evaluation,
Reporting, and
Learning (MERL) for
Peacebuilding
Programs
Pact is a promise of a better tomorrow for all those who are poor and marginalized. Working in partnership to develop local solutions that enable people to own their own future, Pact helps people and communities build their own capacity to generate income, improve access to quality health services, and gain lasting benefit from the sustainable use of the natural resources around them. At work in more than 30 countries, Pact is building local promise with an integrated, adaptive approach that is shaping the future of international development. Visit us at www.pactworld.org.
September 2016
Disclaimer:
Portions of this module have been made possible by the generous support of the American people through the U.S. Agency for International Development (USAID). The entirety of the contents of this module are the responsibility of Pact and do not necessarily reflect the views of USAID or the United States Government.
Recommended citation:
Pact. 2016. Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs. Washington, D.C., U.S.: Pact.
Contact:
Pact’s Results and Measurement Team Pact 1828 L Street NW, Suite 300 Washington, D.C. 20036 202-466-5666 [email protected]
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Contents
Abbreviations and Acronyms ............................................................................................ i
Acknowledgements .......................................................................................................... ii
Introduction ..................................................................................................................... 1
Chapter 1: Overview of MERL Principles ......................................................................... 2 What is MERL? ................................................................................................................................. 2 What is peacebuilding? .................................................................................................................... 3 Why is MERL important for peacebuilding? .................................................................................. 3 Definitions ........................................................................................................................................ 4 Who is on a MERL team? ................................................................................................................. 8 Key Reasons for Developing Quality MERL Systems ..................................................................... 8 MERL in the Project Cycle ............................................................................................................. 10 MERL in Summary .......................................................................................................................... 11
Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like ....................... 13 What are peacebuilding approaches? ............................................................................................ 13 Approach 1: Incentives for Peace ................................................................................................... 14 Approach 2: Addressing Root Causes of Injustice ........................................................................ 14 Approach 3: Addressing Individual Attitudes and Relationships and Building Trust ............... 14
Chapter 3: Situation Analysis .......................................................................................... 16 What is situation analysis?............................................................................................................. 16 Types of Situation Analysis ............................................................................................................ 16 Disseminating Situation Analysis Findings .................................................................................. 19
Chapter 4: Theories of Change and Developing Project Goals........................................22 What are Theories of Change? ....................................................................................................... 22 Theories of Change in the Context of Peacebuilding Programs ................................................... 23 Develop Your Peacebuilding Project’s Theory of Change ............................................................ 25 Develop Project Goals and Outcomes from Your Theory of Change ........................................... 26 Tying Situation Analysis and Theory of Change to MERL System Design ................................. 28
Chapter 5: Conceptual Frameworks and Assumptions ...................................................32 What are conceptual frameworks? ................................................................................................ 32 Advantages and Challenges of Using Conceptual Frameworks ................................................... 33 The Results Framework ................................................................................................................. 34 Logical Frameworks ....................................................................................................................... 39 Basic Logframe Outline (Hierarchy of Objectives) ....................................................................... 39 The Role of Assumptions ............................................................................................................... 40
Chapter 6: Indicators ..................................................................................................... 43 Overview ......................................................................................................................................... 43 How to Develop Indicators ............................................................................................................ 45 Qualities of a Good Indicator ......................................................................................................... 47 Indicator-Related Terms ................................................................................................................ 48 Indicator Definitions ...................................................................................................................... 49 Collecting Indicator Data ............................................................................................................... 49
Chapter 7: Monitoring .................................................................................................... 52 What is monitoring? ....................................................................................................................... 52 Conflict and Context Monitoring ................................................................................................... 53 Implementation Monitoring .......................................................................................................... 54 Assumption Monitoring ................................................................................................................. 56 Monitoring Plans ............................................................................................................................ 57
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation .....................................................................................................59 What is evaluation and why is it important? ................................................................................ 59 Defining Evaluation ........................................................................................................................ 59 Monitoring versus Evaluation ....................................................................................................... 61 Evaluation versus Research ........................................................................................................... 61 Why evaluate? ................................................................................................................................. 62 Types of Evaluation ........................................................................................................................ 62 Internal and External Evaluation .................................................................................................. 63 Involving Stakeholders in Evaluation ........................................................................................... 64 Barriers to Evaluation .................................................................................................................... 64 Planning for an Evaluation ............................................................................................................ 64 Writing an Evaluation Purpose Statement ................................................................................... 67 Evaluation Questions ..................................................................................................................... 68 What is evaluation design? ............................................................................................................ 69 Overview of Evaluation Approaches .............................................................................................. 69 Creating Evaluation Worksheets ................................................................................................... 70 The Evaluation Terms of Reference ............................................................................................... 71
Chapter 9: Learning ........................................................................................................ 74 Overview ......................................................................................................................................... 74 Using an Adaptive Management Approach .................................................................................. 74 Developing a Learning Agenda ...................................................................................................... 75 Ensuring Learning Processes are Participatory ............................................................................ 76
Chapter 10: Reporting .................................................................................................... 80 Overview ......................................................................................................................................... 80 Introduction to Reporting .............................................................................................................. 80 Identifying Appropriate Communication Tools ........................................................................... 81 Reporting Schedule ........................................................................................................................ 81 Generic Outline for Quarterly and Annual Progress Reports ...................................................... 83 Generic Outline for Final Performance Report ............................................................................ 85 General Writing Tips ...................................................................................................................... 87 Success Stories and Photography .................................................................................................. 87
Chapter 11: Data Quality and Ethics ............................................................................... 89 The Importance of Ensuring Quality Data .................................................................................... 89 Commonly used Criteria for Assessing Data Quality ................................................................... 89 Data Management .......................................................................................................................... 90 Getting Started with Developing Systems to Address Data Quality Issues ................................. 91 Improving Data Quality along the Data-Management Chain ...................................................... 92 Data Ethics ...................................................................................................................................... 94
Chapter 12: Mobile Technology for Data Collection ........................................................95 Overview ......................................................................................................................................... 95 What is information communication technology for development? ........................................... 95 Advantages of Mobile Data Technology ........................................................................................ 96 SMS versus GPRS ........................................................................................................................... 97 Developing Your Mobile Technology Strategy and Why it is Important .................................... 98 When You Should Continue Collecting Data on Paper .............................................................. 104 Develop a Budget .......................................................................................................................... 105
Appendix 1: Success Stories Guide ................................................................................ 107
Appendix 2: MERL Plan Template ................................................................................ 109
Appendix 3: References ................................................................................................ 125
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
page i
Abbreviations and Acronyms
APEA applied political economy analysis
CBO community-based organization
CDC U.S. Centers for Disease Control and Prevention
COP chief of party
DCOP deputy chief of party
DFID U.K. Department for International Development
DIY do it yourself
DQA data quality assessment
GIS geographic information system
GPRS general packet radio service
GPS global positioning system
ICT4D information communication technology for development
Log frame logical framework
M&E monitoring and evaluation
MERL monitoring, evaluation, reporting, and learning
MPICE Measuring Progress in Conflict Environments
NGO nongovernmental organization
OECD Organization for Economic Cooperation and Development
PEA political economy analysis
PIRS Performance Indicator Reference Sheet
SMS short message service
SOP standard operating procedure
TH trauma healing
TOC Theory of Change
TOR Terms of Reference
U.K. United Kingdom
U.S. United States
UNDP United Nations Development Programme
US$ United States dollar(s)
USAID U.S. Agency for International Development
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
page ii
Acknowledgements
The initial draft of this module was compiled by Lynn McCoy, Hannah Kamau, and Margaret Elise during their time with Pact.
Special thanks are given to Pact’s PEACE III project MERL team—Jacqueline Ndirangu, Lauren Serpe, Josiah Imbayi Mukoya, Michael Kahindi, and Christopher Kinyua—for customizing this module for use in Pact Kenya’s PEACE III program and for broader publication to share with other peacebuilding programs.
Many other Pact staff contributed to the development of the peacebuilding MERL module. Gratitude goes to Nanette Barkey (technical oversight), Leighton Clark (technical contributions), Maggie Dougherty (graphic design), Rachel Elrom (editing and layout), and Mason Ingram (technical contributions).
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Introduction | page 1
Introduction
Robust monitoring, evaluation, reporting, and learning (MERL) are critical components of successful programming. The MERL components enable program stakeholders to monitor progress and evaluate the achievement of expected results. Reporting processes and timelines should be clearly defined and tailored to meet the needs of key audiences and stakeholders, and provision should be made for the program to continually reflect and learn from experiences gained during implementation.
Measuring the success of peacebuilding programs poses specific challenges that are unique to this program area. This module was developed to guide PEACE III local program partners—peacebuilding practitioners—through the development and implementation of effective and practical MERL systems for their projects. This five-year cross-border peacebuilding program is implemented by Pact in partnership with Mercy Corps and a range of local partners with activities in Ethiopia, Kenya, Somalia, South Sudan, and Uganda. PEACE III aims to strengthen cross-border conflict management in the Horn of Africa and is pursuing two related objectives: 1) to strengthen local cross-border conflict management and 2) to improve the responsiveness of regional and national institutions to cross-border conflict.
This manual was created to support and provide examples to peacebuilding practitioners and is an addition to Pact’s existing MERL Modules.1 Other useful examples of MERL training manuals for peacebuilding programs exist. This module does not seek to replicate those manuals, but rather draws on them and integrates their expertise here. Parts of this manual were also drawn from Pact’s MERL Modules but customized with practices and examples relevant for peacebuilding programs. It is the authors’ hope that this module can serve as an introduction to MERL for peacebuilding practitioners and can point them to other relevant resources in the field. Each chapter begins with an outline of the learning objectives, includes learning activities throughout the chapters, and ends with a summary of key points and learning.
1 Pact has developed four core training modules to guide MERL staff and to provide a framework for Pact’s partners on M&E basics. These are: Module 1: Building Basic Monitoring and Evaluation Systems; Module 2: Field Guide to Data Quality Management; Module 3: Field Guide for Evaluation; Module 4: Mobile Technology Handbook. All modules can be found in Pact’s Resource Library: http://www.pactworld.org/library.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 2
Chapter 1: Overview of MERL Principles
Chapter Objectives In this chapter you will learn:
The four key interrelated processes found in a monitoring, evaluation, reporting, and learning (MERL) system
Key reasons for having a high quality MERL system
How MERL is embedded in the project cycle
Learning Activities
Defining peacebuilding for your project
What do you want to achieve with your peacebuilding project’s MERL system?
Checking and improving the project cycle
What is MERL?
MERL is a contemporary of the commonly used abbreviation, M&E. The “M” and “E” are the
same: monitoring and evaluation. The “R” and the “L” stand for reporting2 and learning. Pact uses this more-comprehensive terminology to emphasize that the four components are inherently linked. Without reporting on and learning from results, monitoring and evaluating programs is pointless. M&E as a standalone activity is like discovering that the brakes on your car do not work, but not telling anyone or getting them fixed. Learning about something is only useful if you apply what you learn.
Figure 1.1: The four separate but interrelated processes in a MERL system
MERL is:
A very important component of effective peacebuilding
A way for you to learn about what in your program works and what does not work
Even more important to conflict-based development initiatives than in stable environments: As the conflict and environment changes, so must peacebuilding activities and foci. MERL systems provide key insights into what has changed and what needs to be changed.
Ideally conceived at the same time as your peacebuilding approaches are developed; however, even if you are already in the midst of program activities, you can still develop useful MERL tools to help guide, improve, and assess your program
2 Of note, the “R” in MERL also can represent “Research;” however, for the purposes of this manual, the “R” will represent “Reporting.”
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 3
MERL is not:
A system separate from your programs and projects; it is closely interwoven with your project planning and implementation
Just for funder reporting
Meant to be a burden to program staff: Even when certain MERL activities are required by funders and parent organizations, your team should always work with them to develop processes that are mutually beneficial.
The job of an M&E specialist alone: MERL is a team effort and involves stakeholders from within your organization and the communities where you strive to build peace.
What is peacebuilding?
Have you ever tried to define the term peacebuilding? Defining peacebuilding can be
challenging, mirroring the complexity and range of activities needed to bring about peace. Despite the term’s complexities, most people working on this issue have a similar understanding of the change they want to bring about. Ending fighting, reducing potential for future conflict, improving relations between peoples and communities, and reducing injustice are common goals of peacebuilding programs.
Peacebuilding approaches can address the effects of violence or the root causes or drivers of conflict. Activities can range from direct interventions to stop fighting to initiatives aimed at improving education, health, and prosperity. In fact, peacebuilding can take place where there has not been violence: helping individuals understand how they can better live with their neighbors, creating just societies, and collaborating to make communities prosper are all part of peacebuilding.
Why is MERL important for peacebuilding?
Organizations and professionals working in the peacebuilding sector often state that they need quality information about their activities and the changes they bring about to improve their work and to meet the needs of communities. Yet, many organizations often make critical decisions based simply on personal and staff judgment, anecdotal information, or haphazardly collected field data.
Areas beseiged by conflict are subject to frequent changes, in the conflict itself and in the overall environment. Peacebuilding programs likewise need to be adaptable to meet the changing needs of people and institutions. But in order to do so, peacebuilding programs must have access to timely and reliable information, not just about the effects of their direct interventions, but about the evolving situation around them.
Developing and implementing a high quality MERL system allows program teams to access information to make good decisions before, during, and after program implementation. Through MERL, a program team can see if they are achieving their targets, whether their work is having its desired (or other) effects, and whether it is necessary to make corrections to the initiative. Also, programs can learn if cumulative efforts of multiple activities are bringing about the peacebuilding changes sought.
It is just as important to learn about what is not working as it is to learn about what is working. Monitoring helps to identify what is not working early so that program managers can make adjustments throughout the life of the program. Periodic formative evaluations have a similar purpose and can help programs understand why they see the results they see.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 4
Learning Activity: Defining peacebuilding for your project
Understanding how you define peacebuilding is important not only to project design but to the project’s M&E system, as we shall learn in upcoming chapters. Consider your answers to the following questions.
1. In my project, peacebuilding is defined as: _______________________________
______________________________________________________________
______________________________________________________________
2. What MERL processes do we use currently in our peacebuilding programs? ________
______________________________________________________________
______________________________________________________________
3. Do our MERL processes align with our understanding of peacebuilding? ___________
______________________________________________________________
______________________________________________________________
Definitions
A MERL system is composed of four separate but interrelated processes: monitoring, evaluation, reporting, and learning.
Monitoring is the systematic process of collecting and analyzing information to
track the program’s performance in achieving goals and to observe changes in the conflict or environment.
Monitoring data can be used to measure how productively inputs (money, time, equipment, personnel, etc.) are used in the creation of outputs (products, outcomes, results) so a program team can compare what they planned to accomplish in a given period to what actually occurred in that time period.
Monitoring also can be used to keep track of the context and, specifically, the conflict in which program activities are occurring, as well as assumptions on which program activities are based. This type of monitoring is particularly important for program effectiveness because changes in conflict and context often necessitate changes in program activities in order to meet desired outcomes.
How program teams use monitoring data
Program staff use monitoring data to manage program implementation, budgets, timelines, and human resources. Monitoring data also is used as a basis for reporting to funding agencies in quarterly, semi-annual, and/or annual reports. Monitoring happens periodically throughout the life of the project and is ideally set up so key information about program efficiency is routinely delivered to program staff to help them actively manage the program.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 5
Monitoring data helps project teams answer key questions, such as the following.
What activities have been completed (compared to those planned)?
At what cost and in what timeframe are activities being accomplished and how does this compare to what we planned for a given period?
What short term results have occurred, for example, how many people have been mobilized, trained, or reached in a given time period?
Have key benchmark activities been completed as planned?
Does data tell us anything about differences between activities and/or program sites?
What has changed around us? Are there reasons to change our upcoming plans?
Evaluation is the systematic collection and analysis of information to
determine the effectiveness of an intervention or of an organization’s achievement of peacebuilding goals.
Evaluation focuses on intervention outcomes and impacts3 (intermediate and long-term effects) and lets program teams know if change is occurring in a community or conflict situation. Evaluation also provides regular feedback that helps organizations assess their relevance, scope, and sustainability. In its simplest terms, evaluation is the collection and analysis of information to assess the impact of the organization’s work.
Like monitoring processes, evaluations can be conducted periodically throughout the project or program. And, while monitoring activities focus on what happened, evaluations look more in depth at how well things happened and whether they resulted in desired outcomes or are making progress toward achieving them. Evaluations often can be broken down into three phases:
Baseline: collecting values at the start of a program or intervention to be compared against at later points
Midterm: checking statutes against baseline figures and planning for changes to the approach, if necessary
Endline: measuring changes from beginning of the project to the end
Figure 1.2: Evaluation phases
3 It is important to note that in this context, impact assessment implies looking at long-term, broader-scale changes in the community that result from activities and processes among smaller populations within that community. This distinction is important because the term “impact assessment” is sometimes used to describe an evaluation whose goal is attributing change to a particular intervention. Such studies are highly controlled, costly, and difficult to implement, even in stable environments.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 6
How program teams use evaluation data
Evaluation happens periodically, and information provided is used by the program staff to help them understand the short-, intermediate- and long-term results of their programs.
Evaluation data can help program teams answer key questions, such as the following.
Are activities effective in achieving desired outcomes (short- and long-term)? Are we heading in the right directions?
Are our approaches as efficient at achieving outcomes as they could be (or is there a better way)?
What are the secondary- and highest-level effects of our interventions? Are our efforts resulting in changes that are leading to stability and peacebuilding?
Will changes persist after our direct influence has left? Will our interventions’ positive changes and effects be able to adapt over time, as needed, to fit the local context?
What unintended outcomes have occurred?
Program teams can use evaluation data to analyze what is working well to maintain, expand, or replicate it and to review what has not worked well to make program course corrections, avoid repeating mistakes, and improve the quality of their initiatives.
Reporting is the systematic and timely provision of useful information at
periodic intervals.
Reporting provides regular feedback that helps organizations inform themselves and others (stakeholders, partners, funders, etc.) on the progress, problems, successes, and lessons of program implementation. M&E report writers have the responsibility of conveying relevant information to various audiences in effective ways. The challenge for MERL teams is to turn raw data into useful knowledge, then communicate that information to the different program audiences (community members, program staff, partners, etc.) in ways that will most effectively meet their needs and enable them to use it as needed.
MERL teams should think of reporting as broader than simply providing written updates to a funder. Rather, the role of reporting in a MERL system is to ensure that data and analysis is being distributed to those who can use the information to better manage the program, provide oversight, or reflect on the events and outcomes. Effective reporting at regular, strategic intervals is critical for informing program staff, funders, partners, program participants, and other stakeholders of the progress of peacebuilding interventions. Reports can take many forms, can be formal (reports, spreadsheets) or informal (phone calls, quick briefs), and should communicate successes and failures, positive and negative results, and any and all lessons learned.
When creating reports and a reporting schedule, MERL team members should consider:
Who needs it
What specifically they need and why
When and how often they need it
Developing a matrix that details this information is immensely helpful to appropriately plan for reporting. An example of this matrix is presented in Table 1.1.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 7
Table 1.1: Example of MERL audience information needs assessment
MERL
Audience
WHAT data they need from our organization
WHY they need the data
WHEN they need the data from our organization and what reporting format is best
How this audience can participate in the MERL program
Internal Audience
Board members
Updates on:
Activities, projects, and programs
Management issues
Financial issues
Resources (inputs)
Lessons learned on program impacts
To monitor the efficiency and effectiveness of the organization and its people, products, and impacts
To provide guidance
To problem solve
To help us fundraise
Monthly memo and financial statement to CEO
Quarterly meeting presentation (January, April, July, and October)
Annual report and financial statement (December)
Review project design and objectives
Review performance data specifically to evaluate the efficiency and effectiveness of the project
Use the data to guide their decision-making
External Audience
Community members
Context/conflict analysis results
Updates on peacebuilding efforts and community participation in ongoing efforts
Outcomes resulting from interventions, such as peace dialogues and peace dividends
To stay updated on the status of peace in their community
To understand their contribution to peacebuilding efforts
To understand the longer term changes in the peace situation and implications for them
Quarterly presentations at community briefing meetings (January, April, July, and October)
Participate in a situation analysis
Review project design and objectives
Participate in project assessments/ evaluations, e.g., as respondents in focus group discussions and key informant interviews
Share their stories of change
Learning involves internalizing information and analyzing it such that it can be
of use in decision-making. Program management and other stakeholders should use the data generated from their M&E processes to review progress, learn from what is working or not working well, and make decisions on the required changes for successful programming.
Learning is a deeper process than simply reading reports; it involves interpreting M&E results and taking action based on what is learned. Learning is the systematic review of achievements against desired results and is used to change course or modify approaches if necessary. Learning benefits the current project and future projects.
Project technical specialists and MERL team members must participate in planning M&E activities because they all are responsible for learning from the results, including successes and failures, and be willing to accept unexpected results and take action to resolve problems that arise. It is also important to involve other MERL audiences in planning and/or review of MERL plans: program staff, funders, community partners, and others.
During the planning phase, it is important to think about who will be responsible for decision-making regarding changes to be made as a result of learning. Developing a learning
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 8
schedule also can be helpful to ensure project staff hold regular learning reviews within your organization and externally with relevant stakeholders. Also think about your program design: If we wanted to change something in the future, could we? How easy or difficult would this change be? Is there a point at which there’s “no turning back” (implying you should learn about the program’s effectiveness before then)?
How to determine if your program is prepared to learn from M&E activities
Are we generally willing to examine our work and challenge what we know and do?
Are we testing our assumptions? Are our assumptions well founded to begin with?
When faced with negative feedback, do we react in a constructive way?
Is our knowledge being shared? Are regularly scheduled learning reviews being held?
Is our knowledge accessible to all key stakeholders? To the wider development and peacebuilding communities?
Is our learning relevant to our programming and ultimate goals or is it limited in focus to funder requirements?
Who is on a MERL team?
Pact encourages all partners to have at least one full-time MERL officer to lead and coordinate MERL activities. But, that does not mean that the MERL officer is the only staff member involved in MERL activities; in fact, it is just the opposite. All program staff play a role in determining what should be measured, assisting with quality data collection efforts, analyzing findings, and, most importantly, using the data to improve program implementation and peacebuilding practices. Community members and partners also play critical roles, helping shape the focus of MERL and offering advice on effective local MERL practices. MERL efforts are team efforts.
Key Reasons for Developing Quality MERL Systems
MERL systems provide peacebuilding programs with various benefits.
MERL provides managers with a tool to gain timely information on the progress of project activities, allowing them to compare what was planned to what is actually occurring.
MERL provides organizations with a tool to evaluate how effectively the organization promotes change.
MERL fosters public and political cooperation and supports particular information needs for target audiences (such as funders, community members, staff, board members).
MERL enables organizational learning and encourages adaptive management.
Other benefits of MERL include ensuring that our projects, services, and activities:4
Meet community needs
Reach those they are intended to reach
Make adequate, timely progress toward our objectives
Make efficient use of resources (capital, human, and other)
Yield results that are in line with our efforts
Are sustainable5
Are appropriate for the changing environment (political, social, conflict context, etc.)
4 Social Impact 2009.
5 For more resources on sustainability, see http://microlinks.kdid.org/library/search/sustainability.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 9
Effective MERL also benefits institutions in developing future programs in the same or another context. Learnings from a particular project can be used to develop future initiatives during planning, design, and implementation and even at the proposal writing stage. Funders may use past MERL results to evaluate the suitability of your organization or your particular approach to a proposed activity.
Commonly, MERL is viewed as complex, time intensive, and costly. Some organizations also consider MERL to be simply a requirement of funders and thus see MERL as an external rather than internal necessity. Therefore, an effective MERL system is organized and focused, affordable, time efficient, and useful to program staff, funders, and other key program stakeholders.
Learning Activity: What do you want to achieve with your peacebuilding
program’s MERL system?
Throughout this manual, you will learn the steps involved in designing and implementing your MERL system, including choosing your goals and the focus of your assessments. A good way to start is to brainstorm a list of specific information or types of information that would be helpful to you in carrying out a successful program. Do not worry yet about how or when you would get the information.
You can do this activity with a group from your project team.
1. Ask several staff members to brainstorm their information desires on their own. You can ask them the following questions to help shape their brainstorming.
a. What would you most like to know about the community(ies) in which you work?
b. What would you like to learn about or from others who have worked in this community or who have tried similar approaches to peacebuilding?
c. What uncertainties do you have (about the community, the context or conflict, the [proposed] program design)? What would you need to know to be certain?
d. What are the biggest misconceptions you think others have? What information would help to correct those misconceptions?
2. If possible, bring in voices from the community and from your funder and/or parent organizations. At minimum, assign someone to create a list of desired information from the perspective of funders and/or community members.
3. Organize a meeting in which each person shares their lists with a goal of determining the most common informational needs and prioritizing the most important needs.
4. Be sure to consider priorities for the organization and project as a whole and for different groups within your team.
5. Assign one person to record both the entire list of brainstormed ideas and the prioritized list.
6. Refer back to these lists during program and MERL design.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 10
MERL in the Project Cycle
It is important to fully understand the project cycle of peacebuilding projects. The project cycle is the process through which projects are conceived, developed, tested, and adjusted based on what happens. It emphasizes the iterative nature of projects and how we must always be willing to adjust and adapt based on successes and failures. This is especially important in peacebuilding initiatives. There are many variations on the project cycle, but all include four main parts, as shown in Figure 1.3. Below is a quick review of the project cycle and some key focus points for MERL work.
1) Project identification
At this stage, as someone who works on peacebuilding, you probably are looking at the conflict in an area and hoping to find a way to improve the existing situation. You develop your visions of peace and peacebuilding. You analyze the conflict, context, and most pressing needs.6 You start to identify the goals and objectives you will ultimately try to achieve and which will be the focus of your MERL system.
Once you understand what is happening and what is needed, you determine what your organization is best suited to do about it. This will likely be done in collaboration with other local organizations, government offices, or other partners. Together you will decide what your intervention will be.
2) Project design
In the design phase you take your idea of what to do and determine more specifically how to do it. Project design is about coming up with a good Theory of Change (see Chapter 4) and developing specific strategies (and activities) for bringing about the desired changes. Project design includes identifying resource needs (funds, time, personnel, equipment, etc.) to complete the program successfully and establishing the project’s basic framework and MERL system.
3) Project implementation
The majority of project activities will take place in this stage. A baseline study is usually conducted just before a project begins to establish your starting point and for later comparison. Monitoring activities take place regularly throughout the life of the project to track progress and resource allocation.
4) Project evaluation
Periodic evaluations can be used to assess ongoing progress toward peacebuilding goals. What you learn from M&E activities leads you back to program design and implementation to make adjustments or initiate new activities as indicated. A comprehensive project evaluation is an important activity to carry out once a project has concluded. It is a way for you to learn what ultimately worked and what did not work, to inform others who remain (if your activity has ended), and to guide future initiatives.
6 Conflict, context, and needs assessments are discussed in more detail in Chapter 3.
Figure 1.3: Project cycle
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 11
MERL in Summary
Figure 1.4 summarizes the MERL processes that will be covered throughout this module.
Figure 1.4: MERL in summary
Summary: Key points from this chapter
Definition of peacebuilding: Ending fighting, reducing potential for future conflict, improving relations between peoples and communities, and reducing injustice are common goals of peacebuilding programs.
Definition of MERL (monitoring, evaluation, reporting, and learning) and its importance for peacebuilding projects
MERL is a team effort! Program and MERL staff must work together for successful implementation.
When preparing reports, it is important to consider the “who, what, when, how, and why” of the end user to package the reports in a useful way for the end user.
It is important to understand the project cycle of identification, design, implementation, and evaluation when designing peacebuilding projects and their MERL systems.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 1: Overview of MERL Principles | page 12
Learning Activity: Checking and improving the project cycle
In this exercise, you will use your past experience following a project cycle to determine how to improve the process. Refer to the detailed project cycle in Figure 1.5 as you complete this activity.
Think of a project you have worked on or you have seen implemented. See if you can list major activities or processes that happened during:
Project identification
Project design
Project implementation
Project evaluation
(If you could not differentiate between the different phases of the cycle, that tells you something, as well: there is a need to review what processes you are following in identifying, designing, implementing, and evaluating your program to determine which aspects of the project cycle need to be strengthened!)
What did you think went well in this project? Is there anything you would have changed?
Figure 1.5: Detailed project cycle
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like | page 13
Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like
Chapter Objectives In this chapter you will learn:
Why defining your peacebuilding approach is critical to program design and monitoring, evaluation, reporting, and learning (MERL)
About three specific approaches to peacebuilding that your project might employ
Pact’s general approach to peacebuilding
Learning Activity
Identify and write your peacebuilding approach
What are peacebuilding approaches?
Peacebuilding approaches describe how your project will go about realizing its vision for peace, i.e. “what peace will look like.”
As noted in the previous chapter, peacebuilding approaches can address the effects of violence and/or the root causes or drivers of conflict. Activities can range from direct interventions to stop fighting to initiatives aimed at increasing education, health, and prosperity. In fact, peacebuilding also can take place where there has not been violence, helping individuals understand how they can better live with their neighbors, creating just societies, and collaborating to make communities prosper are all part of peacebuilding.
Why is this a starting point for talking about MERL? Because we can’t measure our progress toward achieving ultimate goals without setting them. Defining your approach to peacebuilding effectively declares what peace should “look like,” thus giving you goals to monitor progress toward throughout the life of your project.
A MERL system is a tool that organizations and managers use to see if they are achieving change. Understanding how programs define peacebuilding is essential to program staff because it tells us about the critical changes anticipated that will need to be monitored and evaluated during the life of program to see if those changes have been brought about.
Research on effective and ineffective peacebuilding initiatives has shown that lacking a clear vision of what peacebuilding looks like is a significant barrier to success. Peacebuilding approaches that lack a unifying ultimate goal (i.e., “what peace looks like”) are often less effective than those that have a specific aim in mind.
Below are three examples of approaches to peacebuilding. Your intervention may fit one of these approaches or something different. Consider both individual programs (“What is the ultimate peacebuilding goal that this effort hopes to achieve?”) and the collective efforts of your organization in a particular region (“Are our programs united by a singular view of how they will contribute to peacebuilding here?”).
When asked what they most want to learn about their programs efforts, program staff often say that they want to know whether they “work.” To answer this question, we must first know what the project would look like if it were, in fact, working. This becomes the basis for assessment. The more detail the staff can identify about what “working” looks like (specific goals and objectives) the more easily they will be able to determine if it is doing so.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like | page 14
Approach 1: Incentives for Peace
Some peacebuilding projects define their work as being focused on incentives: either encouraging change by providing incentives for peace or by eliminating incentives for violence. Incentives either induce action or motivate one’s efforts or behavior.
Projects that want to increase the incentives for peace might reward countries or communities with aid if an agreement is reached, increase trade between countries when a conflict ends, or develop initiatives to open borders when a level of security is reached.
Projects that want to eliminate incentives for violence might increase legitimate control over natural resources that fuel conflict, try to make it more difficult for parties to acquire power by force or ease, create zones of peace, control borders to make it more difficult to acquire goods that are needed for war, or seek to make weapons less available.
If, for example, a program takes an eliminating incentives-based approach to peacebuilding, we might anticipate seeing MERL programs that focus heavily on measuring changes that will result from making it more difficult to acquire weapons and goods needed for war or offering amnesty to individuals who voluntarily surrender their weapons. As such, we might measure the number of people disarmed, extent of weapons availability, or average cost of a gun at an open market.
Approach 2: Addressing Root Causes of Injustice
Peacebuilding programs that define peacebuilding as addressing the root causes of injustice in a community include those that:
Aim to empower people to use non-violent means to reduce injustice, such as advocacy projects, human rights education projects, and projects to reform government institutions
Include long-term campaigns for social and structural changes, such as truth and reconciliation initiatives, and changes in social institutions, laws, regulations, and economic systems, among others
Aim to address the underlying causes of injustice, oppression, exploitation, threats to identity and security, and people’s sense of victimization, including projects for economic development, decentralization, self-help, education, and health
If a program defines peacebuilding as addressing root causes, then MERL systems would focus heavily on capturing social and structural changes in society and people’s lives. Some example changes we might study include the number of people reporting an improvement in the level of security, perceived or legislated level of independence of the judiciary, the number of arms passing through border points, or the number of development projects in resource-poor or marginalized communities.
Approach 3: Addressing Individual Attitudes and
Relationships and Building Trust
Peacebuilding programs that define peacebuilding as addressing individual attitudes and relationships and building trust include those that:
Encourage dialogue through activities such as roundtables, peace meetings, mediation efforts, and other events where multiple parties gather to discuss the conflict or environment in general
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 2: Peacebuilding Approaches: Defining What Peace Looks Like | page 15
Encourage coexistence and include joint problem solving projects, multi-ethnic community and economic development initiatives, training and community events where parties at conflict attend, and promoting collaboration between parties in conflict
Train individuals and groups on ways to deal with differences, including conflict resolution and negotiation training events, community mediation training and set-up assistance, justice initiatives, and non-violence training events
If a program defines peacebuilding as addressing individual attitudes and relationships and building trust, then MERL programs would focus on assessing changes in perceptions, attitudes, and values at the individual, community, and institutional levels. Changes might include perceptions of whether co-existence has improved between communities/parties, the number of people reporting an improvement in the level of security, the number of joint initiatives between conflicting parties, and perceptions of tolerance among conflicting communities.
Summary: Key points from this chapter
Defining your peacebuilding approach is important for developing a good MERL system because the former is based on how the project describes what they think “peace looks like.” This forms the basis for developing the project goal and, thereafter, how the project will monitor and measure achievement of this goal.
Research on effective peacebuilding has shown that lack of a clear vision of what peacebuilding looks like is a significant barrier to the project’s success!
Three types of peacebuilding approaches are: 1. Incentives for peace 2. Addressing root causes of injustice 3. Addressing individual attitudes and relationships
Pact’s approach to peacebuilding uses a combination of the “addressing root causes of injustice” and addressing individual attitudes and relationships” approaches.
Learning Activity: Identify and write out your peacebuilding approach
Refer to the three peacebuilding approaches discussed in this chapter. Do any of them apply to your project? If yes, identify which one and describe how your project uses this approach. If no, what approach do you use in your project? Describe it.
Name of project: __________________________________________________
Peacebuilding approach: _____________________________________________
Description: _____________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 3: Situation Analysis | page 16
Chapter 3: Situation Analysis
Chapter Objectives In this chapter you will:
Learn the basics of conducting situation analyses by reviewing three types: conflict analysis, context analysis, and needs assessment
Learn about situation assessment’s relevance to program design and routine monitoring
Learning Activity
Carrying out situation analyses
What is situation analysis?
In the project identification stage of the project cycle, your aim is to choose the best way to improve the situation in the communities, region, or country where you work. In most cases you do not yet know exactly what project you will implement. In other cases, you have an idea of the project you want to implement, but want to make sure that the idea you have is appropriate for the situation and will have the desired effects.
It is critical to understand the specific conflict, context, and needs of the communities in question to avoid misguided program activities or poor indicators for your project. If program activities are misguided, then the indicators developed to measure the results of these activities will not be well aligned to the project objectives and cannot be used to demonstrate achievement of results. Because of the importance of these preliminary situation analyses, we review them here, with special attention paid to how they relate to MERL processes.
By the time you are done carrying out a situation analysis, you will have a clearer picture of the situation on the ground in the area you hope to affect, as well as the larger national or regional scope of the situation. And, you should be keenly aware of what others are doing to address the conflict.
Using this information, the project team can develop strategies (and related activities) to bring about the changes sought. Some basic questions for reflection are:
What would make the challenges to peace diminish?
What can we do (with our partners and stakeholders) to increase successful peacebuilding?
Types of Situation Analysis
Situation analyses are key to finding answers to these questions. In this chapter, we will look at three examples of situation analysis that programs can carry out:
Conflict analysis
Context analysis
Needs assessment
Table 3.1 summarizes the types of situation analyses, when they are conducted, and how to carry out a situation analysis. Detailed descriptions of each type of analysis follows the table.
It’s never too late to do situation analysis. Even if you are in the middle of program activities and have never taken a formal, close look at the conflict around you, take some time to update yourself and your organization on the current state of things.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 3: Situation Analysis | page 17
Table 3.1: Types of situation analysis
Situation analysis type What is it?
When is it conducted?
How is it conducted?
Conflict analysis Examines the root causes, drivers, nature, and primary actors of a conflict
Ongoing through the life of the project
Asking key questions during:
Key informant interviews
Group assessments
Context analysis7 Analyzes the political, economic, and social systems and actors in a system, even if they do not have a role in the existing conflict8
Ongoing through the life of the project
A review of the existing systems and key actors
Key informant interviews
Group assessments
Needs assessment Identifies gaps in systems or services
Proposal stage or months 1–4 of the project
Desk research
Key informant interviews
Group assessments
Conflict analysis examines the root causes, drivers, nature, and primary
actors of a conflict with the goal of gaining an in-depth understanding of the conflict’s dynamics.
Some of the key questions often incorporated into conflict analysis are:
Who are the key actors/stakeholders in the conflict?
What are the main causes of the conflict? What is dividing certain groups and uniting others? How did it start? Are there different driving factors now?
What are the economic, political, and socio-cultural contexts of the conflict at the local, national, regional, and international levels?
What are the current conflict trends? What is happening?
What are the current windows of opportunity to address the conflict?
What efforts are currently in place to build peace/end the conflict?
What has been successful in reducing the conflict? What has failed?
Understanding what caused the conflict to start and what is causing it to continue is important if we are to find ways to mitigate it and build peace. However, because of the dynamic nature of conflict, it is important to regularly re-assess the conflict to see if the drivers have changed or if new barriers to peace have emerged. Ongoing conflict assessment focused on the factors most likely to change should be part of the MERL process.
7 One type of context analysis is political economy analysis (PEA). The Organization for Economic Co-operation and Development (OECD) defines PEA as being concerned with the interaction of political and economic processes in a society: the distribution of power and wealth between different groups and individuals and the processes that create, sustain, and transform these relationships over time. PEA is used to understand the explicit legal, policy, and economic frameworks and the implicit and unwritten norms, values, and interests that help determine how individual and group actors behave. See DFID. July 2009. Political Economy Analysis How to Note. Available at https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/3797.pdf.
8 Pact’s Applied Political Economy Analysis (APEA) aims to provide project teams with practical help in identifying and mapping the political, economic, and social incentives that influence key stakeholders’ actions and decisions. This project- or problem-based methodology is designed to directly support project decision-making. A full description of the approach is available at at http://www.pactworld.org/library/applied-political-economy-analysis-tool-analyzing-local-systems.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 3: Situation Analysis | page 18
Conflict analysis and the project cycle
Project cycle management is closely related to conflict analysis. While analysis of project-level data deals with a wide range of project issues, conflict analysis deals with broader conflict-specific issues that the project aims to address. While the two are complimentary, they are distinctly important to designing a comprehensive project-specific MERL system. While conflict analysis helps to focus on the changes the project wants to achieve in the conflict setting—a transformation from conflict to peace—analysis of the project’s design and implementation helps the organization establish sound operational systems and processes to support achievement of anticipated changes. At the same time, the MERL system is used to assess the efficiency and effectiveness of actions needed to bring about the changes sought.
Context analysis examines existing political, economic, and social structures
at the local, regional, and national levels (or across a larger area if the conflict crosses state borders).
Conflict analyses look at these same systems and structures, but with a focus on those directly affecting the conflict. Context analysis looks more broadly at these systems, even if they do not have a direct role in propelling the conflict. Such factors are important because they impact the way people live their daily lives and are subject to being brought into or affected by the conflict. Assuming your intervention will affect people’s daily lives, it is important to understand all of the factors that impact their habits, activities, interactions, and livelihoods.
Some of the key questions often incorporated into a context analysis are:
What policies, systems, structures, and forces exist inside and outside the immediate local context (village, district, province, nation) of the conflict?
How can the policies, systems, structures, and forces outside the immediate local context be addressed?
What kind of cooperation or linkages between the local, national, regional, and international levels need to be made?
Who are the key actors that can be used to leverage this cooperation (who could be the project’s “champions”)? Who may want to see the project fail (who may be the project’s “spoilers”)?
Where does our organization fit within the context? On what systems do we rely? With whom are we directly aligned?
Like conflict analysis, context analysis needs to be regularly updated to monitor changes. Context analysis often includes looking at systems not directly linked to the conflict. As such, it is important for peacebuilding programs to continuously monitor the emergence of new conflict drivers and how these affect their program context.
Needs assessment involves finding out what services or systems are missing
or require improvement in a given context.
Needs can be assessed on a large scale (national level) or more locally, depending on the scope of your intervention, keeping in mind the impacts local changes have on the larger scale. Such assessments help to determine what has to happen to move from the current toward the desired state of things. The project team’s goal is to develop strategies (and related activities) to fill the gaps and meet needs.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 3: Situation Analysis | page 19
Some of the key questions often incorporated into a needs assessment are:
What are the gaps/needs for the community or target group?
What action is being taken to address them? What more is needed?
How is the community currently trying to meet its needs?
How can the challenges be addressed?
What resources are currently available within the community to address needs and gaps?
What barriers have prevented these needs from being met before?
Ongoing monitoring can help program staff track when needs have been met, changed, or reprioritized. As with other analyses, this requires close collaboration with multiple community members, leaders, experts, and others with direct local knowledge.
Institutional capabilities
Alongside your situation analysis, it is important to constantly consider your place: your own institution’s capabilities, strengths, and potential role in peacebuilding. Consider your organization’s history in the community and region, partnerships, staff capacity, interests, expertise, and any other factors you feel would strengthen or weaken your institution’s ability to carry out successful peacebuilding efforts.
Disseminating Situation Analysis Findings
It is important to disseminate situation analysis findings because they can inform what other stakeholders are doing and identify opportunities for collaboration. Dissemination of findings can be done via different media, including:
Dissemination workshop
Circulating summary reports to targeted stakeholders
Posting on the website
Community events
Periodic newsletter
The process used will depend on resources available, target audience, and purpose of the dissemination. For instance, a dissemination workshop would be ideal to collect input from stakeholders or validate findings.
Summary: Key points from this chapter
Situation analysis helps peace-builders understand on-the-ground realities of the conflict environment.
Situation analysis can be done in different ways, including conflict analysis, context analysis, and needs assessment.
It is important to disseminate situation analysis findings, usually to targeted stakeholders.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 3: Situation Analysis | page 20
Learning Activity: Carrying out situation analyses
Situation analyses can be time consuming and complex. A good first step is to look for existing analytical reports, data collection instruments, surveys, or other sources of information to help guide your conflict analysis, context analysis, or needs assessment.
When looking to these other sources of information (either within your local organization or from other development agencies), take into consideration the following.
What factors affect the conflict your project is designed to address?
Who may be the “champions” or “spoilers” of peace in your project areas?
What is your vision for peace?
What changes are needed in individuals, communities, institutions, organizations, cultures, and/or societies to realize your vision of peace? Refer to Table 3.2.
In your groups, respond to the above questions as they relate to your project. Make note of where there may be existing data to answer these questions or where you may need to undertake your own situation analyses.
Table 3.2: Examples of specific changes9
Type of change Examples of specific changes
Relationship From adversaries to partners in problem solving
From suspicion to solidarity
From different ethnicities to a common nationality
Former neighbors reconciled
Status From soldier to veteran
From rebel leader to parliamentarian
From criminal to entrepreneur
Behavior From violent behavior to assertiveness
From disrespecting women to respecting women
From ignoring youth to taking their interests into consideration
Circumstance From politically marginalized to able to vote
From displaced victim to community member
Functioning Increased transparency
From authoritarian to consultative policy development
Increased cost efficiency
Attitude Greater tolerance of different perspectives
From fear of others to trust in others
From apathy and fatalism to hope and self-determination
From a narrow focus on the neighborhood to a broad focus on inter-communal interests
Knowledge Understand interdependence of groups
Understand how globalization affects local livelihoods
Understand rights and how justice systems should work
Knows how political resources are allocated
Skills From power-based to interest-based negotiations
Moving discussions from mutually exclusive interests to framing issues in mutually acceptable language
Able to introduce items onto the agenda in local governance
9 This table was taken from Church & Rogers, 2006, pp. 18–19, which was adapted from M.Q. Patton. 1997. “Outcome Examples.” Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, C.A., U.S.: Sage, Inc.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 3: Situation Analysis | page 21
Type of change Examples of specific changes
Maintenance Continue to celebrate cultural heritage
Maintain existing social cohesion
Continue to practice traditional dispute resolution processes
Prevention Peaceful transfer of power
Increase awareness of military accountability to civilian ministries
Prevent exodus of trained and educated professionals
Process From shuttle diplomacy to face-to-face negotiation
From hate-mongering to balanced reporting
From divisive methods to methods that bring people together
From concentrations of authority over others to equitable engagement with others
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 22
Chapter 4: Theories of Change and Developing Project Goals
Chapter Objectives In this chapter you will:
Learn what Theories of Change (TOCs) are and how to develop them
Review several TOCs that can be used in evaluating peacebuilding programs
Review several conceptual models
Learn the importance of testing assumptions
Understand the relevance of conceptual models to program design and monitoring, evaluation, reporting, and learning (MERL)
Learning Activity
Describe your project’s TOC
What are Theories of Change?
TOCs are a set of beliefs about how change happens.10 They form an important part of a project because they are used to describe how the micro-level aims (activities/outputs) will have a macro-level result (outcomes/impact). In essence, TOCs pay attention to how the immediate aims of the project can be linked to how society can achieve sustainable peace and how progress along both these paths can be measured.
TOCs help guide thinking during the project design and evaluation stages. Peacebuilding practitioners select project goals, methods, approaches, and activities based on underlying theories of how peace can be achieved in a specific context. Effective projects usually clarify their TOCs early in the life of the project and continually test them against on-the-ground realities.
TOCs can be used for several reasons within a project:
To reveal and understand project assumptions: Generally, two assumptions are inherent in every TOC: 1) how change works and 2) the strategic advantage of the chosen theory over other theories for the context. By linking the immediate or internal project goals, the TOC develops an idea and tests the assumptions of how it will contribute to change in society.
To ensure alignment with all levels of the program design when setting goals and objectives: TOCs explain the basis of change behind particular choices/decisions, thereby tightening the program logic and enabling identification of gaps/unmet needs.
10 Church & Rogers, 2006, p. 118.
Definition of “theory”
An assumption about how something works, or a prediction of what will happen as a result of an action (Lederach et al., 2007, p. 4)
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 23
Theories of Change in the Context of Peacebuilding
Programs
Table 4.1 summarizes 10 TOCs that can be applied to peacebuilding programs.
Table 4.1: Peacebuilding Theories of Change11
Theory Basis of theory Methods/intervention activities
Individual change
Peace comes through transformative change of a critical mass of individuals and their consciousness, attitudes, behaviors, and skills.
Invest in individual change through training
Personal transformation/consciousness-raising workshops or processes
Dialogues and encounter groups
Trauma healing (TH)
Healthy relationships and connections
Peace emerges from a process of breaking down isolation, polarization, division, prejudice, and stereotypes among and between groups.
Processes of intergroup dialogue, networking, relationship-building, joint efforts, and practical programs on substantive problems
Withdrawal of the war resources
Wars require vast amounts of resources (e.g., weapons, supplies, transport) and human capital. If the project can interrupt the supply of people and goods to the war making system, it will collapse and peace will break out.
Anti-war campaigns to cut off funds/national budgets
Conscientious objection and/or resistance to military service
International arms control
Arms embargoes and boycotts
Reduction of violence
Peace will emerge as we reduce the level of violence perpetrated by combatants or their representatives.
Cease-fires
Creations of zones of peace
Withdrawal from direct engagement
Peacekeeping forces
Observation missions
Promotion of non-violent methods for achieving political, social, and economic ends
Root causes/justice
Peace can be achieved by addressing the underlying issues of injustice, oppression, exploitation, threats to identity and security, and people’s sense of injury/victimization.
Long-term campaigns for social change, truth, and reconciliation changes in social institutions, laws, regulations, and economic systems
Institutional development
Peace is secured by establishing stable/reliable social institutions that guarantee democracy, equity, justice, and fair allocation of resources.
New constitutional and governance arrangements/entities
Development of human rights, rule of law, and anti-corruption
Establishment of democratic/equitable economic structures
Economic development
Democratization
11 See Church & Rogers, 2006, pp. 14–15. The text on those pages was used to create Table 4.1.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 24
Theory Basis of theory Methods/intervention activities
Political elites Peace comes when it is in the interest of political and other leaders to take the necessary steps. Peacebuilding efforts must change the political calculus of key leaders and groups.
Raise the costs and reduce the benefits for political elites to continue war and increase the incentives for peace
Withdraw international support/funding for warring parties
Grassroots mobilization
“When the people lead, the leaders will follow.” If we mobilize enough opposition to war, political leaders will have to pay attention.
Mobilize grassroots groups either to oppose war or to advocate for positive action
Use of the media
Non-violent direct action campaigns
Education
Mobilization effort
Organizing advocacy groups
Dramatic events to raise consciousness
Economics People make personal decisions, and decision-makers make policy decisions based on a system of rewards/incentives and punishments/sanctions that are essentially economic in nature. If we change the economics associated with war-making, we can bring about peace.
Use of government or financial institutions to change supply and demand dynamics
Control incentive and reward systems
Boycotts
Public attitudes
War and violence are partly motivated by prejudice, misperceptions, and intolerance of difference. We can promote peace by using the media to change public attitudes and build greater tolerance in society.
TV and radio programs that promote tolerance
Modelling tolerant behavior
Symbolic acts of solidarity/unity
Dialogues among groups in conflict, with subsequent publicity
There are several reasons why monitoring, evaluating, and reporting on peacebuilding projects is more difficult than other types of development. Some of these challenges include the different definitions of peacebuilding, the different theories of what causes conflict and brings about peace, the complexity of conflict, and the need to understand the conflict before a peacebuilding project can start. In addition, the general lack of security and resultant lack of trust of people interviewing them about issues in their community is also a challenge in these types of programs.
When monitoring and evaluating a peacebuilding project, the difficulties named above lead to a greater problem: frequent disagreement about the ultimate ideal outcome(s). Not all people agree on what peace looks like, thus it may be difficult to come to consensus on your ultimate goals. If your ultimate goals are not clear, it is impossible to assess whether you have reached them.
A second difficulty in evaluating a peacebuilding project is the question of attribution. How do projects, for instance, attribute a desired TOC to their interventions? Were there other factors such as environmental issues or a change in governance system that can be attributed to a desired TOC alongside the project’s interventions?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 25
Despite these challenges there are steps to take that can help you develop your TOC. Conducting a situation analyses for instance can help to identify what other factors in the environment are likely to affect your TOC and how the project will account for them.
Develop Your Peacebuilding Project’s Theory of Change
After completing a thorough situation analysis, you have a clear view of the factors affecting the conflict. Among other things, you should understand the following three critical elements. 1. What is making the conflict worse? 2. What conditions or factors could reduce the conflict? 3. What are the ongoing peacebuilding activities/efforts in the target areas, and how can
your organization best contribute to these ongoing efforts?
In this context, TOC describes processes that you believe will lead to peace.12,13
For example, suppose that the situation analyses identified tensions between ethnic groups in a particular community as being a driving force of conflict. Because these groups cannot get along (manifesting in a number of problems with service delivery/provision and violence), your peacebuilding strategy involves changing attitudes of and relationships between individuals in these groups. You believe that changing people’s attitudes about people of “the other” ethnicity through education will encourage peace on a larger scale. Your TOC clearly spells out what you perceive to be the link between what you plan to do (educate) and what you expect to happen (peace).
Your TOC can be articulated in various formats depicting “IF we do this, THEN this will happen.” Here, we depict this TOC as a model and as a sentence.
Figure 4.1: Example of a peacebuilding program Theory of Change
The above TOC example falls under the “Individual Change Theory.” As is always the case when adapting ideas from general frameworks, it is important to base your TOC within the specific context in which you are working.
Points to remember
1. Your TOC does not have to reference specific program activities at the outset, though the program activities you later choose should fit into the TOC you have established. In the above example, “Changes in attitudes of educators and youth views of ‘the other’” will lead to “decreased levels of conflict and violence in the community,” among other effects.
12 See Church & Rogers, 2006, p. 118. There are also others who look at a TOC as a conceptual model and use it much like a results framework or logical framework. This is not the meaning used in this chapter.
13 There are a number of resources available online with more detail and examples of TOCs. One such resource is the Center for Theory of Change: http://www.theoryofchange.org/.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 26
2. Being able to write your TOC in a clear and convincing manner is important and may take time. Prepare to have your TOC challenged. In the previous example, people may take exception to the third part of the theory, that teaching young people to be more tolerant will in-turn make their adult family members more tolerant. Similarly, if your TOC does not reference specific or general program activities, you will eventually need to include more details about how change will occur. When confronted, you must be prepared to defend your TOC and/or amend to ensure it’s clear and convincing to project stakeholders. One way to defend your TOC is to cite existing supporting evidence of the logic applied in your TOC. A defendable TOC should also have plans for collecting data that tests your theory and if the data disproves your theory, the program should be ready to adapt your programming to reflect the reality.
3. You should make sure your TOC is reflected throughout your project.14 Your activities and events, their outputs and outcomes, should be aligned to your theory of how peace will be built. If your project has several objectives, you will likely need multiple TOCs to describe the expected changes under each objective. As your program activities develop, return to your TOCs to make sure they fit. If not, develop new TOCs or adjust program activities to be aligned. It may seem obvious, but it also is important to make sure you do not adopt conflicting TOCs.
Some additional examples of TOCs from peacebuilding projects are as follows.
IF communities have a greater understanding of the drivers of conflict and violent extremism, THEN they will be more likely to engage in processes that build social capital and cohesion and that address community grievances.
IF networks of community peace leaders are strengthened to work collaboratively with each other and local government, THEN local cross-border conflict management will be strengthened.
Develop Project Goals and Outcomes from Your Theory of
Change
Once you have developed your TOC, the next steps are as follows.
1. Map your theories to strategic goals.
Your TOC says that doing X will lead to Y (directly or indirectly). Therefore, Y represents your project goal.
Goals, when achieved, should point directly to your ultimate vision of peace.
2. Map your theories to specific types of change.
Types of change include (but are not limited to): attitude, knowledge, skills, behaviors, maintenance, prevention, process, and structure.15
Changes can be at the individual, group, or institutional levels.
14 Church & Rogers, 2006, p. 14.
15 Church & Rogers, 2006, pp. 18–19.
Developing and explaining a TOC
This is often overlooked in peacebuilding initiatives, yet it is essential for both developing initiatives with a clear peacebuilding aim and for monitoring your progress toward achieving the intended change.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 27
3. Identify specific outcomes for each type of change.
If you are going to change skills, what skills will be changed? Prevention: What will you stop from happening? Maintenance: What will you strive to keep the same? Behaviors: What will people do differently?
For example, here is the TOC we presented earlier in the chapter.
IF young people have a better understanding of the need to be tolerant of
other ethnic groups, THEN their attitudes and views of the “other”
communities will change and have a trickle-down effect on adult family
members. THEN communities will be more tolerant of one another and the
levels of conflict and violence will decrease.
Goals16 developed from this TOC could include increased tolerance for customs and traditions of “the other” among youth and their family members, and reduced reports of conflict between groups to community leadership.
The types of change indicated by the above examples of goals are an attitude change17 (tolerance) and a knowledge change (familiarity and understanding).
The specific outcomes we would expect to see that would indicate the goal has been reached include youth taking the initiative to organize joint events such as sports and cultural events, and fewer community members expressing fear of “the other.”
Below are the same steps, summarized in a table.
Table 4.2: Steps to develop goals from project Theory of Change
Step 1: Goals derived from TOC
Step 2: Type
of change18 Step 3: Anticipated outcomes
a. Reduced reports of conflict between groups to community leadership from 100 to 10 per month by 2016
Attitudes Decreased fear of “the other”
Reduction in distaste for customs and traditional practices of “the other”
Increase in acceptance of “the other” as part of the community
Decrease in blaming “the other” for past grievances
16 The goals stated here include specific targets (how much will change by when). Your initial goals do not have to have specific targets. They can be added (or at least refined) after your baseline study (evaluation that tells you your starting point). Targets should not be chosen arbitrarily, but based on what is reasonable and feasible (as opposed to idealistic or unrealistic).
17 From what we know about this (hypothetical) TOC, this effort also could have a complementary TOC about changes in attitudes leading to improved relationships, leading to peace. In this case, “relationship” would be the “type” of change we are promoting.
18 Church & Rogers, 2006, pp. 18–23, provides examples of how to align specific types of change with specific peacebuilding TOCs.
More on goal setting
A strategic goal should:
1. State the most ambitious impact that an organization can effect (long term)
2. Express the highest level for which the project can reasonably be expected to be held accountable
3. Not include the means of achieving the goal
4. Be as specific as possible and stated as an accomplishment
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 28
Step 1: Goals derived from TOC
Step 2: Type
of change18 Step 3: Anticipated outcomes
b. Increased tolerance for customs and traditions of “the other” among youth and their family members
Knowledge and attitudes
Young people are familiar with specific customs of “the other”
Increase in understanding of why it is important to respect customs and traditions of “the other” community
Young people and family members are more accepting and respectful of customs and traditions of “the other” community
These goals won’t be achieved for some time (several years). However, outcomes can be evaluated periodically. If we see a marked decrease in community members expressing fear after year one, we might know we are on track toward meeting our goal. Or the opposite could be true. Either way, it is better to know when we are on or off track with meeting our goal than to wait until the intervention is over and “hope” we have been successful.
These outcomes will be very important for moving to the next stage. They will be integrated into your chosen conceptual model.
At this stage in the process, if you do not yet have specific project activities in mind, establishing goals and objectives will help you develop your project activities.
Tying Situation Analysis and Theory of Change to MERL
System Design
Figure 4.2, on the next page, illustrates how situation analysis and your TOC are connected to design of a good MERL system. Starting with the situation analysis, each level in the flow chart gets more specific toward designing a good system.
Level of intervention shows the type of change sought as a result of understanding
the nature of the conflict and context, as well as perceived, on-the-ground needs. One change sought in this example is “individual personal change” (as opposed to system-wide or community-level changes).
Areas of intervention/strategies further breaks down the type of change sought
into five specific changes, one of them being “healing and recovery.”
Proposed program activity and target of intervention shows the types of
activities this intervention will offer to promote healing and recovery. The first activity involves establishing trauma centers in five communities with a target of reaching more people than are currently being served by such centers (even if current outreach is zero). The second is to target certain key people (armed group leaders) who are perhaps in greater need of healing and recovery or whose successful healing and recovery is perhaps more important to overall peacebuilding goals than targeting others.
Examples of key changes sought proposes even more specific types of changes
the programs will seek. The first is an output (simply getting people to the trauma centers) and the second is an outcome (improved emotional wellbeing) that should result from applying the knowledge and experiences gained from the trauma counseling.
Examples of ties to MERL system design describes generally how the
organization will monitor and/or measure progress towards achieving these goals.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 29
Figure 4.2: Linking situation analysis to MERL in peacebuilding
Current Situation
Conflict analysis, context analysis, needs assessment, other baseline data
Level of intervention
Areas of intervention/ strategies
Proposed program activity and target of intervention Examples of key changes
sought Examples of ties to MERL system design More people Key people
Individual personal change
(TOC category)
Healing and recovery
(type of change)
Establish trauma healing (TH) centers in 5 communities
Provide trauma counseling for armed group leaders
(specific outcomes)
Increased access to trauma care
Improved emotional well-being
Monitor the number of people who received quality trauma counseling
Evaluate emotional health of targeted populations
Perceptions and attitudes
Facilitate a number of joint social events between two communities with tensions
Hold outreach activities to engage leaders of ex-combatant youth
Increased tolerance between communities
Reduced violence
Monitor how the community perceives changes in their relationships
Evaluate if change in perception has led to reduced violence
Skills and capacity
Train peace committee structures on mediation skills
Train key leaders in the community in conflict negotiation
Improved capacity in mediation skills
Conflicts successfully mitigated
Monitor if training resulted in real skills
Evaluate if the improvement in skills actually resulted in a reduction in conflict
Individual behavior
Establish women’s micro-enterprise groups
Train enterprise mentors in loan making
Increased capacity to make loans
Increased household income
Improved economic recovery
Monitor the number of women engaged in micro-enterprises
Evaluate if micro-enterprise program result in a significant increase in household income
Relationships Hold joint music events for communities with historical conflicts
Gather chiefs of both communities to discuss conflict
Improved social relations between the communities
Improved level of trust
Reduced conflict
Monitor participation from both groups in joint events
Evaluate the effectiveness of social activities in building trust
Evaluate if increased perception of trust resulted in reducing conflict
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 30
Level of intervention
Areas of intervention/ strategies
Proposed program activity and target of intervention Examples of key changes
sought Examples of ties to MERL system design More people Key people
Socio-political change
Group behavior/ relationships
Design a mechanism to improve access of marginalized communities to rangeland resources during the dry season
Conduct outreach to key community leaders to participate with government in designing the system
Increased joint planning between citizens and government
Reduced incidence of conflict between marginalized pastoral communities and rangeland management authorities
Monitor if citizens’ perspectives are being incorporated into government plans
Evaluate if there has actually been improved access to rangeland by marginalized communities
Evaluate if improved access led to a reduction in conflict
Public opinion Mobilize NGOs to disseminate information on a reform policy
Train media practitioners on fair coverage
Improved public opinion on the reform policy
Increased number of unbiased media stories
Reform passed
Monitor public opinion on the policy
Evaluate if media training led to better media coverage
Evaluate to what extent advocacy activities helped pass reform
Institutional change
Advocate for the inclusion of a peacebuilding curricula for teachers in the national teacher training curricula
Train teachers in the curricula
Improved capacity of teachers to help youth mitigate conflict
Reduced violence between school-going youth from different ethnic backgrounds
Monitor teachers’ ability to teach youth how to mitigate conflicts
Evaluate if youth exposed to the curriculum reduced violence using in those skills compared to schools where the curriculum was not taught
Structural change
Lobby for the decentralization of land boards
Reduce the minimum qualifications for leaders vying for leadership posts on land boards
Improved access to land management boards for citizens
Equitable access to leadership positions for marginalized communities leadership
Monitor the number of people from marginalized communities vying for leadership positions in the next elections
Evaluate if land boards have more representative structures
Evaluate the quality of decisions of land boards in terms of equity
Desired vision/societal change
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 4: Theories of Change and Developing Project Goals | page 31
Summary: Key points from this chapter
TOCs are a set of beliefs about how change happens.
Peter Woodrow’s 10 types of TOCs that can be used in peacebuilding programs are: individual change, healthy relationships and connections, withdrawal of the war resources, reduction of violence, root causes/justice, institutional development, political elites, grassroots mobilization, economics, and public attitudes.
When developing a TOC, it is important to understand: 1) What is making the conflict worse? 2) What conditions could reduce the conflict? 3) Who else is doing peacebuilding, and how can our project complement ongoing
efforts?
It is important to ensure that your TOC is reflected throughout the program.
Strategic project goals and outcomes should be informed by your TOC.
Learning Activity: Describe your project’s TOC
Based on your situation analysis findings, describe your project’s TOC (refer to the 10 types of TOCs in Table 4.1). _______________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
______________________________________________________________
Then, review your project’s goals to ensure they are aligned with the TOC and vision of peace.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 32
Chapter 5: Conceptual Frameworks and Assumptions
Chapter Objectives In this chapter readers will:
Understand what conceptual frameworks are and how to develop them
Review examples of conceptual frameworks for peacebuilding programs (results framework, logical framework)
Understand the relevance of conceptual frameworks to program design and monitoring, evaluation, reporting, and learning (MERL)
Learn the role of assumptions
Learning Activities
Create a results framework
Create a logical framework
What are conceptual frameworks?
A conceptual framework is a summary description of how an organization/project thinks change will be brought about. The framework describes the process of planned change and can be presented as a table (e.g., the logical framework) or as a diagram.
Conceptual frameworks enable projects to explain the ideas and reasoning behind their Theory of Change (TOC) and link project activities to overall peacebuilding goals.
Describing your projects through conceptual frameworks helps you and your organization think about (and design/present) project activities and their results as contributing to larger peacebuilding efforts. Conceptual frameworks can help you expand on peacebuilding approaches to further define goals that relate to your TOC and proposed outcomes.
Conceptual frameworks can be used at any stage of the project cycle, even if your project is already in the midst of program activities, but has never clearly laid out how they relate to peacebuilding goals. You can start by populating the framework with the information you do have and fill in the pieces you do not.
Also, revisiting a conceptual framework periodically during the project cycle is a good way to make sure you are still on track as minor (or major) changes are made to programs, evaluations, objectives, etc. throughout the life of the project.
There are many types of conceptual frameworks used by development and peacebuilding initiatives. This chapter presents two that you may find useful:
Results framework
Logical framework (also called a “logframe”)
Each of these models, while different in appearance and structure, include the same four key components,19 as listed in Figure 5.1. Though they may be named differently in different models, (such as “outcomes” instead of “objectives”) for clarity we will label these components “goals,” “objectives,” “outputs,” and “activities” in each of their corresponding sections in the frameworks we are about to describe.
19 Adapted from Church & Rogers, 2006, p. 30.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 33
Figure 5.1: Four levels of conceptual frameworks
In Chapter 4’s learning activity, we wrote program goals and objectives based on our
TOC. Using these as a starting point, we can begin to list short-term program outputs and define the activities that will shape them. Once we’ve established clear goals (keeping in mind that every stage of this process should be collaborative, i.e., goals of all stakeholders should
be considered), it is easier to define specific activities and the resulting outputs.
Advantages and Challenges of Using Conceptual
Frameworks20
Conceptual frameworks bring with them the following advantages. They:
Help a project focus on specific objectives/outcomes
Highlight the key linkages in the TOC that underpin the intervention
Help establish an evidence-based approach to MERL
Offer a systematic approach for program implementers to plan their strategies and to select interventions that are most likely to address targeted problems
Help guide corrective adjustments to activities, reallocating resources, and reevaluating targeted objectives or underlying assumptions
Provide a model for communicating about the resources, activities, and outcomes to project staff, development partners, or other stakeholders
Help build consensus, coordination, and ownership of the project among key stakeholders
Serve as the basic accountability tool for developing an evaluation approach
20 Adapted from International Bank for Reconstruction and Development & World Bank, 2012, pp. 8–10, 41.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 34
But, conceptual frameworks also bring about some challenges.
They require that staff and stakeholders make an up-front investment of time and resources at the start of an intervention.
Interventions’ effects can be difficult to measure fully, especially the unintended consequences that likely were not captured in the framework.
They can become overly complicated, especially if the project indicators associated with the framework comprise a mix of qualitative and quantitative measures being used to measure complex programs (such as peacebuilding programs).
The Results Framework
The results framework organizes your activities into distinct stages, outlining what
happens (your results) in the short-, intermediate-, and long-term within four levels, as shown in Figure 5.2.21
Figure 5.2: Four levels of a results framework
In the previous chapter, we developed desired goals and outcomes from our TOC. The results framework can be used to build on them (and add more or make adjustments) and to tie in the related activities (inputs and processes) and outputs. Below we describe in more detail the four components of the results framework.
Inputs and processes (analogous to project “activities” in Figure 5.1) are the resources
and methods employed to conduct an activity, project, and/or program.
Inputs can be:
Physical, such as equipment rental or purchase
Material, such as supplies and provisions
Human, such as labor costs for salaries, technical assistance, and staff
Financial, such as travel, per diem costs, direct costs, and indirect costs
Processes are the methods or courses of action selected to conduct the work, such as training, organizing, publishing, lobbying, service provision, and message promotion. Direct results from inputs and processes are generally seen quickly (0–2 years) and often are measured through monitoring activities.
Outputs are information, products, or results produced by carrying out activities or
projects. Outputs relate to completion of activities and are the type of results over which managers have a high degree of influence. Outputs reflect what you hoped to produce from a particular input or process (or set of either). For example: You decide the process you want to use is to train people. The “number of people trained” is the result at the input/process level, while “knowledge level increased” would be the result at the output level, the assumption
21 Inputs and processes are sometimes segregated into two separate levels.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 35
being that if you train people they will increase their knowledge on a given subject.22 Outputs usually reflect a result achieved in a relatively short time period (0–2 years) and are often measured through monitoring activities.
Outcomes are broad changes in development conditions. Outcomes help us answer the “so
what?” question. For example: We trained 100 people and increased their knowledge, but did they change their behavior? Outcomes often reflect change in attitudes and/or perceptions, emotional well-being of individuals, household income, access to rangeland or water for marginalized communities, and other changes and help us analyze how our activities and projects scale up or contribute towards these development outcomes. Outcomes usually reflect a result achieved over an intermediate time period (2–5 years).
Impacts are the overall and long-term effects of an intervention. Impacts are the ultimate
result, such as peaceful co-existence of previously conflicting communities. Impacts usually reflect a result achieved over a longer time period (5–10+ years).
A results framework organizes these four levels of results so you can see how inputs and processes lead to outputs, outputs lead to outcomes, and outcomes lead to impacts. If you choose to use this framework, you will later construct/select indicators (in Chapter 6) to help you measure attainment of each of these stages. Generally, a results framework will be accompanied by a separate MERL and indicator plan. Table 5.1 describes the type of information normally present in a results framework and how it is structured, and Table 5.2 shows the PEACE II results framework as an example.
Learning Activity: Results framework
Take what you already know about your program (it might be the activities, desired impacts, some of everything) and fill in the sections of the results framework in Table 5.3 for which you have relevant information. Then see if you can fill in any missing pieces (or add on to those you have).
Does this framework “make sense” to you? Does it seem like it will help structure your program and give guidance for later MERL? How does it align with the project’s TOC(s) that you developed earlier?
22 However, it is worth noting that some may consider “number of people trained” as an output indicator, with “knowledge level increased” as a shorter-term outcome indicator. The levels at which you consider indicators depends on several factors, such as project length, overall goal/objective, donor definitions.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 36
Table 5.1: Example of a results framework
Inputs/Processes Outputs Outcomes Impacts (Goals)
Resources and processes used to produce a change/result
Examples
Inputs delivered or implemented:
Staff hired
Grants awarded
Funds disbursed
Materials/equipment purchased
Processes conducted:
Situational analysis completed
Appropriate fora, alliances, committees established
Stakeholders mobilized
Communities mobilized
Trainings conducted
Meetings held
Networks established
Advocacy (radio, print, dramas) messages developed
Short-term change, effects, and results
Examples
Knowledge/awareness, understanding change:
Increased understanding of the conflict dynamics
Increased demand for grassroots peacebuilding
Capacity/skills change:
Improved skills and abilities to negotiate conflicts
Stakeholder engagement and dialogue expanded
Access/expansion change:
Networks for peace supported
Quality change:
Laws and policies developed which promote stability
Intermediate change, effects, and results
Examples
Proportion/coverage change:
Increased action by an enlarged constituency for demanding and managing grassroots peacebuilding
Attitude change:
Increase acceptance of members of ‘the other’ community
Behavior change:
Improved pastoral and land use management
Actual policy change
Laws and policies implemented which promote stability (e.g., land use rights and management)
Long term change, effects, and results
Examples
Political change:
Reduction in violent conflict
Human rights, socio-cultural, and empowerment change:
Increased civil society participation in political processes
Socio-economic and/or health status change:
Improved livelihoods (income, health, education)
Resource management change:
Increased legitimacy, transparency, and accountability in governance
Project Level (implementation/progress monitoring)
Program Level (progress/results monitoring)
Strategic Level (Effectiveness) (results/impact monitoring)
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 37
Table 5.2: Example of PEACE III results framework
Inputs/Processes Outputs Outcomes Impacts
Conduct participatory learning and action workshops with community representatives to identify and prioritize peace dividends
Create space for dialogues on cultural norms and practices
Identify and train trauma healing (TH) counselors
Conduct TH sessions
Assess community peacebuilding capacities to identify capacity building needs
Facilitate formation and/or strengthening of peace networks
Conduct Appreciative Inquiry/strengths, weaknesses, opportunities, and threats (SWOT) analysis with local government peace institutions
Targeted capacity building interventions for government peace institutions
Develop customized capacity assessment tool for PEACE III partners
Identify capacity building priorities for PEACE III partners
Peace dividends constructed
Communities sensitized on the role cultural practices play in conflict and peacebuilding
Cadre of trauma healers with knowledge and insight on who can assist communities withstand conflict shocks
Targeted capacity support extended to communities
At least one peace network per corridor established and strengthened
Joint visioning and planning conducted and joint activity plans developed
Increased number of cross-border mechanisms established or strengthened
Increased local government participation in peacebuilding initiatives (initiated by PEACE III)
Organizational Capacity Assessment conducted for local partners
Targeted capacity support extended to implementing partners
1.1: Communities are more open to social reconciliation
1.2: Communities’ peacebuilding capacities mobilized
1.3: Local governments partner with their cross-border counterparts and with communities in conflict management
1.4: Capacity of Horn of Africa NGO partners to support cross border conflicts management increased
Strengthened cross-border conflict management
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 38
Table 5.3: Blank results framework for the Learning Activity
Inputs/Processes Outputs Outcomes Impacts
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 39
Logical Frameworks
A logical framework or logframe is similar to a results framework in that it explains
how a project’s day-to-day activities will achieve results. One main difference between a logframe and a results framework is that the latter describes the results as if the program had already been completed while a logframe describes what will happen in the future.23
A logframe promotes good project design by clearly stating the defined project logic and components. The logframe is usually formed as a chart that shows a hierarchy of:
Four levels of the causal relationship: activities, outputs, purpose, and goal
Indicators24 of performance
Means of verifying the indicators
Important risks and assumptions
The logframe provides a summary of what the project aims to achieve and how, what the main assumptions are, and a basis for developing the activity’s MERL system. Progression from one level to the next is based on “if/and/then” logic; for example, if the activity at the lowest level takes place and the assumptions hold true (or the risks are not realized), then the expected output at the next level can be achieved.
Basic Logframe Outline (Hierarchy of Objectives)
The basic structure of a logframe is provided in Table 5.4. For each of the four levels of
causal relationships (goals, purpose, outputs, and activities), you explain:
The direct or indirect measures that will verify to what extent the objectives have been
fulfilled (the objectively verifiable indicators)
The source of the information for the measurements or verification specified in the
indicators column (the means of verification)
The important events, conditions, or decisions necessarily outside the control of the project, but which must remain favorable for the project objective to be attained (the
assumptions or external factors)
Table 5.4: Logframe structure25
Objectively verifiable indicators
Means of verification
Assumptions (external factors)
Goal (goal)
Long-term impact sought/overall vision
Qualitative/ quantitative evidence to judge achievement of the goal (e.g., strengthen cross-border conflict management)
Specific sources of data to quantify or assess indicators of the goal (e.g., percentage of community respondents who perceive effectiveness in local conflict-management institutions)
External factor necessary to sustain the objectives in the long term (e.g., stability of operating environment)
23 Church & Rogers, 2006, p. 38.
24 We will go into more detail about constructing indicators in Chapter 6.
25 Adapted from DFID. 1997. Guidance on Humanitarian Assistance. Available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/404342/Humanitarian-Response-Funding-Guidelines-2015.pdf.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 40
Objectively verifiable indicators
Means of verification
Assumptions (external factors)
Purpose (objective/result)
Intended immediate impact of the project
(purpose to goal)
Qualitative/ quantitative evidence to judge achievement of the purpose (e.g., openness to social reconciliation)
Specific sources of data to quantify or assess indicators of the purpose (e.g., level of community openness to social reconciliation)
External conditions necessary for project purpose to contribute to the achievement of the project goal (e.g., willingness of communities to devote their time to participate
in the project)
Outputs (output/ intermediate result)
Tangible/intangible result(s) to achieve the purpose expected from the project
(output to purpose)
Qualitative/ quantitative evidence to judge achievement of the outputs (e.g., construct peace dividends)
Specific sources of data to quantify or assess indicators of the outputs (e.g., number of people accessing peace dividends)
External factors that, if present, can restrict progress from outputs in achieving project purpose (e.g., conducive environment to support engagement with communities during program implementation period)
Activities (activities)
Actions needed to achieve each output of the project (these are things that the project can control)
(activity to output)
Evidence for monitoring and reporting implementation in line with the logframe (e.g., conduct Participatory Learning and Action to identify and prioritize peace dividends)
Specific sources of data to confirm implementation of planned activities (e.g., number of peace dividends completed)
External factors that, if present, can restrict progress from activities in achieving outputs (e.g., conducive environment to support engagement with communities during program implementation period)
The Role of Assumptions
Assumptions are the “external factors [that] are not influenced by the project, but may
affect its implementation and long term sustainability.”26 Identifying assumptions is critical at the project’s design stage. If you are designing the project using a logframe, the assumptions are included in framework itself. If you are using a results framework, you can still state your assumptions for each goal, objective, output, and activity in a simple list or table, such as:
Goal 1: Assumption:
Objective 1: Assumption:
Output 1: Assumption:
Assumptions can come in different forms.
Examples of things we assume will (or will not) take place include:
Approval of community-based organizations (CBOs) to be funded under the program by the funder
26 Better Evaluation. 2015. Logframe [web page]. Available at http://betterevaluation.org/evaluation-options/logframe.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 41
Communities in target areas will support the peacebuilding initiatives
Cross-border local authorities will work together to promote peace in their areas
Local, national, and regional peacebuilding entities will work together to promote peace in target areas (vertical linkages)
There will be no new conflict flare-ups or natural disasters that inhibit the work
Assumptions that we take for granted or believe to be true—or we believe are very likely to be true—that affect results include:
Partner organizations will value capacity building and will make time for institutional strengthening activities
Enough people will be interested in supporting peacebuilding efforts
The program will mobilize a critical mass to effectively advance peacebuilding initiatives
We can get off track in our planning, budgeting, and problem solving if we don’t list our assumptions or have assumptions that are wrong.
We need to deeply think about the things we are “assuming” because they can affect the program results.
We need to write down our really important assumptions.
We need to test our assumptions to be sure that our assumptions are based in reality.
You can assess the risks coming from assumptions using Figure 5.3. Note that if an assumption is very likely to prevent the program from taking off or advancing, you either need to redesign the program or address the assumption directly to make it true.
Apart from redesigning a program based on the likelihood of an assumption preventing the execution of such a program at the start of a program, programs also could be redesigned during the course of the program if foreseen assumptions increasingly become likely to affect the intended outcome of such a program. Assumptions could give grounds for redesigning programs.
Figure 5.3: Assessing risks associated with program assumptions27
27 Church & Rogers, 2006, p. 35.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 5: Conceptual Frameworks and Assumptions | page 42
Learning Activity: Logical framework
Take what you already know about your program (it might be the activities, desired impacts, some of everything) and fill in the logframe sections in Table 5.5 for which you have relevant information. Then see if you can fill in any missing pieces (or add on to those you have).
Does this framework “make sense” to you? Does it help to structure your program and give guidance for later MERL? How does it align with the project’s TOC(s) that you had developed earlier?
Table 5.5: Blank logframe
Objectively verifiable indicators
Means of verification
Assumptions (external factors)
Goal
Purpose
Outputs
Activities
Summary: Key points from this chapter
Conceptual models help present a project’s TOC by showing how activities at the lowest level lead to achieving the project goal.
There are many types of conceptual models in use today, but two common ones are the results framework and the logical framework.
All models include four levels: activities, outputs, outcomes, and impact, though terminology and how each type of conceptual model is structured will differ.
Conceptual models provide program implementers with many advantages, but also present several challenges. For example, they require investment of time and other resources at the program design stage and they do not account for unintended consequences that result from the project.
It is important to recognize and test the risk associated with your project assumptions.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 43
Chapter 6: Indicators
Chapter Objectives In this chapter you will:
Learn how indicators are used to answer questions about your project's outputs, outcomes, and impact
Learn how to develop and assess indicators
Learn how to use an indicator plan
Get familiar with the U.S. Government Foreign Policy Indicators for Conflict Prevention and Mitigation
Be introduced to the PEACE III indicator protocols to help determine indicators that apply to your project
Learning Activity
Indicator plan
Overview
Indicators are the building blocks of monitoring and evaluation. Indicators signal change.
The Organization for Economic Cooperation and Development (OECD)28 defines an indicator as a “quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor.”
Indicators are not just anything your project wants to measure, and every measure is not an indicator. Indicators are meant to reduce a large amount of data down to its simplest form. For example, when you want to buy a car, unless the engine has been replaced, the best indicator of a car’s condition is its odometer because it tells you how far the car has driven and, therefore, how much wear and tear the engine has undergone.
Indicators also are not the same as goals, objectives, results, or targets. They are developed to measure results at the four levels of results depicted in conceptual models: activities/input, output, outcome/objectives/results/purpose, and goals/impact. Indicators do not specify a particular level of achievement; thus, words like improved, increased, gained, and decreased do not normally belong in an indicator. Rather, indicators describe the unit of information to be measured over time, for example, “change in knowledge,” which could increase or decrease, but this is further specified in the indicator target. When compared with targets, indicators may signal the need for management action and help us determine if objectives are being met.
Indicators can be especially useful when straightforward data cannot tell us what we need to know. Instead, we can ask a series of questions that indicate the answer to our question. In other words, when a question cannot be answered directly by one piece of data, we collect one or more indicators instead that give us a good approximation of what the answer may be. Table 6.1 provides an example of overall project questions and their illustrative indicators.
28 OECD 2002.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 44
Table 6.1: How indicators help answer project questions
Question Indicator
1. How many trainings were held?
1. Number of trainings held
2. Were trainings effective?
2a. Percentage of participants correctly answering 8 of 10 knowledge-based questions on the quiz at the end of each training
2b. Percentage of participants self-reporting improved understanding of [the topic] after the training
2c. Percentage of participants self-reporting that they did [x thing that they learned at the training] within 6 weeks of the training
3. Has trust increased between groups A and B?
3a. Percentage of individuals from group A indicating they would do business with a member of group B after the training
3b. Percentage of businesses in region A operated by individuals from group B after 3 years
3c. Percentage of parents/guardians from group A indicating they would allow their children to be educated with a member of group B after the training
Indicator 1 in Table 6.1, about the number of trainings held, is straightforward because the question it answers is straightforward. This indicator represents output data we can track through training sign-in sheets or registration records. Questions 2 and 3 are less straightforward. Often a single indicator is not sufficient to answer our questions. Sometimes we want to look at the issue from multiple perspectives; other times the issue at hand is too complex to be addressed with a single line of inquiry. For example, indicators 3a and 3b could be used to answer the question “Have business opportunities increased for people in group B?” In fact, if we see positive change in these indicators, they might have little to do with trust building and more to do with increased business opportunities or a better enabling environment.
This brings up another point: Indicators do exactly as their name implies, they indicate the answer to a question that cannot be directly answered. It is important when both selecting indicators and analyzing data to carefully consider if an indicator’s ability to show a result is caused to some extent by the project-sponsored activities or whether results demonstrated by the indicators do not stem from project activities.
We ask a particular question because we think it will tell us something about our intended program outputs, outcomes, or impacts. For example, we would ask a question about trust building if a goal of our program is to increase trust among opposing groups. This also presumes that we did something specific in our program that aimed to build trust (directly or indirectly). But, even if the indicators we choose all show positive results (there is an increase in people doing business across groups and there are more businesses in region A operated by people from group B), we still can’t be certain that those changes happened because of increased trust.
Also, even if they are the result of increased trust among groups, we can’t be certain that our program efforts directly led to that change. Perhaps there was a government-run program operating in the same region with the same goal of trust building that was actually responsible for the change. Perhaps a sub-group or leader within the community that had been supporting division and separation of the groups is no longer active. The point is that it is very difficult to control for all the factors that influence change in individuals and communities.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 45
The primary function of indicators is to show us what changes have occurred, to what degree, and over what period of time. They may not tell us for certain how or why a change has occurred, thus it is important to supplement indicator data with documented qualitative data to help tell this story of change.
How to Develop Indicators
Indicators are written to measure the goals and objectives29 we have chosen (and their relevant outputs, inputs, and processes). Indicators can be written for overarching program goals and specific intervention objectives. They can contain up to seven specific pieces of information:30 1. The thing to be measured/changed 2. The unit of measure that informs the
change 3. The starting point (baseline
information) 4. The scope, size, or degree of
intended change 5. The quality or standard of change
desired 6. The target population 7. The timeframe
We can divide indicators into two basic categories: quantitative and qualitative.
Quantitative indicators measure
information in specific amounts or quantities, such as the number of people who did something, the number of times something specific happened, or the amount of increase or decrease seen in some discrete activity. An example for a peacebuilding program may be “Number of community members participating in trauma healing (TH) initiatives.”
Qualitative indicators include more subjective information, such as opinions, beliefs, or
perceptions, and typically are not able to be quantified in the way quantitative indicators are because of the way data is collected, such as through open-ended or guided questions during focus groups or key informant interviews. For example, a qualitative indicator in a peacebuilding program may be “Improved sense of security among TH participants.” To gather data for this indicator, you are unable to survey all or a sample of participants, rather you speak with individual or groups of community members to understand the general sense of how they feel about security after participating in TH. This qualitative indicator complements the straightforward count of participants in TH with participants’ sense of security as a result of those sessions.
29 See Chapter 4 for more on determining goals and objectives.
30 Church & Rogers, 2006, p. 45.
Example of an indicator with five of the seven pieces of information
Percentage of cattle herders in Karamoja who report disputes over grazing land to their local council one or more times per month in the last 3 months.
1. The thing to be measured: who report discputes over grazing land to their local council
2. Unit of measure: percentage 5. Quality or standard of change desired: one or
more times per month 6. Target population: cattle herders in Karamoja 7. Timeframe: by December 2016
Note: Once baseline data is collected, the project can establish the starting point and size of change that it wants to achieve within the specified timeframe. For example, the project can at baseline establish that currently only 5% of cattle herders report disputes to their local council one or more times per month. Taking into consideration various project factors (e.g., planned interventions, duration, funds) the project can decide that the size of change it can reasonably achieve is raising the percentage by
25%, thus the target of 30% for this indicator.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 46
One way of looking at the difference between quantitative and qualitative indicators is that quantitative data tells us what happened and how much or how often it happened, while qualitative data tells us how what people thought about it and what impact it had on them. It is highly recommended to use a mixture of qualitative and quantitative indicators to measure the attainment of your objectives.
Table 6.2: Examples of qualitative and quantitative indicators and data sources
Example indicator Potential data source
Qualitative
Community openness to social reconciliation Focus groups
Improved sense of security among TH participants
Key informant interviews
TH participants less inclined to partake in violence
Key informant interviews
Quantitative
Number of peace dividend initiatives benefitting two or more conflicting communities
Project monitoring documents
Performance of community conflict management structures
Pact’s Community Performance Index
Number of initiatives led by community peace actors to address local conflicts
Project monitoring documents
Number of cross-border peace initiatives that involve local government
Project monitoring documents
Percentage of community peace initiatives that receive tangible local-government support
Project monitoring documents
Review of government actions
Number of new linkages among local peace-building organizations
Pact’s Organizational Network Analysis
Percentage of community respondents who perceive effectiveness in local conflict management institutions
Community-wide, face-to-face survey
The U.S. Department of State’s Standard Foreign Assistance Indicators31 are a good place to search for indicators that have been tested and used. Most U.S. Government-funded projects will be required to use some of these indicators. PEACE III has adopted several of them for use in the program.
Developing or choosing the right indicators is the heart of designing a practical MERL system. This process can be tedious and exacting, so you cannot expect to sit down in one afternoon and develop all the indicators your project needs. It should be an iterative process involving brainstorming a range of possibilities, seeking opinions from various staff and stakeholders, and assuming different perspectives as you develop indicators.
31 See http://www.state.gov/documents/organization/231895.xlsx, specifically the conflict mitigation and reconciliation indicator definitions.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 47
Qualities of a Good Indicator
Once you’ve gathered a large list of potential indicators, you should refine them. In addition to being appropriate for measuring achievement of your project objectives, all indicators should meet three specific quality standards: reliability, feasibility, and utility.32
Reliability is the extent to which one can reasonably assume that they can acquire correct
or accurate information. If it is likely that two or more people seeking the same information will come back with different results/answers (repeatability), the indicator may not be reliable. An indicator may be deemed unreliable if:
It comes from generally unreliable sources (e.g., subjective assertions)
The question asked can be interpreted in multiple ways (e.g., the act of “participating” in an activity can be interpreted in different ways by different people)
Factors such as time of day or time of year of collection are likely to influence answers (e.g., during the dry season people will be supportive of activities that involve encroaching on other communities’ grazing land)
People are prone to lie about it, not know the answer, or give an inaccurate answer (e.g., which individuals were involved in the cattle raid?)
Feasibility33 is the extent to which the information needed can be readily acquired. It
requires that you be able to access the source of information and the specific data you need. Barriers to access include:
Security or safety concerns
Sensitivity of the topics
Confidentiality of information sought
Unwillingness of sources to participate
Physical barriers (e.g., information is in a location that is difficult to get to, that is prone to severe weather)
Utility is the extent to which the information is actually useful. How much will what we learn
help us to make or adjust programmatic or strategic decisions? When considering whether an indicator has utility, imagine having the results in front of you and asking yourself, “What will I do with this information?” Individual indicators might not, by themselves, inform decision-making, but may be considered collectively for utility in some instance.
Another way projects use to assess the quality of indicators is based on the five criterion:
Specific, Measurable, Achievable, Relevant, and Time-bound (SMART). Table
6.3 describes the criteria for SMART indicators.
Table 6.3: SMART indicator criteria34
S Specific: Is the indicator specific enough to measure progress toward results?
M Measurable: Is the indicator a reliable and clear measure of results?
A Attainable: Are the results in which the indicator seeks to chart progress realistic?
R Relevant: Is the indicator relevant to the intended outputs and outcomes?
T Time-bound: Are data available at reasonable cost and effort?
32 See Church & Rogers, 2006, p. 45.
33 In situations where tensions and/or conflict are high, one cannot underestimate the importance of indicator feasibility. See Chapter 11.
34 Adapted from UNDP, 2009, p. 63.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 48
Table 6.4 provides examples of indicators that are well formed (“good”) and poorly formed (“bad”). Note that indicator wording does not need to capture the direction of change you are hoping for. Rather, this would be visible in the targets set—the targets would show increases—and the actual achieved figures could be increases or decreases.
Table 6.4: Examples of good and bad indicators
Good indicators Bad indicators
Number of community members involved in peacebuilding
Increased number of community members involved in peacebuilding
Number of workshops conducted (disaggregated by type of workshop)
Number and type(s) of workshops conducted
Number of conflict incidents reported Number of people killed in violent conflict35
Amount of rainfall Level of rainfall
Level of beneficiary satisfaction Degree of happiness
Indicator-Related Terms
Table 6.5: Other terms used in relation to indicators
Term Definition
Baseline This is a record of what exists in an area prior to the start of a project. It is primarily a benchmark for the future. The baseline values establish the starting point from which change can be measured.
Targets These are values against which the actual program/project achievements are measured. They describe the magnitude or level of results expected to be achieved. Targets should be realistic and, where possible, informed by past achievements.
Proxy indicators These are alternate measures used to stand in for another indicator when obtaining direct information is too difficult, time consuming, or sensitive. For example, household consumption of maize could be a proxy indicator for household income.
Indicator protocols
These are instruction sheets that describe the indicator in precise terms and identify the plans for data collection, analysis, reporting, and review. Examples of precise information contained in an indicator protocol are indicator definition; unit of measure; disaggregation; data collection, collation, analysis, reporting, storage, and use; data quality; and targets.
Sources of verification
These are sources you have or use to confirm or substantiate the data, for example attendance lists and event reports.
Unit of measure This specifies what exactly the indicator will measure or count, for example number of events funded by the U.S. Government or number of people trained.
Disaggregation This refers to how the indicator data will be broken down at the time of reporting to stakeholders, for example by location, gender, or age.
35 This may be considered a “bad indicator” from an early warning perspective. This indicator has less value in terms of informing program action to “save lives” when compared to the number of incidences reported. The assumption here is that tracking occurrence of incidences can help a program take timely action to focus on hotspots, thereby reducing the likelihood of escalation and loss of lives.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 49
Indicator Definitions
In order to ensure that our indicators are not subject to misinterpretation and are clearly understood by everyone who will use them, it is important for us to define each indicator. An indicator definition clarifies the key terminologies used within the indicator to ensure data collectors and users measure or interpret the same thing. Using the example indicator that we developed earlier in the chapter, “percentage of cattle herders in Karamoja who report disputes over grazing land to their local council one or more times per month by December 2016,” it is important to define what the project means by “cattle herders in Karamoja” and “report disputes to the local council.” Doing so will help ensure there is common understanding of which group of people within the Karamoja community the project will count as “cattle herders” and what the project will count as an “incident” that has been “reported” to the “local council.” Below is an example of an indicator definition.
This indicator will count cattle herders from among the Pokot and Turkana communities who are residents and derive their livelihood from keeping and grazing cows within the administrative boundaries of Pokot and Turkana counties. A dispute refers to a disagreement between two cattle-keeping communities over issues of grazing land that may or may not result in fighting between the two communities. Reporting to a local council means formally informing the local authority responsible for ensuring equitable use of grazing land within these communities about a dispute over grazing land with members of the other community.
In addition to defining the indicator, it is important to specify the disaggregation that will be used when reporting on each indicator, for example location, name of community, and type of local authority.
Collecting Indicator Data
Once we have developed and defined the set of indicators we want to track and assess, they are integrated into the project’s data collection instruments, such as event documentation; reporting templates; household surveys; and observation, focus group, or interview protocols.
All of the information about the indicator is tracked in the project’s indicator plan. Table 6.6 provides an example from a peacebuilding program.
Note: Implementers of U.S. Agency for International Development (USAID) programs will be asked to complete Performance Indicator Reference Sheets (PIRSs) as part of their MERL plan.36 PIRSs are a short table for each indicator detailing the indicator definition, unit of measure, source, disaggregation, and other key pieces of information. PIRSs are invaluable resources for all MERL and program staff to understand exactly what is committed to be measured and how to follow the protocols established.
36 See more on PIRSs at https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 50
Table 6.6: Example indicator plan
Indicator (type)
Baseline and Target 2015 Definition and disaggregation
Data source(s)
Unit(s) of measure
Data collection tool(s)
Data collection frequency
Reporting frequency
Goal: Strengthened local cross-border management
Percentage of community respondents who perceive that the local conflict management institutions are effective
(Type: outcome)
Baseline: 5%
Target for 2015: 10%
Definition: Community respondents are defined as those living in PEACE III target areas. The respondents could be people from the same ethnic group/clan living in either rural or urban areas. PEACE III takes success in managing conflicts as a proxy measure for effectiveness in conflict management.
Perception of effectiveness is measured by survey respondents expressing how successful local conflict management institutions are in managing conflict on a scale of 1–4, denoting very unsuccessful, unsuccessful, successful, and very successful.
Local conflict management institutions denotes the following types of institutions:
Local peace committees
Government security committees
Traditional conflict management institutions
Community-based organizations (CBOs)
Disaggregation: Location (cluster, corridor), age, youth, gender, and type of local conflict management institution
Baseline survey, midterm survey, endline survey, perception survey
% of total community respondents
Household survey questionnaire
Focus group discussion guide
Baseline, midterm, endline
Baseline, midterm, endline
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 6: Indicators | page 51
Summary: Key points from this chapter
Indicators are not the same as goals, objectives, results, or targets.
Indicators describe the unit of information to be measured over time.
The primary function of indicators is to show us what changes have occurred, to what degree, and over what period of time.
Indicators can be classified in two basic categories: quantitative and qualitative.
Quantitative indicators measure information in specific amounts or quantities, such as the number of people who did something or the number of times something specific happened.
Qualitative indicators include more subjective information, such as opinions, beliefs, or perceptions, and are not typically able to be quantified in the way quantitative indicators are because of the way the data is collected, such as through open-ended or guided questions during focus groups or key informant interviews.
Criteria based on the following five characteristics are used to assess the quality of the indicators: Specific, Measurable, Achievable, Relevant, and Time-bound (SMART).
In addition, all indicators should meet three specific quality standards: reliability, feasibility, and utility.
Learning Activity: Indicator plan
Refer to the indicators and targets in your award document. Read through the indicator definitions found in the indicator protocols that are in the award document or that you are given for this exercise, and reflect on your project objectives/activities. Review the activities currently linked to each indicator and update indicators and the ways of measuring them as necessary, taking into consideration the following.
The appropriateness of the activities/individuals that you are counting to report on this indicator: Do they fit with the definition in the project’s indicator protocol of types of activities to be counted?
Are your targets realistic given your available budget, staffing, award period, and past experience? Please update targets accordingly for your final award.
Prepare to submit to your funder an updated indicator plan that’s well aligned to your funded project.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 52
Chapter 7: Monitoring
Chapter Objectives In this chapter you will learn:
What monitoring is and how it helps peacebuilding efforts
Three types of monitoring
How to develop a monitoring plan
Learning Activity
Develop a monitoring plan
What is monitoring?
Monitoring is an activity carried out throughout the life of the intervention that provides
project stakeholders with concrete information from which to make strategic decisions. In peacebuilding, we often monitor three specific areas: the conflict/context, our program implementation, and our assumptions. In other words, we use monitoring to periodically answer the following questions.
Has the conflict context changed?
Have we done what we planned to do by this point?
Has what we’ve done had the effect we expected it to?
Let’s look at our example Theory of Change (TOC) and goals from Chapter 4, with added sample indicators, as shown in Table 7.1.
Table 7.1: Example TOC with goals, outcomes, and sample indicators
TOC: IF young people have a better understanding of the need to be tolerant of other ethnic groups, THEN their attitudes and views of the “other” communities will change and have a trickle-down effect on adult family members. THEN communities will be more tolerant of one another and the levels of conflict and violence will decrease.
Step 1: Goals derived from TOC
Step 2: Type of change37
Step 3: Anticipated outcomes
Sample indicators (more and others are possible)
a. Reduced reports of conflict between groups to community leadership from 100 to 10 per month by 2016
Attitudes Decreased fear of “the other” % change in # of members of Group A who self-report fear of members of Group B
Reduction in distaste for customs and traditional practices of “the other”
% change in # of members of Group A who self-report that they disagree with the customs or traditions of Group B
Increase in acceptance of “the other” as part of the community
% change in # of members of Group A who, when asked to describe their community, mention individuals or groups from Group B
Decrease in blaming “the other” for past grievances
% change in # of members of Group A who self-report that past conflict with members of Group B was at least in some part their own fault
37 Church & Rogers 2006 provides examples of how to align specific types of change with specific peacebuilding
TOCs. See www.sfcg.org/programmes/ilr/ilt_manualpage.html, Chapter 2, pp. 18–23.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 53
TOC: IF young people have a better understanding of the need to be tolerant of other ethnic groups, THEN their attitudes and views of the “other” communities will change and have a trickle-down effect on adult family members. THEN communities will be more tolerant of one another and the levels of conflict and violence will decrease.
Step 1: Goals derived from TOC
Step 2: Type of change37
Step 3: Anticipated outcomes
Sample indicators (more and others are possible)
b. Increased tolerance for customs and traditions of “the other” among youth and their family members
Knowledge and attitudes
Young people are familiar with specific customs of “the other”
% change in # of young people of Group A who can name 2–3 specific customs of Group B
Increase in understanding of why it is important to respect customs and traditions of “the other” community
% change in # of members of Group A who self-report that they understand the importance of respecting the customs and traditions of Group B
Young people and family members are more accepting and respectful of customs and traditions of “the other” community
% change in # of members of Group A who self-report that they do respect customs and traditions of Group B
How might we monitor progress toward our goals? We would probably track program participation (e.g., monthly, quarterly) and resources being used (e.g., cost, staffing) to be sure we are not currently in danger of surpassing budgets. Tracking progress toward specific indicators is likely to be done through process (periodic) evaluations (see Chapter 8).
Conflict and Context Monitoring
Because the environments where we conduct peacebuilding activities are highly dynamic and often changing, it is important to monitor the context and conflict throughout an intervention (see Chapter 3). Doing so helps us know whether our activities are still appropriate and if we need to make minor or major changes, either immediately or in the future.
Table 7.2: Example conflict/context monitoring outcomes and potential resulting program changes
If monitoring shows us that… We might have to…
Violence has escalated Increase security measures
Postpone or cancel the project
Change the focus and/or TOC
Relevant legislation has been passed Adapt or develop new interventions to meet new requirements or better align with legislative goals
Types or sources of conflict have changed
Re-evaluate our strategic value: Are our objectives still in line with what is needed for peacebuilding?
Deciding when and how to monitor these changes depends on resources available (does someone have time to monitor this monthly? quarterly?) and your perception of how quickly the context may change. How fragile is the situation? Are there particular triggers or flashpoints (specific events that would impact the conflict positively or negatively) that are likely to occur? Are there patterns to watch for (e.g., movement of populations, frequency of aid delivery) that would indicate a change in the conflict? In ideal circumstances, how often
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 54
should we monitor the situation? Focusing on key contextual elements (rather than trying to cover everything) will help you design a context-relevant monitoring plan.
The Measuring Progress in Conflict Environments (MPICE) framework38 is a good reference for identifying the specific questions you will want to answer through conflict monitoring. Suggested methods39 for getting the information you need to monitor a conflict include:
Qualitative methods
- Content analysis: surveying newspapers, blogs, and other public media sources (secondary sources)
- Expert knowledge: panels of local program staff, local or national leaders or others in positions of authority, journalists, researchers, people involved with other local groups or community organizations, or anyone with a direct interest in knowing what is happening on the ground (including those involved in the conflict itself)
Quantitative methods
- Primary data: information that can be collected and quantified (counted) related to security, economic development, living conditions, etc.
- Survey or polling data:
Primary sources: Public opinion surveys, including surveying local program staff
Secondary sources: Datasets or information from international organizations that monitor crises and publish relevant information publically (Keep in mind that these broader monitoring tools cannot replace the valuable information you can learn through monitoring your specific context locally.)
Implementation Monitoring
Regardless of the duration of the intervention, it is important to monitor step-by-step progress toward achieving ultimate goals in terms of project administration, inputs, processes (activities), and outputs.
Generally speaking, implementation monitoring:
Compares what has been planned against what has been accomplished
Is used to assess and validate what has happened in the past and set the stage for what has to happen in the future
Is quantitative in nature, measuring how much or how many of a thing happened
Example questions implementation monitoring answers include: How many trainings were held and how many people completed the training? How much money was spent on each of X activities? How many times did Y happen?
For example, consider a project that has a plan to deliver 20 trainings to community leaders in years 2–4 of a five-year project. At the end of year 2, monitoring data might show that the
38 See Dziedzic et al. 2008.
39 Ibid. See the full report for descriptions of the pros and cons of each of these methods.
Note on using external sources of information
Accessing reliable data is a considerable challenge in fragile environments. When getting information (e.g., data, stories) from sources external to your organization, be careful to consider its quality and any potential biases. Is the data source reliable? Does the source have reason to give out false information? Were ethical and/or statistically sound practices used to collect the information? See Chapter 11 for more information on data quality.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 55
project has delivered two trainings, which may be on target with the project plan or may indicate that the team is behind on holding these trainings. Or perhaps 15 trainings have been held by the end of year 2, which could indicate higher than anticipated demand for the activity, in which case the organization may want to adjust its planning or search for more funding, staffing, or opportunity to increase the number of trainings it can hold in subsequent years.
The previous example was of monitoring the magnitude of outputs (how many of this,
how much of that). We can similarly monitor progress toward achieving changes.
If a program’s target is to increase “something” from 10% to 75% over 5 years, we may use monitoring tools to look for incremental improvements every six months or a year, for example, increasing to 20% after year 1 or 35% after year 2.
Achievement of planned project stages also is important to monitor. Interventions often are planned incrementally (e.g., first introduce a savings product, then offer business development training, then offer business loans) and subsequent activities may each require
significant planning and preparation. Monitoring the implementation of each project
stage is important for informing how and when to plan and adjust future stages. In the
examples in Table 7.3, it may be wise to monitor each activity even more closely, such as every four to six months, to check progress toward each specific activity’s target.
Table 7.3: Example changes in project plans based on implementation monitoring outcomes
Objective to be monitored If the target is not met we may have to…
If the target is exceeded we may have to…
Year 1: 200 community members complete trauma healing (TH) sessions
Understand why we may be struggling to reach 200 participants
Expand and lengthen the duration of this particular intervention until we get 200 participants
Review our community mobilization strategy to draw more members to attend the TH sessions
Review current budget to identify budget savings and determine whether we can reach even more people with existing resources
Review our strategy for year 2 to provide follow on support to those who completed the sessions to see how they are coping with trauma
Year 2: Facilitate formation of 20 support groups for individuals who have completed the TH sessions
Review strategy for mobilizing TH participants to join support groups
Establish reasons individuals are not ready to join support groups and identify alternative strategies to provide the support (e.g., one-on-one follow up)
Make budget amendments to identify resources for the extra support groups
Re-evaluate the strategy to prioritize areas in which support groups will be formed in year 2
Source additional funds for support groups in year 3
Year 3: Train 40 community members as TH facilitators
Review strategy for identifying additional community members who can make good TH facilitators
Develop interim strategies to address the shortfall and ensure program results are achieved (e.g., using project staff)
Source additional funds to form additional TH session groups
Pair up TH facilitators to lead the group TH sessions
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 56
Personal narratives, stories, and anecdotes also can provide valuable monitoring
information, including for report writing, adding a human element to complement quantitative data. However, make sure your program is not reactionary based on a small number of accounts. If one program officer constantly complains about a particular problem (or success), it may or may not be worth looking into. However, several program officers reporting the same problem (or success) with or without prompting is even stronger evidence that something needs to be addressed.
Funders often have requirements for monitoring and expect regular reports (e.g., every three or six months), such as budgetary reports and status updates. While they may provide you with a set of metrics on which they want to see reports, most funders will be willing to work with you to come up with metrics that both meet their needs and the project’s needs.
Assumption Monitoring
In Chapter 5, we talked about the importance of collecting information about assumptions in the baseline study to help ensure that programs are appropriately designed from the outset. Regular monitoring of these important assumptions (e.g., about the population, governing regulations) is particularly important in fragile states because of how quickly and unexpectedly changes can occur. Examples of such changes include population migration patterns, leadership (which can dictate changes in laws), food security, economic landscape, and allegiances. Many of the methods previously described for monitoring conflict can be used to monitor assumptions. Table 7.4 uses several examples to illustrate assumptions that could change and how they might affect our interventions. Project staff should be part of the monitoring process and the learning that goes along with it (via, e.g., reports, staff meetings, bulletins) in order to identify the appropriate responses to changes in assumptions.
Table 7.4: Example: Potential project changes based on assumption shifts
If we assume… But this happens… We might have to…
A population with whom we want to work lives in a particular area
The populations has moved out or a new population has moved in
Move project locations
Expand or shrink size of intervention
Change the intervention to meet the needs of the new population
Communities are willing to devote their time to participate in the program
Communities are not willing to devote time to participate in the program
Meet with local leaders to establish why communities are not willing to devote their time to the program.
Identify ways of addressing the issues that are causing the community to stay away from the program
There is political will to support the engagement of local CBOS and NGOs in conflict management and peace building initiatives
Political restrictions hamper the program’s ability to engage with local CBOs and NGOs in conflict management and peacebuilding initiatives
Prioritize building relationships with the local and national government to ensure the program can advocate for the space for local peace actors
Propose to work with those CBOs/NGOs that operate within the government’s legal framework
It is important to note that monitoring can tell us when changes in our assumptions occur, but it does not necessarily tell us if and how to react (as in the “We might have to…” column in the table not being labeled “We should…” or “We must…”).
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 57
Monitoring Plans
In order to appropriately monitor progress against targets, each project should have a monitoring, evaluation, reporting and learning plan. A monitoring plan should include the elements in the table below: type of monitoring (it is a good idea to include all types of monitoring), indictors, decisions to be informed by data, how often to monitor the data point, and how data is collected and by whom. For crafting indicators, please refer to Chapter 6.
Table 7.5: Monitoring plan outline with examples of the three monitoring types discussed previously
Type of monitoring
Monitoring data to collect (indicators)
Decisions to be informed by data
How often to monitor
How data is collected and by whom
Conflict # of cross-border raids reported
What level of security should we maintain?
Should we maintain or postpone any key activities?
Weekly
Monthly
Interns monitoring specific news sources
Meetings with local hospital administrator
Implemen-tation
# of local leaders trained
% of those involved deeming training a “success”
Should trainings continue and at what rate?
Should training curricula be adjusted?
Quarterly Written evaluations completed by participants after training and read by program staff
Written evaluations completed by training facilitators
Assumptions # of people from “the other” community living in the target area
Should we prepare for increased incidences of violence between the two communities?
Should we increase programming efforts related to this violence?
Quarterly Staff conversations with community elders about community participation in peacebuilding initiatives
Summary: Key points from this chapter
Monitoring:
Is an ongoing process, done at regular intervals throughout the life of a project
Is aligned with baseline and evaluation studies, and should be planned accordingly
Tells us what has happened and what is happening, but does NOT tell us what to do about it; Considering “what if” scenarios at the beginning of a project cycle and engaging project staff in monitoring activities will help the learning and decision-making process.
Can involve collecting information in a variety of ways, including quantitative data kept by program staff and administrators, expert and local opinions, survey data, and media content
Helps us look back at what has happened and look forward to what needs to happen, given what has already happened
Is important for tracking progress toward targets: Given where we started and where we hope to end up, has adequate progress been made?
Can be resource intensive, so it should focus on processes and activities that are essential for immediate learning, planning, and reporting needs
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 7: Monitoring | page 58
Learning Activity: Develop a monitoring plan
Write out a specific plan for how you will report on and learn from your project results (what decisions will be informed by the data). It is easy to lose track of monitoring without a detailed plan in place to follow. Determine in advance who will be responsible for reviewing monitoring reports and the protocols for acting on any important findings. Use the template below to prepare your plan.
Type of monitoring
Monitoring data to collect (indicators)
Decisions to be informed by the data
How often to monitor
How data is collected and by whom
Conflict
Implemen-tation
Assumptions
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 59
Chapter 8: Evaluation
Chapter Objectives In this chapter you will learn:
The definition of evaluation and why it is important
Different types of evaluations
Evaluation frameworks and questions
Components of an evaluation Terms of Reference (TOR)
Learning Activities
Create an evaluation plan
What is evaluation and why is it important?40
The previous chapters identified the measures and methods you can use to track data for your organization or project. However, without analysis from an evaluation angle, that monitoring data cannot tell managers why results are or are not being achieved. To get this often-crucial decision-making information, organizations must conduct evaluations to analyze the longer-term, higher-level outcomes or impacts in their programs.
Measuring success in peacebuilding programs requires going beyond just counting activities, such as the number of trauma healing (TH) activities held. Rather, we must understand what outcomes have occurred as a result of conducting TH activities in the project communities. Evaluation can focus on the change we created in the lives and minds of others.
Evaluation is most valuable when a project wants to look not only at results on a cursory level, but seeks to understand the underlying reasons why change is occurring or not occurring in the field, then uses that information to learn and adapt both its actions and its conceptual framework.
In contrast to monitoring, which provides ongoing structured information, evaluation is periodic. Evaluation focuses on why results are or are not being achieved, unintended consequences, or issues of interpretation, relevance, effectiveness, efficiency, impact, or sustainability.
An effective monitoring, evaluation, reporting, and learning (MERL) system includes a program and schedule for evaluation that is based on a learning agenda (see Chapter 9) in combination with an adaptive management approach (committed to changing course and approaches if necessary based on findings from monitoring and evaluation [M&E] data) to ensure the re-examination of management decisions/activities based on new information or learning.
Defining Evaluation
There is no universal definition for the term evaluation. British mathematician and
academician Michael Scriven (1991), one of the founders of evaluation as a field, noted nearly 60 different synonyms, based on such verbs as appraise, assess, critique, examine, grade, inspect, and judge.
40 This chapter is merely an introduction to evaluation. To read about evaluation in much greater detail before planning for an evaluation, please review Pact’s MERL Module 3, Field Guide for Evaluation: How to Develop Effective Terms of Reference, available at http://www.pactworld.org/library/field-guide-evaluation-how-develop-effective-terms-reference. Portions of this chapter are taken directly from Module 3.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 60
As managers and leaders of evaluations, it is important for you to understand how others may understand the term. A common language for evaluation helps us all communicate better.
In this chapter, we will present several common definitions. None is particularly better than another. Instead, each emphasizes a different aspect of evaluation, including its purpose and utility. Understanding the similarities and differences among these definitions will directly help us manage evaluation work in our communities.
According to Michael Quinn Patton (1997, p. 23), a leader in the field of program evaluation, evaluation is “the systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.”
According to the Organization for Economic Co-operation and Development
(OECD; 2002, pp. 21–22), evaluation is “the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.”
According to the U.S. Agency for International Development (USAID; 2011, p. 2),
evaluation is “The systematic collection and analysis of information about the characteristics and outcomes of programs and projects as a basis for judgments, to improve effectiveness, and/or inform decisions about programming.”
In common, all the preceding definitions assert that evaluation is systematic,
that it is specific to a program or project, and that it can answer different
types of questions.
Systematic
First, evaluation is systematic. In other words, evaluation is grounded in a system, method, or plan. To arrive at credible conclusions, a high-quality evaluation uses consistent methods that are clearly outlined in the evaluation design. In this way, evaluation is closely related to research, and both research and evaluation use many of the same tools.
Specific
Next, evaluation is specific to a program or project; this is what distinguishes evaluation from research (see more below). For example, someone might investigate whether children who live near the garbage dump get sick more often than children who live far from the dump. This is research, but it is not evaluation. Another person could study whether children who attend a certain nutrition program get sick less often. Both studies are research, but only the second example is evaluation because it is specific to a project and the first is not.
Versatile
Last, the three definitions also show that evaluation can answer many different types of questions. For instance, an evaluation may ask:
Did the program improve the well-being of community residents?
Were resources used effectively?
What factors were most important to the success of the intervention?
Why did the program fail?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 61
Monitoring versus Evaluation
Sometimes the terms monitoring and evaluation are used synonymously. Although it is important to both monitor a project and evaluate it, the two activities are not the same; Table 8.1 shows some of these differences. Monitoring is primarily intended to provide information about a project’s operations and outputs. Impact or performance evaluations generally look at a project on a broader and higher level, assessing whether it is meeting strategic goals. There is overlap; for example, monitoring data can be used in evaluations and some evaluations are primarily oriented toward assessing operations or processes (often called process evaluations). Consequently, monitoring and evaluation often are linked and are complementary.
Table 8.1: Characteristics of monitoring and evaluation compared41
Characteristic Monitoring Evaluation
Subject Operational management issues addressed
Usually focused on strategic aspects
Character Systematic Subject and methods flexible
Frequency Continuous Periodic
Primary client Program managers Stakeholders and external audience
Party conducting
Internal, but should also be shared with funders, stakeholders, and communities/beneficiaries
Can be external or internal
Approach Utility Objectivity
Transparency
Methodology Rapid appraisal methods
Routine data collection tools
Rigorous research methodologies
Sophisticated tools
Primary focus Focus on operational efficiency, outputs, and outcomes
Focus on relevancy, outcomes, impact, and sustainability
Objectives To identify and resolve implementation problems
To assess progress toward objectives
Learning for adaptations
To check outcomes and impact
To verify developmental hypothesis
To document successes and lessons learned
Evaluation versus Research
Both research and evaluation systematically seek answers to questions; in fact, they use many of the same techniques to answer those questions. However, their purposes sometimes differ (see Table 8.2). Research usually seeks to create new and generalizable knowledge or understanding, while evaluation seeks to inform decisions and judgments about a specific project.
41 Adapted from Jaszczolt et al. 2003.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 62
Table 8.2: Differences between research and evaluation42
Factor Research Evaluation
Purpose To add to knowledge in the field
To develop laws and theories
To make judgments
To provide information for decision-making
Who sets agenda or focus
Researchers
Academic institutions
Stakeholders and evaluator jointly
Generalizability of results
Important to add to theory/contribute to field
Across the program areas, but not necessarily beyond the program
Focus is on particulars of the program or policy and the context
Intended use of results
Usually for publication and knowledge sharing
Usually directly affects the project’s activities or decisions of stakeholders in development of future projects
Why evaluate?
Demanding time and resources, evaluation often competes with resources that are needed to implement programs or deliver services. Many managers ask the question, “Why evaluate?” The Patton, OECD, and USAID definitions, above, suggest clear reasons:
To measure a program’s value or benefits
To improve a program or make it more effective
To better understand a program
To inform decisions about future programs
Other reasons to evaluate include:
To demonstrate to planners, funders, and other decision-makers that program activities have achieved measurable improvements
To understand whether and where resources are being used efficiently and where resources may need to be used differently
To be accountable to funders and community members
To learn about which interventions work and which do not, and why
Knowing why a program is being evaluated is essential to the evaluation’s success. After all, evaluations are meant to be used. How an evaluation is used depends on what questions have been asked, the reasons for evaluating the program, funder requirements, timing, and other factors. Evaluation reports sometimes sit on shelves gathering dust. However, if we are clear about an evaluation’s purpose, if the evaluation is conducted systematically, and if the right questions have been asked about the program during the evaluation, the results should be useful and actionable.
Types of Evaluation
Types of evaluation vary by purpose and program stage. The five main types are:
Formative
42 Fitzpatrick et al. 2011.
Summative Process Outcome Impact
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 63
Most useful during program design and early in the implementation phase, formative
evaluations examine how a program, policy, or project is implemented, whether or not the
program Theory of Change (TOC) corresponds with its actuality, and what immediate consequences the implementation produces.
Summative evaluation is the final assessment at the end of a project. A summative
evaluation determines the extent to which anticipated outcomes were produced. It is intended to provide information about the program’s worth. Results help decide whether to continue or end a program.43
Sometimes called an implementation evaluation, a process evaluation looks at
whether a program has been implemented as intended: whether activities are taking place, whom they reach, who is conducting them, and whether inputs have been sufficient.
Outcome evaluation, also known as performance evaluation, examines a project’s
short-term, intermediate, and long-term outcomes. While process evaluation may examine the number of people receiving services and the quality of those services, outcome evaluation measures the changes that may have resulted in people’s attitudes, beliefs, behaviors, and health outcomes. Outcome evaluation also may study changes in the environment, such as policy and regulatory changes.44 Outcome evaluations usually occur at midterm or endline.
The most rigorous types of evaluation are impact evaluations, which use statistical
methods and comparison groups to attribute change to a particular project or intervention. USAID defines impact evaluations as evaluations that:
Measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual [examination of what would have happened if the intervention did not exist] to control for factors other than the intervention that might account for the observed change.45
The counterfactual is what differentiates the outcome evaluation from an impact evaluation.
Internal and External Evaluation
Whether an evaluation is internal or external depends on who is conducting it. An internal evaluation is conducted primarily by a member of the organization implementing the project. An external evaluation is led by a consultant or other person who does not regularly work for the organization.
Internal evaluations may allow for a more complex, multi-stage evaluation design and can take advantage of in-house staff members’ understanding of the project, either to produce the evaluation more efficiently or to yield more nuanced findings.
External evaluations can be (or can be perceived as) more objective and can bring additional expertise and external perspective that can add value to the evaluation.
Which type of evaluation uses resources the most efficiently depends on an organization’s capacity. Often, evaluation involves both internal staff and external consultants in a joint effort that can leverage the strengths of each.
43 World Bank 2007.
44 U.S. Centers for Disease Control and Prevention (CDC) 1999.
45 USAID 2011.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 64
Involving Stakeholders in Evaluation
It is important to involve stakeholders—representatives of all the people with an interest in the project—in all stages of the evaluation process. Stakeholders fall into three general categories:46
People involved in program operations (e.g., staff, partners, funders)
People served by or affected by the program (e.g., clients, community members, officials)
People who intend to use the evaluation results (e.g., staff, funders, government, general public)
It is important to identify and understand stakeholders, whether they are beneficiaries, project implementers, funders, or another audience. Research has demonstrated the value of the personal factor, and a key component to successful use of evaluations is “the presence of an identifiable individual or group of people who personally care about the evaluation and the finding it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact.”47
Barriers to Evaluation
If evaluation is important, why does it not always happen or happen well? What stands in the way? Key factors are:
Lack of time, knowledge, and skills
Lack of resources for evaluation, including a limited budget
Poor project design, for example, evaluation activities not being integrated into project design
Start-up activities competing with baseline measurement or delaying baseline measurement
Project capacity overwhelmed by complex or overly ambitious evaluation designs
Fear of the consequences of negative findings
The perception of M&E as “police work” or “auditing,” that is, a fault-finding exercise
Arguments by stakeholders that M&E resources would be better spent on program expansion
Difficulty convincing others how useful evaluation will be as a learning exercise
The perception that because no baseline data was collected, it is too late to evaluate
Barriers to program evaluation are worth overcoming so we can learn what works and what does not enables us to better serve the needs of our communities.
Planning for an Evaluation
Evaluation objectives
Determining the components of your evaluation objectives will be the most challenging part of the planning process. It is easy to become overwhelmed by all of the possible “things” that could be measured. It is neither advisable nor feasible to measure everything. Instead, consider prioritizing your objectives. Ask:
What is most important for each of your audiences to learn and why?
What will be done with the knowledge gained?
46 CDC, 1999.
47 Patton, 1997, p. 44.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 65
There are several useful frameworks available to help you choose your priorities and evaluation questions in a systematic way. Framework 1 is more common across a range of different types of development programs; however, Framework 2 is applicable specifically for peacebuilding programs.
Framework 1: Criteria for evaluating development assistance48
The OECD Development Assistance Committee criteria are:
Relevance Effectiveness Efficiency Impact Sustainability
Relevance “The extent to which the aid activity is suited to the priorities and policies of the
target group, recipient and funder.” This criterion is used to determine if (and to what degree) the activity is (still) aligned with ultimate goals and objectives (near- and long-term) and whether those objectives have changed (as the situation has evolved) or remained the same.
Example question: To what extent does the concentration of aid on peacebuilding correspond to the needs in the communities?
Effectiveness “A measure of the extent to which an aid activity attains its objectives.” This
criterion is focused on determining if and why or why not the activity is (or is not) working in a way such that the activity’s goals have been or are on the right track to being achieved.
Example question: To what extent has the aid contributed to reduced incidents of violence?
Efficiency A measure of “the outputs—qualitative and quantitative—in relation to the
inputs.” This criterion examines cost-effectiveness and timeliness (internally and as compared to other approaches).
Example question: Have implementation techniques allowed the same results to be achieved with less budget?
Impact “The positive and negative changes produced by a development intervention,
directly or indirectly, intended or unintended.” This criterion looks deeper into the effects of interventions and considers higher-level outcomes, impacts, and peacebuilding goals. An example question for a peacebuilding program could be: What is the long-term impact of TH initiatives on peace and reconciliation in project areas?
Sustainability Measures “whether the benefits of an activity are likely to continue after
funder funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.” This criterion takes a longer-term view than even impact to look at whether interventions resulted in perpetual long-term changes. This is the most difficult of the criteria in this model to assess.
Example question: To what extent has the aid contributed to durable peace in the communities?
Framework 2: Conflict transformation framework49
The following framework, taken from Church and Rogers (2006), follows the same evaluation criteria as Framework 1, but includes detailed examples of data to evaluate for peacebuilding programs particularly (organized along themes).
48 Framework information and all quotes taken from OECD Development Assistance Committee, n.d. 49 Church & Rogers, 2006, as modified from Church, C., & J. Shouldice. 2002. The Evaluation of Conflict Resolution Interventions: Framing the State of Play. Belfast, U.K.: INCORE at Ulster University. p. 100.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 66
Theme 1: Goals and assumptions This theme explores the justifications for specific
interventions and their methodologies.
Relevance
Attempts to discover whether a given intervention is appropriate for the situation (context and conflict), which has likely changed since initial program design
Assesses whether the intervention is still the appropriate means by which to achieve ultimate peacebuilding goals
Closely related to conflict and context assessment and monitoring
Important in highly volatile environments
Should be pared with outcome identification (Theme 3) to truly understand the intervention’s effectiveness
Is more often included as part of formative evaluations than summative evaluations
Strategic Alignment
Determines whether program activities and intended outcomes are aligned with organizational values and mission
Primarily used to validate internal assumptions about whether a project fits within an organization’s mandate (at a local or international level) or if it has gone off course
Is distinct from looking at an intervention’s alignment with national strategic peacebuilding efforts
Theme 2: Effectiveness and efficiency This theme looks at what was done and the
extent to which it was done efficiently.
Management and administration
Explores communication effectiveness, team dynamics and supervision, logistical effectiveness, and other administrative and managerial processes
Determines how well the program is operating (or was operated) and whether the overall process and individual components were efficient
Useful to consider if resources could have been more-efficiently used if the project was done another way
Cost accountability
Examines the cost-effectiveness of the intervention
Asks if resources were spent wisely (efficiently) and were accounted for appropriately
Practical applications include budget management, funder reporting, and cost projecting
Goal is not simply spending as little money as possible, but the best possible use of resources given the challenges and realities of operating in a time of conflict
Implementation process appraisal
Focused on the quality processes employed in the intervention
Examines how well things are done with respect to known or established standards
Looking in-depth at how activities are carried out is critical for establishing what does and does not work; identification of poor processes can explain poor results
Valuable for projects promoting innovative activities or models
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 67
Theme 3: Range of results This theme measures outputs, outcomes, and impacts of
the intervention(s).
Output identification
Measures near-term, (often) tangible results
Determines what happened and to what degree (how much/many)
Very commonly collected data; useful for meeting funder requirements and monitoring specific, countable targets
Can have reasonable and reliable information earlier in project timeline than outcome or impact assessments
Outcome identification
Assesses the changes brought about by the intervention
Steps beyond “what happened” to “what happened next” and “to whom”
Looks at both positive and negative, intentional and unintentional outcomes
Essential to look both at outcomes related to program objectives and other possible outcomes not initially considered50
Impact assessment51
Evaluates the role of an intervention’s outcomes in the larger conflict
Determines the extent to which the outcomes of activities and processes lead to changes in the overall conflict.
Should include both positive and negative, direct and indirect, intentional and unintentional consequences
Adaptability of change
Explores the extent to which change is both sustainable and adaptable
Asks whether the initial changes seen will be able to withstand and adjust with the changing environment52
Requires an extended program cycle to observe changes and adaptation to new conflict scenarios
Writing an Evaluation Purpose Statement
Deciding on the purpose of the evaluation should begin only after the program logic is clear and stakeholders are engaged. At its most basic, the purpose will be to see whether the program is having the desired results. But, the formal statement of purpose goes one step further; it helps answer the question, “Why do we want to know these answers?”
50 Looking only at anticipated program outcomes is a criticism of many M&E practices. Looking for more open-ended views of change helps ensure that results don’t just include what we looked for or wanted to happen. We can miss out on other outcomes simply by not asking questions outside the scope of our objectives. See Stave, 2011.
51 It is important to reiterate that in this context impact assessment implies looking at long-term, broader-scale changes on the community that result from activities and processes among smaller populations within that community. Identifying impacts does not necessarily prove causality; that is, even if you can prove that a large change (impact) has occurred, you cannot attribute it to a specific intervention component. The systems that influence change are vast and complex, and isolating the factors that lead to change is impossible in some situations. This distinction is important because the term impact assessment is sometimes used to describe an evaluation whose goal is attribution. Such studies are highly controlled, costly, and difficult to implement, even in stable environments.
52 For example, consider a reconciliation program that is successful in reducing people’s view of “the other” and lessens incidences of violence and feelings of animosity between the groups. If a new “other” group migrates to the region, will the same principles be applied and lessen the likeliness that violence and/or animosity will grow with the new group?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 68
There may be a need to justify the program to policymakers or funders by proving that resources are being used efficiently. You may want to improve the program or strengthen the organizations that are a part of it. There are many possible purposes for evaluations.
But, no matter what they are, a clear and well-written purpose statement is important for clarifying the aim of the evaluation so much so that it is often required when planning evaluations and writing grant proposals.
Key questions to be addressed in the purpose statement are:
What will be evaluated?
Why are we conducting the evaluation?
How will the findings from the evaluation be used?
Another way to write the purpose statement is to complete the blanks in the following sentence.
We are conducting an evaluation of [name of program] to find out ________________________________________________ and will use that information in order to __________________.
This purpose statement is an essential part of the evaluation TOR.
Evaluation Questions
All managers have questions about the programs they manage. The following questions are the raw material for creating evaluation questions.
Is the program making a difference?
Is the course of action we’re following the best way to do things?
Are the participants benefiting from the program as expected?
An evaluation question is different than the evaluation purpose (discussed above), but the evaluation questions should help to fulfill the evaluation purpose. For example, if the purpose is to influence policymakers to fund similar programs in other parts of the country, it might be appropriate to ask:
How did the communities that received the program benefit, compared with those that did not?
How cost-effective was the program?
What elements of the program were most important in creating the desired outcomes?
On the other hand, if the evaluation purpose is to show program staff how to improve the program, you might ask:
How do participants of the program perceive it?
What are the program’s strengths and weaknesses?
Why did some program sites perform better than others?
Evaluation questions used to better understand results of peacebuilding efforts
Did we meet or exceed our written objectives? If our stated objective was not reached, why not?
Did we develop mechanisms to receive feedback on our progress throughout the project? Which feedback methods allowed us to alter strategies in a timely manner and measure our impact?
Did the campaign result in positive change in the lives of our beneficiaries?
Are there fewer conflicts in project areas?
Is there more cooperation between communities in conflict?
Have perceptions changed? Are people more tolerant or less prone to partake in conflict?
Did our work have any unintended side effects?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 69
What is evaluation design?
The evaluation design is the plan for answering the key evaluation questions. Evaluation
design is critical to the evaluation process and should begin as soon as program planning begins. The evaluation design process should involve key stakeholders.
The design specifies:
Which people or units will be evaluated
How they will be selected
The kinds of comparisons that should be made
By what approach the comparisons will be made
The evaluation’s timing
The intervals at which groups will be studied
Overview of Evaluation Approaches
Programs can be evaluated using several approaches. There are quantitative
approaches and qualitative approaches and many ways to use them both. These
approaches are often called methods, this manual uses the term approaches, because each actually encompass a variety of specific methodologies. More detail on the various approaches can be found in Chapter 3 of Pact’s MERL Module 3; however, Table 8.3 succinctly summarizes qualitative versus quantitative approaches. Many solid evaluation designs strive to include both qualitative and quantitative approaches, called mixed methods.
Table 8.3: Features of qualitative and quantitative approaches
Approach characteristic Qualitative approaches Quantitative approaches
Aim Identify common themes and patterns in how people think about and interpret something
Classify features, count them, compare them, and construct statistical models that explain what is observed as precisely as possible
Evaluator relationship to results
May only know roughly in advance what he/she is looking for
Decides in advance what he/she is looking for
Data form Words, ideas, pictures, or objects Numbers
Research focus Focuses on fewer selected cases, people, or events
Measures a limited range of responses (forced answer) from more people
Main advantage of approach
Greater depth and detail is possible Facilitates comparison and statistical aggregation for concise conclusions
Analysis methods Categories for analysis must be developed and are specific to the particular evaluation
Uses standardized measures that fit various opinions and experiences into predetermined categories, often using questions that have been verified and tested in other programs or studies
Approach to the program
Can ask questions about the program holistically or about specific parts of a program
Views components of a program separately and uses data from the different pieces to describe the whole
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 70
Creating Evaluation Worksheets
There are two different planning documents that should be considered when planning your evaluation. First, the evaluation worksheet
provides a structured outline used to design your evaluation and describes the evaluation’s objectives, questions, and information you will
collect to answer them and how you will get that information, among other things (Table 8.4). The Terms of Reference (TOR) is another
planning tool, described after Table 8.4.
Table 8.4: Example evaluation worksheet
What needs to be evaluated? (key result area or other key tenets of program)
What evaluation questions should we be asking?
How will we obtain the data? (method) Attach survey instruments as appropriate
When will we obtain the data? (dates) Who will be involved?
Example: Progress made in reducing violence in conflict areas
Outputs (process evaluation)
What have we delivered in terms of short-term change or as an immediate result of our activities?
E.g.: # people involved in TH
Can we claim knowledge or understanding, skills, or capacity change? Were there any changes relating to improved access or quality?
E.g.: Improved level of tolerance toward other community (survey)
What evidence do we have to illustrate these changes actually occurred?
Outcomes (summative evaluation)
What has been achieved as a result of the outputs?
E.g.: Decrease in incidents of violence
Have people changed their behaviors, the way they interact with other communities, and/or their underlying attitudes?
Review of monitoring data
Quarterly Community members
Focus groups with community members
October 2015 Community leaders
Key informant interviews with local officials
October 2015 All key stakeholders and staff
Survey of community members
October to November 2015
Community members (likely a sample of community members)
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 71
The Evaluation Terms of Reference
The second planning tool, the TOR, is a comprehensive, written plan for the evaluation. After completing an evaluation plan, you are ready to create the TOR to either solicit an external evaluator to conduct an external evaluation or for your own purposes if your team is conducting an internal evaluation. Essentially, the TOR describes the scope of work. It assigns roles and responsibilities, activities, and methodologies and is more narrative in form than an evaluation plan.
Developing the TOR yields a shared understanding of the evaluation’s specific purposes, questions, objectives/themes, the design and data collection approach, the resources available, the roles and responsibilities of different evaluation team members, the timelines, and other fundamental aspects of the evaluation. The TOR facilitates clear communication of evaluation plans to people inside and outside of the organization/project.
Importantly, if the evaluation will be external, the TOR helps communicate expectations to and then managing the consultant(s). Because external evaluators are less familiar with the project than the individuals commissioning them, it is important to have a TOR that clearly sets forth all the necessary background, specifically to alert the evaluator to the questions that are most important to stakeholders.
Key components of a TOR are: 1. Overview of the evaluation 2. Brief description of the program 3. Purpose of the evaluation 4. Evaluation questions 5. Evaluation methodology/approach 6. Ethical considerations 7. Implementation information: schedule and logistics, evaluation team 8. Reporting and dissemination plan 9. Application guidelines: budget, timeline
Table 8.5: Components of a Terms of Reference53
Component Description
Overview Summarizes the type and scope of the evaluation, what is being evaluated, and where
Introduces the project to potential project partners, staff, and/or consultants
Program description
or
Organization overview
Summarizes what is being done (the interventions), the projects’ intermediary and long-term objectives (outcomes and impacts), and where and with whom the project is taking place
Describes the scale and timing of the project
Provides an overview of the implementing organization
Purpose of the evaluation:
Goal
Objectives
Questions
Audience
Methods/approach
Distinguishes the evaluation goals and objectives from those of the project
Provides further detail of the evaluation plan, including lines of inquiry (the broad questions we seek to answer) organized by theme, methodologies to be employed, and audiences
Example:
Project goal: to increase peace in community X
53 Church & Rogers, 2006 and OECD Development Assistance Committee, 2008.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 72
Component Description
Project objective 1: to decrease instances of domestic violence in the community
Evaluation goal: to improve the processes and programs offered by our organization to promote peacebuilding
Evaluation objective 1: To determine whether co-ed educational programs are more effective at decreasing domestic violence than single-sex programs
Evaluation approach: Household survey and key informant interviews with community leaders
Ethical considerations Requiring informed consent
Ensuring data is not traceable to individuals (de-identified data sets)
Implementation plan:
Schedule and logistics
Evaluation team
Describes logistical components of the implementation plan
Describes what is needed of the evaluator or evaluation team sought, similar to a job description
Reporting and dissemination plan
Explains requirements for report format
Explains how the evaluator hands the raw data
Specifies how the report will be disseminated and to whom
Application guidelines
Budget guidelines
Timeline
Contact details
Provides information for external consultants and survey firms who wish to apply to offer their services
Explains how to submit applications and by which date
Summary: Key points from this chapter
There is no one universally accepted definition of evaluation, but many of the definitions refer to evaluation as being systematic, specific to a program or project, and able to answer different types of questions.
Evaluations can be designed to take place periodically or continuously alongside implementation. Three types of periodic evaluation are: baseline, midterm, and endline.
You cannot evaluate everything during an evaluation, so it is important to use existing frameworks to identify your priorities in a systematic way.
Evaluations are often outsourced, so it is import to use a template to develop scopes of work or terms of reference with ease.
Learning Activity: Create an evaluation plan
Write an evaluation purpose statement
Reflecting on your organization’s context, write an evaluation purpose statement for your program by filling in the blanks in the following sentence.
We are conducting an evaluation of ____________________ [name of program] in order to find out __________________________________________________, and we will use the information to _____________________________________.
Crafting evaluation worksheets
Take a few minutes to think about your own program, your work on your organization’s evaluation purpose statement, and your organization’s context. Identify key potentially relevant evaluation questions for your program and jot them and all relevant information down on the evaluation worksheet in Table 8.6.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 8: Evaluation | page 73
Table 8.6: Evaluation worksheet
What needs to be evaluated? (key result area or other key tenets of program)
What evaluation questions should we be asking?
How will we obtain the data? (method) Attach survey instruments as appropriate
When will we obtain the data? (dates)
Who will be involved?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 9: Learning | page 74
Chapter 9: Learning
Chapter Objectives In this chapter you will learn:
Some best practices for organization/project learning
The basics of the adaptive management approach
How to develop a learning agenda
Learning Activity
Create a learning agenda
Overview
Learning is an essential component of strong organizations and projects. It directly builds
on monitoring and evaluation (M&E) and uses the findings from M&E activities to strengthen program design and best practices. Learning is most successful when it:
Comes in the form of regular performance data learning reviews, i.e., scheduling and holding periodic learning reviews with the technical team and partners to review performance and discuss any changes in approach that may be necessary
Also occurs post-evaluation and includes action planning (course-correction if necessary)
Involves funders, project beneficiaries, and other stakeholders
Using an Adaptive Management Approach
Adaptive management is an approach to evaluation, learning, and decision-making that
involves a cycle of planning, implementation, monitoring, evaluation, research, and subsequent re-examination of management decisions based on new information that may alter existing plans and priorities.
In its simplest form, adaptive management is action in response to learning.
Adaptive management is ideal for learning about and understanding complex systems and structures because it recognizes that systems are inherently changing and unpredictable.54 Adaptive management copes with the uncertainties by monitoring decision-making results and by re-examining choices in light of these results and based on new information that becomes available.
The ability of adaptive management to provide learning opportunities and foster programming responses to complex systems makes it especially valuable in the peacebuilding field, where contexts remain extremely dynamic. Many managers of peacebuilding programs already engage in this type of assessment and analysis. By further incorporating adaptive management processes, peacebuilding programs can formalize learning to ensure sound programming decisions.
Adaptive management requires intentionally building a series of information feedback loops
into project management during the design phase. This establishes clear pathways for sharing
knowledge and information on the project from early stages of implementation. With
feedback loops engrained into the project from the start, there will be routine opportunities
54 The U.S. Agency for International Development (USAID) has written about the local systems approach in a paper that explains how development programming can best work within existing systems. See Local Systems: A Framework for Supporting Sustained Development, available at https://www.usaid.gov/policy/local-systems-framework.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 9: Learning | page 75
to engage in learning around contextual changes and project outcomes, which will feed
directly into project decision-making.
During the design phase, projects should identify key sources of information and create clear
mechanisms for capturing and learning from this information. Practical examples of
information sources are listed below. When considering sources, projects should keep in
mind that the goal is not to gather more information, but to carefully consider what types of
information is most useful to inform their decisions.
Consultations with key informants
Basic field visit reports
Project monitoring data
Discrete applied political economy analyses (APEAs) and/or conflict analyses
Beneficiary feedback mechanisms (like community scorecards, social audits, etc.)
Close monitoring of news and social media
As part of this process, projects should include consideration for how frequently managers
want to make substantive pivots to the project. The context of the project also should be
considered. A highly volatile context may need more frequent information review, such as on
a weekly basis, to inform resource deployment, while a more stable context might only
necessitate quarterly learning reviews of key environmental and/or programmatic
information.
The adaptive management approach requires that we periodically evaluate activities and revisit our results frameworks, hypotheses, and cause-and-effect linkages (the premises on which an organization selects activities to carry out) to ensure they are still valid based on what we have learned.
Developing a Learning Agenda
Regardless of which type of learning format an organization has planned to use, it initially needs to identify what it wants to learn, how it will learn, and how frequently the learning will be done; these are the foundations of the learning agenda.
Learning agendas are similar to evaluation plans, but differ in some key ways. While questions in the evaluation plan may be similar to those in the learning agenda, the latter enables the program/project/organization to plan for how it will ensure that improvements are made along the way based on learning from immediate processes, experiences, and activity outputs. Also, evaluations tend to be more rigorous in their methodology and analysis because findings are often published for use by stakeholders outside the organization. Learning agenda questions do not necessarily require as much rigor because findings are used internally to inform continuous program improvement.
Developing a learning agenda includes the following steps. Table 9.1 provides an example of a completed learning agenda that includes information for all these steps.
1. Determine the components of your program you want to learn about and identify what needs to be assessed. To do this, review your implementation plan, deliverables, and results framework and identify key components, sub-components, or other aspects of your program that you will analyze in terms of your organization’s efficiency to implement them and/or their effectiveness in obtaining results.
2. Clarify what you want to learn about each component you identified and determine the questions you will answer. Also review the planned deliverables, results frameworks, and indicators for this step. Example questions are:
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 9: Learning | page 76
What has been learned from the project that can contribute to improved program implementation or to building relevant knowledge in the peacebuilding field?
What do we want to know about the subject?
What was changed as a result of our program?
How do targeted stakeholders perceive our programs?
What sort of reach do we have?
How many home visits are we supporting?
How was the target population affected?
How much money did we spend?
For example, if you determine that the training component should be evaluated, you might ask what evidence is available to show that the training implemented has resulted in new ways of doing things or increased participant knowledge and skills.
3. Identify how you will obtain the data. What data do you already have to help analyze this issue, and what data do you need to be able to answer your questions? For example, will you need to facilitate a focus group discussion, hire a research consultant, hold a staff meeting, or use data from specific indicators?
4. Identify who should be involved in answering the questions and in participating in analysis of the answers.
5. Determine deadlines for obtaining the data and conducting the analysis. Do you need the information every month, each quarter, at the end of the project, etc.?
6. Identify how you plan to document the things you have learned, disseminate findings, adapt your program activities, and/or update underlying premises or results frameworks, thus altering the program design.
Ensuring Learning Processes are Participatory
To ensure that a breadth of opinion is captured relating to organizational/program performance and because learning initiatives include important capacity development and growth dimensions, they should be as participatory as possible. Stakeholder involvement in learning promotes a sense of partnership among all the key people and/or groups interested in the organization. A participatory process is essential to provide insight into programs and analysis of how well the needs of different stakeholders are being met. A variety of different perspectives is particularly helpful in analyzing unintended consequences and sustainability of activities.
Summary: Key points from this chapter
Learning is an important aspect for any program that wants to continually improve programming and increase chances of achieving the desired results.
Adaptive management is one method for programs that choose to conduct learning. It involves a cycle of planning, implementation, monitoring, evaluation, research, and subsequent re-examination of management decisions.
Learning Activity: Create a learning agenda
Consider your evaluation objectives from Chapter 8 and identify the types of questions you would like answered during your program implementation. Complete the learning agenda matrix in Table 9.2 (on the next page) to address a couple of these objectives and questions.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 9: Learning | page 77
Table 9.1: Example learning agenda
Which program components do we want to learn more about?
Which questions do we want answered or explored?
What data do we have to help us analyze this question/topic?
What further data will we need to acquire?
Who should be involved in either giving input or analyzing information?
When will we obtain the information or conduct analysis?
When and how will we disseminate the data and adapt our program?
The trauma healing (TH) activities across two communities during our five-year project
Do our TH sessions lead to increased understanding of the relation between trauma and conflict?
What kinds of resolutions do TH participants make that demonstrate willingness to forgive or reconcile with the other community(ies)?
What are we learning about the kind of community members to target for TH sessions?
Pre/post questionnaires
Most Significant Change stories from TH participants
Attendance records for the TH sessions
Interviews and surveys with TH participants and community members during mid and end term evaluations to establish actions taken towards reconciliation
Program staff
TH participants
Other community members
At the beginning and end of each TH session collect pre- and post-test data
3–6 months after completing the TH sessions, collect MSC stories
Midterm
Endline
Pre and post-test training results discussed at quarterly program review meetings
MSC stories analyzed annually and findings discussed at the annual reflection meeting
Midterm and endline assessment completed in years 3 and 5
Following midterm, results framework reviewed for remaining project
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 9: Learning | page 78
Which program components do we want to learn more about?
Which questions do we want answered or explored?
What data do we have to help us analyze this question/topic?
What further data will we need to acquire?
Who should be involved in either giving input or analyzing information?
When will we obtain the information or conduct analysis?
When and how will we disseminate the data and adapt our program?
Peace dividends in corridors between two communities
Are peace dividends benefitting two or more communities?
Are peace dividends contributing to increased interaction among members of previously conflicting communities?
Are certain peace dividends more inclined to increase interaction between communities than others?
Quarterly peace dividend utilization reports
Stories of Most Significant Change resulting from the peace dividend
Context updates from program staff and implementing partners
Observation of how members from different communities are accessing the peace dividend
Interviews and surveys with participants to establish if peace dividends are changing attitudes and behaviors during the mid and end line evaluations. Are communities more willing to reconcile? What reconciliation actions have they taken?
Changing context through periodic context analysis
Program staff
Community members
Quarterly during the quarterly review meetings
6-12 months after completing construction of the peace dividend collect MSC stories from members of the two communities
Midterm and endline evaluations
Peace dividend utilization reports and context updates discussed at quarterly program review meetings
MSC stories analyzed annually and findings discussed at the annual reflection meeting
Midterm and endline evaluation completed in years 3 and 5
Following midterm, results framework reviewed for remaining project
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 9: Learning | page 79
Table 9.2: Learning agenda matrix
Which program components do we want to learn more about or evaluate?
Which questions do we want answered or evaluated/ explored?
What data do we have to help us analyze this question/topic?
What further data will we need to acquire?
Who should be involved in either giving input or analyzing information?
When will we obtain the information or conduct analysis?
When and how will we disseminate the data and adapt our program?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 80
Chapter 10: Reporting
Chapter Objectives In this chapter, you will learn:
What reporting consists of
Why reporting results, milestones, and successes to stakeholders is important
Different types of reports and communication tools typically produced by organizations
Formats for quarterly and final project reports
How to write success stories
Learning Activity
Review progress reports
Overview
Reporting, or communicating your project results, milestones, and successes to stakeholders, is an essential component to ensuring that the information you collect is put to its best possible use. It is important to know who can best use the data and how. You also need to know how to turn the data into useful knowledge that can be used by decision-makers. This chapter will show you how to easily turn your monitoring, evaluation, and learning data into reports and communication tools that can be broadly distributed among a wide range of audiences and that are useful both for your management and your funders.
Introduction to Reporting
A report is a compilation of descriptive information. It is a communication tool to present monitoring, evaluation, and research results by presenting raw data and information as knowledge. A report is an opportunity for project implementers to inform themselves and others (stakeholders, partners, funders, etc.) on the progress, difficulties encountered, successes, and lessons learned during implementation of programs and activities.
Reporting enables assessment of progress against work plans and helps focus audiences on the results of activities, enabling the improvement of subsequent work plans. Reporting helps form the basis for decision-making and learning at the program level. Reporting also communicates how effectively and efficiently the program is meeting its objectives.
Effective reporting and communication plays an important role in peacebuilding by providing accurate and unbiased information to populations affected by conflicts and by proactively delivering programs that aim to impact knowledge, attitudes, and behaviors of the population taking part in peacebuilding efforts. Reporting also can enhance receptivity, a sense of ownership and partnership when shared with community members and stakeholders, and is an effective process for promoting accountability and transparency by project partners.
A good report:
Focuses on results and accomplishments within the context of the project
Assesses performance over the past reporting period, using established indicators, schedules, baselines, and targets
States explicitly whether and how much progress or results surpassed, met, or fell short of expectations and why
Specifies actions to overcome problems and accelerate performance, where necessary
Explains the influence of comparative performance by objectives on the resources needed
Identifies the need to adjust resource allocations, indicators, or targets, where necessary
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 81
Discusses the way forward for programming in light of the findings; annual or final reports also may address prospects of successful program closeout and expected sustainability of results
Identifying Appropriate Communication Tools
Funders often require four types of communication, in addition to regular financial information:
Events and incident reports
Written progress reports (quarterly and annual)
Evaluation reports (baseline, midterm, and endline)
Final performance report
However, reporting should not end with funder requirements. Other types of communication tools you could use include:
Oral presentations/lectures
Discussion sessions/community meetings
Informal contacts and conversations
Press and media releases
Brochures and pamphlets
Formal academic papers and books
Visual presentations (e.g., videos, photos)
Internet, email, and websites
Plays, music, and dances
Use the following steps to determine the most appropriate communication tools for your organization or project’s needs.
1. Identify your audiences’ information needs. For each audience, ask yourself what key information you want to communicate.
2. Determine how you will report to each audience by selecting a tool/format that best suits the information you want to convey. Think about their primary interest in the organization/project to help you decide on the data to report.
3. Review your information database and identify what data you have to address that interest.
Reporting Schedule
Table 10.1 illustrates one way to manage various reports required by funders, internal staff, and other stakeholders. The process for writing, compiling, and submitting reporting, as shown in the table, occurs in order from left to right.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 82
Table 10.1: Example reporting schedule
Type of report Person writing the report Time frame
Information source Reviewers Submitter Submitted to
Incident Partner organization staff
Immediately after conflict occurs
Program staff and beneficiaries
M&E manager and peacebuilding staff
Field staff from the partner organization or primary recipient
Primary recipient
Conflict assessment Sub-grantee/partner organization staff
With each quarterly report
Incident reports, APEA study, and project records
Primary recipient’s peacebuilding staff
Project lead or head from the partner organization
Primary recipient
Partner quarterly Sub-grantee/partner organization
15 days after the end of the quarter
Activity leads and activity records
Head of the partner/sub-grantee and the primary recipient M&E manager
Project lead and/or head of partner organization
Primary recipient
Quarterly programmatic
Primary recipient’s technical staff and M&E manager
30 days after the end of the quarter
Partner reports and project records/database
Chief of party (COP) or deputy chief of party (DCOP) and technical leads
COP or DCOP Funder
Annual programmatic
Primary recipient’s technical staff and M&E manager
Thirty days after the end of the project year
Partner’s/sub-recipient’s quarterly reports and project records/database
COP or DCOP and technical staff, plus headquarters staff
COP or DCOP Funder
Program/project evaluation
Evaluation consultant
Following the baseline, midterm, and final evaluations
Study data and project information
COP and M&E manager
Evaluation consultant to primary recipient and primary recipient to funder
Funder
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 83
Generic Outline for Quarterly and Annual Progress Reports
Quarterly and annual progress reports are the most common and important format for conveying information about a project to its funder(s), partners, and stakeholders. Progress reports should focus on presenting in a concise format the advances (or lack thereof) made on a project during a specific quarter or year. The report should maintain a neutral tone and should discuss any “failings” and “challenges” directly and dispassionately.
Quarterly reports often are laid out in the following manner.
1. Executive summary or highlights
This section describes and summarizes the significant activities and results during the reporting period, either in paragraph form or laid out as bulleted highlights, depending on funder requirements. While the information conveyed here should be addressed more fully elsewhere in the report, the executive summary or highlights may be the only portion of the report many people will read. So, this section, in a maximum of two pages, should briefly describe the:
Program background
Qualitative progress/impact
Quantitative progress/impact
Constraints and opportunities
Program administration
Subsequent quarter’s work plan (omit this section in an annual report)
2. Introduction/program background
This section should include standard language in one to two paragraphs about the project’s objectives, beneficiaries, and funders and what the report includes.
3. Key achievements (qualitative progress)
Provide a narrative description of the key achievements during the reporting period. Sort results by location or other geographic region or by type of project activity, as appropriate. The structure of this section should closely follow the project’s conceptual framework so the reader can easily see the relationship between activities (outputs) and the resultant achievements (outcomes).
4. Key achievements (quantitative progress)
Provide a quantitative description of the key achievements of the reporting period, and complete the indicators table provided in the reporting template with the basic data required to assess progress toward achievement of targets under each development objective.
5. Other performance monitoring updates
In a maximum of one page, describe other performance monitoring activities conducted in the reporting period not explained in above sections. If data quality issues were identified, please explain any action(s) proposed or taken to address any/all issues. Where possible, please include additional performance or learning information that was collected during routine monitoring activities (e.g., site visits, pre- and post-tests). Raw data collection materials (e.g., site visit forms) may be included as attachments.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 84
6. Constraints, opportunities, and lessons learned
In a maximum of one page, address any significant operational challenges, including changes in scheduled or planned implementation tasks and/or issues related to coordinating tasks. Discuss changes in activities that may be required due to practical considerations “on the ground,” including any changes in scheduled or planned implementation tasks. If activity resources need realignment or new “windows of opportunity” have opened, discuss these as options. This discussion should be limited to statements of facts and conclusions that might be drawn from the facts. If issues from the previous reporting period remain significant, be frank and open in discussing any step(s) toward their resolution.
This section also can include agreements between you and your funder to take specific actions. If decisions are made (in consultation with the funder) to either change the geographic location or the strategic elements being emphasized in the activity, they should be noted here for the official record. If contract modification or amendments are needed, they can be described in the administrative review section.
7. Administrative review
In a maximum of one page, discuss the status of your program administration. During the reporting period, were there any changes in staffing/management, institutional strengthening plan actions, contract modifications or amendments to the program?
8. Progress on funder priorities
In a maximum of a half page, identify any contributions to funder priorities, such as Value for Money,55 localization of foreign aid,56 environmental impact or gender. It is appropriate to discuss how the project’s focus reflects the signature campaigns of or global initiatives for project funders.
Examples of funder priorities could include:
a. Progress on counter-terrorism and violent extremism
In a maximum of a half page, provide an update on the national or local government’s terrorism-prevention programs, while looking at effectiveness of nongovernmental, community-based initiatives and the program being implemented in countering violent extremism in the conflict areas.
b. Progress on climate change and natural resource management
Many rural populations are heavily reliant on natural resources for their livelihoods. In a paragraph, describe how violent conflict has influenced vulnerability to climate change and natural resource impacts during the reporting period for people living in affected areas. In addition, discuss stress that climate change is having on these resources and how conflict-related barriers to accessing pasture and water for livestock may increase negative coping mechanisms in response to climate-related shocks and stresses. Further, and as applicable, show how violent conflict erodes the possibility of collaborative natural resource management, which is crucial for adaptive capacity to climate change.
55 Value for Money is a DFID requirement. Guidelines are available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67479/DFID-approach-value-money.pdf.
56 Localization of foreign aid is a USAID requirement. See https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 85
9. Following quarter’s work plan
In a maximum of one page, expand on the amount of detail from the annual work plan for the subsequent quarter, either by listing action items or offering a three-month table, calendar, or timeline describing the completion of an activity, such as in Table 10.2. You may omit this step in an annual report or final performance report.
Table 10.2: Example projected work plan for the following quarter
Activity
January to March 2015
January February March
Program start-up and cross-cutting issues
Cross-cutting and start-up activities
Result 1.1: Communities more open to social reconciliation
Activity 1.1.1: Trauma healing
Activity 1.1.2: Peace dividends
Activity 1.1.3: Cultural adaptation
Result 1.2: Communities peacebuilding capacities mobilized
Activity 1.2.1: Promote local peacebuilding leadership
Activity 1.2.2: Promote youth leadership and engagement opportunities
Activity 1.2.3: Expand the impact of peacebuilding organizations
Activity 1.2.4: Establish/strengthen peace networks
Result 1.3: Local governments partner with their cross-border counterparts and communities in conflict management
Activity 1.3.1: Promote emergence of local-government peacebuilding leadership
Activity 1.3.2: Expand impact of local-government peacebuilding initiatives
Activity1.3.3: Promote partnership between local government and peace networks
Generic Outline for Final Performance Report
Funders usually require that recipients submit a final performance report at the end of the funding cycle. Final reports often are laid out in the following manner.
1. Executive summary
This section captures the essence of the report and provides an overview of its contents. It is the last section to be written and does not exceed two pages.
2. Introduction
This section presents a very concise overview of the need for and history of this program in a couple paragraphs. It describes the results, objectives, context, and activities that were anticipated under the program during the period of agreement.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 86
3. Methods and approach
This section briefly describes, in at most a page, the methods and approaches used to implement the program.
4. Results/impact
This section compares planned versus actual achievements. In this section, you should:
Summarize program accomplishments or failings
Present findings as to why progress toward planned results was unexpectedly positive or negative
Present findings on how well needs of different customers were met (e.g., by gender, age, ethnic group)
Present indicator results/tables and anecdotal information to support findings
Assess the value of the program’s contribution and clarify exactly how the achievement of your objectives contributed to the development outcome and impact
Review the validity of hypotheses and assumptions underlying the results framework based on lessons learned in implementation
Describe mitigating factors that disrupted what was planned and the organization’s response to the disruption
Describe the facilitating factors that helped spur results
Identify and analyze unintended consequences and effects of assistance activities
5. A summary of the prospects for sustainability
In a page or less, describe the progress made in meeting the sustainability objectives of the program, including the approaches used to build financial sustainability (e.g., local financing, cost recovery, resource diversification, corporate sponsorships). Present any cost leveraging achieved during the project. Identify what the beneficiary communities say about sustaining project services through alternative funding sources at project close.
6. Review of deliverables
Review the deliverables submitted, preferably in table format for easier reading. Explain any unfinished work and recommend whether and how it should be completed.
7. Lessons learned/best practices/recommendations for future programs
Distill lessons learned or best practices identified through program implementation that may be useful elsewhere and/or by others. Recommend how program activities and outputs could continue or what new programs should/could be developed.
Appendix 1. Fiscal report
Describe the budget in detail, including closeout matters, cost share results, and how any matching funds were used.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 87
General Writing Tips
Use active voice, for example: - Passive: Wells were tested by 80 percent of the participants. - Active: Eighty percent of the participants tested their wells.
Use short, complete sentences.
Write in plain English, using simple words and avoiding jargon.
Be concise.
Avoid abbreviations/acronyms.
Use your name, program name, or county name rather than “this agent,” “I,” “our,” or “we.”
Write in paragraph style, with a topic sentence.
Success Stories and Photography
Success stories are human-interest pieces that illustrate project outcomes and impact by
detailing an individual’s or community members’ positive experiences as a result of the program. They include the when, what, where, how, and why of projects outcomes, similar to a newspaper article. Appendix 1 presents some general guidelines and tips on how to write success stories and take photographs, with links to specific examples of success stories from some key funders.
Success stories are written to:
Show accountability for public funds and/or funder funds
Share successes so individuals inside and outside the program can learn from our work and results
Illustrate program outcomes and impact
Demonstrate the “human feel” that numbers often don’t show
Success stories are used:
To assess team progress in relation to planned activities
To share information among existing partners and generate interest among potential partners
To help partners and the general public better understand the program’s work and value
For funder reporting purposes, including for funders to report to their stakeholders and superiors
To post on program-area websites
As a source of news stories for local media
To document activities and accomplishments of program and staff
Photographs are images that convey events or ideas. Photographs are an important
component of success stories and help bring them to life.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 10: Reporting | page 88
Summary: Key points from this chapter
Reporting entails compilation and communicating useful data and information relating to the program, including for use in decision-making.
A good report is able to convey required information to the person who needs it when he/she needs it.
The most important program reporting tools include quarterly and annual progress reports, final performance reports, events and incident report forms (both written and phone submissions).
Main sections of the quarterly and annual progress reports are the executive summary/highlights, introduction/background, key achievements (qualitative progress/impact), key achievements (quantitative progress/impact), constraints/opportunities/lessons learned, progress on any special funder priority areas, and subsequent quarter’s work plan (omitted in annual progress reports).
It is important to observe reporting schedules to ensure information is conveyed to those who need it on time.
Success stories with photographs helps communicate change in ways that conventional reports do not.
Learning Activity: Review progress reports
Review past and recent quarterly and annual progress reports submitted by your program based on the guidance provided in this chapter. Share with the group some of the obvious mistakes seen in the report and discuss how best to fix these mistakes.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 11: Data Quality and Ethics | page 89
Chapter 11: Data Quality and Ethics
Chapter Objectives In this chapter, you will learn:
Key data quality management concepts and assessment criteria
Steps for addressing common anticipated data quality issues during routine data management processes
Factors or tips to remember when creating a good data management system
Learning Activities
Review key data quality issues and indicator plans
The Importance of Ensuring Quality Data
Data quality refers to the accuracy and precision of the information collected against
monitoring and evaluation (M&E) standards. It focuses on ensuring that the process of data capture, verification, and analysis is of a standard that meets the requirements of an internal or external data quality assessment (DQA) or audit.
We can select the best indicators and write the best protocols, but if tools are not properly used and if standards of reporting set by the organization are not respected in terms both of quality and timeliness, data could be at risk of poor value and/or could be over/under/or miscounted.
Commonly used Criteria for Assessing Data Quality
There are five key criteria commonly used to assess data quality: 1. Validity 2. Reliability 3. Timeliness 4. Precision 5. Integrity
Issues and risks relating to data quality need to be thought through and documented to ensure quality standards are developed and maintained. Thus each organization or project needs to develop and document how it checks the following.
1. If the data adequately represent performance and are thus valid
Specifically, we need to check if there are any issues with:
Face validity: Is there a solid, logical relation between the activity or program and what is being measured, or are there significant uncontrollable factors?
Data transcription: Are steps being taken to limit transcription error (e.g., double keying of data for large surveys, electronic edit checking program to clean data, random checks of data entered by supervisors)?
Sampling methodology: If we report on data where we use a sample (rather than census- or case study-derived data), we need to be sure there are no significant measurement or representation errors (i.e., our samples are representative, the survey questions are clear, response rates were sufficient, the population was appropriate, sampling methods accurate). We also need to be sure that anyone who is collecting data is appropriately trained and that we account for potential bias on the data collector’s behalf.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 11: Data Quality and Ethics | page 90
2. If the data collection processes are stable and consistent over time
and are thus reliable
Are data collection procedures consistent (e.g., from one reporting period to the other, from location to location, is the same instrument being used)? Are we checking the data to ensure it is correct and free from errors? Are data problems reported? Would we get the same results if someone else went out again to collect the same data in the same way?
3. If the data is collected frequently enough and is current, and thus timely
Is data coming in on a frequently enough to inform program management decisions? For example, are you receiving training feedback data before planning the next round of training (so as to use learning from those training to shape the upcoming training?) Do people report when they are supposed to? Data that is required today but not received until tomorrow might not still be helpful.
4. If the data have an acceptable margin of error and are thus precise
Is the margin of error less than the expected change being measured?
5. If there is data integrity
Are data free from bias or manipulation? Are there procedures in place to reduce the possibility that data are manipulated for political or personal reasons? Does an independent review, assessment, or audit of the data take place?
Data Management
Data management process involves collecting, collating, analyzing, reporting, using, and storing project activity information. There can be no monitoring, evaluation, reporting, and learning (MERL) without a good data management system, with the components described below.
1. Data source
This is where the data originates, such as:
Training sign in sheets
Village Elders records
District peace committee meeting minutes
Community peace agreement records
Interview transcripts
Questionnaire responses
2. Data collection
Data collection is the process of gathering data generated from the various activities implemented by an organization and relevant to an organization’s MERL framework. Data collection involves obtaining data from original sources and transferring it into tools (paper or electronic) from which it can be analyzed or transferred to another data analysis system for analysis and reporting. Many times, data quality is compromised at this stage; therefore, you should exercise caution during data entry—garbage in, garbage out! See Chapter 12 for tips on collecting data through electronic means.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 11: Data Quality and Ethics | page 91
3. Data collation
Data collation refers to aggregation of data from different sources, such as different program sites or different field workers who collect the data, into summarized formats. Collation can be done electronically using MS Excel spreadsheets or databases or manually using paper-based systems. For example, you likely want to sum up female and male participants of different dialogue sessions across regional sites/partners in the quarter to get the total dialogues conducted and total female and male participants who attended peace dialogues implemented by the organization in the quarter.
4. Data analysis
Data analysis involves reviewing and manipulating data to assess progress made toward desired objectives and targets. Analysis enables data users to:
Associate variables (test underlying theories or assumptions)
Predict relationships (cause and effect/outcomes)
Indicate confidence in results
5. Reporting
Reporting entails compiling descriptive data and presenting raw data and information generated from data analysis as useful knowledge. Reporting provides an opportunity for project implementers and others to learn about the progress, problems, difficulties encountered, successes, and lessons from implementing program/project activities. Remember, if it is not reported it never happened because others do not know about it. See Chapter 10 for more on reporting.
6. Data use
Data use refers to making timely data-driven decisions that relate to program/project implementation. Information need to be available when required so it serves the purpose for which it was collected, then is used for learning.
7. Data storage
Data storage entails securely keeping and maintaining information or data collected for current and future reference. Data storage is done best when both electronic and hard copy (original data source documents) are well organized and kept safe.
Getting Started with Developing Systems to Address Data
Quality Issues
The indicator plan and/or protocol provides a starting point for identifying potential issues that may affect data quality.
To get it right from the start, it is absolutely essential that all indicators are defined upfront to avoid potential problems with data validity or reliability that could arise from indicator definitions being unclear. Indicator protocols allow programs to address common data validity and reliability problems by providing standard definitions of program indicators and related units of measure.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 11: Data Quality and Ethics | page 92
Staff responsible for MERL must ensure common indicator understanding and interpretation by program team members who would be involved in collecting or analyzing indicator data coming through the system at any given time. This could best be done through a team meeting or partners/stakeholders MERL training. It is also important for the program manager(s) to consider and determine the likely risks to managing the individual indicator data right from source to data use.
The “data quality issues” section of the standard USAID performance indicator reference sheet (PIRS)57 allows programs to document all key data-management processes and the anticipated data quality risks that may affect data use. This also allows the program MERL and/or program managers to pay attention to key data quality considerations, such as timeliness of the data, which is addressed by detailing the frequency and timing of data acquisitions and analysis. Data integrity also is addressed to some extent by the provision on the indicator protocol to assign responsibility for data storage along with protocols and systems necessary to ensure that data remains sure and uncorrupted. However, additional issues, such as policies addressing data ethics within the organization, are still required to ensure that data integrity is addressed comprehensively.
Improving Data Quality along the Data-Management Chain
To help ensure data quality at the source:
Design simple-to-use instruments and correctly implement them
Include data providers (community stakeholders) and data processors in decisions about feasible review processes and drafting of instruments
Ensure all personnel are trained in their assigned task; use the same trainer for all trainings, if possible
Develop and document instructions for the data collectors and on the collection forms, as well as computer or gadget for data-collection procedures
Develop an appropriate sample
Ensure all data collection, entry, and analysis needs are available (e.g., pens, paper, forms, computers, phones, chargers)
When using technology, make sure to have simple scripts and forms that users will find easy to understand
To help ensure data quality during collection:
Develop specific instructions for data collection
Routinely check to see if instructions are being followed
Identify what to do if you (or someone) wants to make a change to the data collection process or if you have problems during data collection (change management process)
Check to see if people follow the change management process
Train data collectors on how to collect information
57USAID Performance Indicator Reference Sheet (PIRS): https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf
10 tips for developing a good data management system
1. Keep it simple 2. Make the interface attractive 3. Remember only techies read the users’
guide! 4. Craft well-defined indicators 5. Consider user needs 6. Ensure compatibility of systems 7. Ensure quality of reports generated 8. Assign a dedicated database
administrator 9. Watch out for accessibility issues
10. Conduct Back up and maintenance
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 11: Data Quality and Ethics | page 93
Develop standard operating procedures (SOPs) for managing the collected data (moving data from one point to the next)
Develop SOPs for revising the collection tool
Communicate the process and establish processes for supportive supervision of data collectors
Conduct on-site reviews during the data collection process
To help ensure data quality during collation:
Develop checklists and sign off on key steps
Conduct reviews during the data entry process
Create an electronic or manual format that includes a data verification process by a second individual who is not entering the data
Randomly sample data and verify
Ensure problems are reported, documented, corrected, communicated, and tracked back to the source of the problem
To help ensure data quality during analysis:
Run audit report for review by experts with knowledge for reasonableness
Ensure analysis techniques meet the requirements for proper use
Disclose all conditions/assumptions affecting data interpretations
Ensure that appropriate analysis that is management useful to program managers is undertaken routinely
Document changes made to the dataset, including creation of new variables and how they were calculated
To help ensure data quality during reporting:
Make sure you have all the data required to back up your report
Synthesize results for the appropriate audience
Maintain integrity in reporting: don’t leave out key information
Engage multiple report reviewers within the organization prior to dissemination
Protect confidentiality in reports/communication tools
Review data and provide feedback with stakeholders
Provide appropriate and easy-to-understand reports based on your audiences’ information needs
To help ensure data quality during use:
Consistently promote use of the data
When possible, provide data in terms of spatial or graphic representation
Routinely make available data at key decision points (e.g., staff meetings, program review)
Understand the data’s strengths and weaknesses and incorporate results appropriately
Build in processes to review how data has been used for decision-making over time
Take corrective action to enhance data use where necessary
To help ensure data quality during storage:
Take note of cumulative data and carefully track and store using a secure, accessible, and well-organized database
Remember to periodically back up incoming data, if possible in different locations; web-based storage systems are preferred
If using raw data, remember to keep it with metadata information for easy retrieval for future use
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 11: Data Quality and Ethics | page 94
Hard copy or primary information sources need be well filed for future reference and possible DQAs
Stored data access needs to be available only to authorized team members as per the data management protocol; this ensures data remain safely secured
Consider implementing basic system maintenance tools like “restore” and “check data integrity” commands
Data Ethics
Data ethics refers to the rules or standards governing the conduct of a person collecting,
collating, reporting, or using data. Common ethical considerations that programs should make in relation to data include the following.
Ensure that program participants/beneficiaries are provided with sufficient information to enable them to make informed decisions about their participation in data collection efforts. No participant/beneficiary should be coerced to give information.
Participants need to be made aware of how much privacy/confidentiality/anonymity they can expect for their responses. This is especially important for peacebuilding programs where safety is of the utmost concern.
Programs should take steps to ensure that data is not misrepresented or falsified by anyone involved in the data-management process.
Summary: Key points from this chapter
Accuracy and precision are important attributes to ensure data is of good quality.
The five commonly used criteria for assessing data quality are validity, reliability, timeliness, precision, and integrity.
Data-management processes involve collecting, collating, analyzing, reporting, using, and storing project activity information.
Indicator plans and protocols are the most important tools to start with when assessing data quality.
The team leader needs to ensure that indicators are clearly defined to ensure that everyone involved in data collection and reporting have the same and the correct understanding of all indicators and data collection methods.
A good data-management system should be simple with attractive interphase, user friendly, compatible, accessible, able to produce quality reports, and easy to maintain.
Data ethics must be observed by all involved with data along the data-management chain.
Learning Activity: Review key data quality issues and indicator plans
Divide participants into groups and ask them to discuss and identify key data quality issues likely to affect their indicator data reporting.
Review partner indicator plans to ensure they include measures for addressing any data quality issues moving forward and have strong indicator definitions.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 95
Chapter 12: Mobile Technology for Data Collection
Chapter Objectives In this chapter, you will learn:
About short message service (SMS)-based data collection versus general packet radio service (GPRS) data collection
The benefits of using mobile technology
How to develop a mobile technology plan
When to use and not use mobile technology
How to capture GPS coordinates
Learning Activities
Brainstorming on mobile technology for data collection
Evaluating experience with mobile data collection
Overview
Pact’s MERL Module 4: Mobile Technology Handbook58 covers in great detail the use of mobile technology for data collection. This chapter is an overview of the different types of mobile data collection platforms available and considerations to make when deciding how to use mobile technology in a peacebuilding project. Much of this chapter is drawn directly from MERL Module 4.
Mobile technology already has proven itself a powerful and efficient tool that accelerates
achievement of project objectives and, ultimately, of development goals. Efficiency and
data quality gains have been accepted as the norm for many applications, and the
frontier of possibilities expands daily.
What is information communication technology for
development?
Information communication technology for development (ICT4D) is the application of common, everyday technologies to the development field. This chapter focuses on mobile technology (specifically, cellphones and tablets) and how they are used around the world to accelerate development program goals. Projects from diverse technical areas have adopted mobile technology to perform:
Survey data collection
Routine monitoring
Dissemination of general or targeted information
Awareness and sensitization campaigns
Beneficiary/patient tracking
Surveillance and reporting
Crowdsourcing
58 Pact. 2013. MERL Module 4: Mobile Technology Handbook. Available at http://www.pactworld.org/blog/pact%E2%80%99s-mobile-technology-handbook-debuts
Logistics management information systems
Mobile banking and cash transfers
Coordinating service delivery or referral services
Conflict early warning systems
Public dialogue
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 96
Advantages of Mobile Data Technology
The advantages of using mobile data technology over traditional, paper-based methods include the following.
Faster, more accessible data: Data collected in the field with mobile devices can be submitted and stored online (in the cloud) or on central office servers, allowing for real-time access to data for analysis and decision-making.
More reliable data: Collecting data in the field with mobile devices eliminates the need to enter data from paper into a database, thereby reducing the opportunity for error or loss.
Cleaner data: Mobile applications embed skip logic that directs respondents to specific sections of the questionnaire based on previous responses (e.g., different questions for males and females). Thus, if skip patterns are correct, data are never entered where they don’t belong.
Ability to monitor data collection: By monitoring submitted survey GPS locations and time stamps you can track field worker performance and quality of incoming data.
Richer data: Integrated GPS coordinate collection, camera capabilities, and audio capture broaden the scope and type of information that can be collected.
Built-in data visualization: In some systems, online interfaces permit data visualization or have dashboards that allow data to be manipulated, customized, summarized, or downloaded.
More than just data: Mobile devices support a range of capabilities beyond data collection (e.g., mobile banking, patient tracking, crowdsourcing, communication).
Mobile data collection in conflict/post-conflict environments
When working in a fragile, conflict location, data collectors collecting data on paper have to either arrange for transportation of the data to a centralized location for aggregation and analysis or ask for data entry to be done then emailed to the central project location. Removing this step reduces risks for, e.g., data loss, longer data entry time, reduced safety of the personnel needed to transport the data.
The real-time nature of mobile data collection is beneficial for peacebuilding programs. Because these programs operate in environments with ever-changing contexts, having real-time data allows programs to make decisions and reorient strategy if necessary.
A MERL officer with the Pact PEACE III project touts the benefit of using mobile data collection. “We benefit a lot from the real-time nature of mobile data collection as this helps us to track partner activities in real-time and hence be able to make decisions faster. Also real-time incident reporting helps us quickly obtain information on trouble hotspots allowing us to make quick decisions. The mobile data platform also includes visualization feature that offers different views of the data such as maps and graphs allowing us to quickly get some insight of the data in real-time as this gets reported.”
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 97
SMS versus GPRS
Data can be transmitted in two main ways using mobile technology: SMS and GPRS.
About using SMS (basic cell phones)
SMS is a system for sending short messages of a fixed length, traditionally a maximum of 160 characters, in English, with other lengths in other languages. Select SMS for your data collection if:
You are using basic phones (e.g., if your initiative is using the personal phones of data collectors who do not own more technologically advanced mobile devices, specifically Java enabled feature phones or smartphones
Data submissions are typically 160 characters or fewer in length
Total data volume is low
Data collectors lack internet access
Your project is a small one-off initiative that will not be scaled up
Data collectors need to send their messages at no cost, that is, if you need the ability to reverse-bill SMS charges
About using GPRS (internet)
GPRS is a data-transmission system similar to SMS, but without limits on the number of characters or transmission size. This is more commonly known simply as “the internet” on phones. Select GPRS for your data collection if:
Data collectors are equipped with more technologically advanced phones (e.g., Java-enabled feature phones or smartphones)
You need to submit large volumes of data at one time
Total data volume is high
Multimedia data (e.g., photos, videos) will be transmitted
Internet access on the phone or via intermittent connection to Wi-Fi is not an issue for data collectors
Example: Using SMS in a peacebuilding project
Setting up an SMS-short code can be an effective way of receiving conflict monitoring or other such data best gathered from community crowdsourcing. If community members are made aware of the code, they may text the number about incidents they see, often at no expense to them. There are ways you can ensure that your project pays for the SMS messages that are sent in, rather than making the respondents pay for that cost out of their own phone credits.
A word of caution: If you work on a project that crosses country borders, it may be difficult to set up an SMS system. This is because you would have to set up short codes with various service providers in the different countries and would not receive the data in one central location. In addition, using SMS is only appropriate when you have few data items to collect.
Taking GPS Locations
The GPS receiver on a mobile phone can collect location data even when there is no mobile data connection. However, getting an accurate GPS location requires clear view of the sky and good satellite signal reception. Avoid collecting GPS locations when inside buildings or when standing next to large obstructions. If you do not have a clear view of the sky, your location will not be recorded accurately.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 98
Figure 12.1: Screenshot of PEACE III event form on iFormBuilder (at right)
Developing Your Mobile Technology
Strategy and Why it is Important
Developing a strategy for incorporating mobile technology into your work processes is critical. The strategy will ensure that the human, material, and financial resources you will need are available; that your technology is appropriate for the project and its context; and that mobile technology will enhance program management or implementation.
Example: Using GPRS (internet) in a peacebuilding project
Pact’s PEACE III project collects data via the mobile platform iFormBuilder (see Figure 12.1). The project uses mobile data for event and conflict incident reporting. Using mobile technology for data collection has increased the speed in which data is received from each event, removed the burdensome step of data entry, allowed for collecting cleaner data through complex validation controls, and enabled the project to collect richer data, including GPS coordinates and images. The GPS coordinates have enabled the project to look at trends in the data from different perspectives using geographical information systems (GISs).
Many GPRS mobile platforms, such as iFormBuilder and others, provide an offline data collection mode. This means that data collectors in remote areas with no internet or Global System for Mobile Communications (GSM) connectivity can collect data, store it in the phones’ memory, then upload/“send” the data later when they get to an area with Wi-Fi or GPRS. This is especially important for areas without any or with poor mobile service coverage.
A word of caution: Sometimes you may need to change platforms mid-project if it does not suit your needs. For example, the type of form you use or data you collect may be better suited for one mobile platform over another. Free trial periods (before purchasing software) to test forms are ideal to allow you to extensively test all the software features before purchasing. Also, while most mobile platforms allow offline data collection, it is crucial that users are well trained on how to go about this to minimize data loss.
In hostile places, using a mobile phone could endanger the life of the data collector. For example, using the mobile platform to take pictures in events where military and/or militia members attend might accelerate tensions because data collectors may be viewed as spies or worse. Extreme caution must be taken in these circumstances. In instances where security may be a concern, use discretion while collecting data in unsafe locations and delay collecting data if the situation is too insecure.
Mobile phones also have their limitations. Smartphones regularly run out of power, and storage space is limited. For week-long activities in areas where there are few charging sockets, it is important to plan ahead and have backup power (e.g., extra batteries, solar charging stations) or to switch off the phone when it is not in use to conserve power. In the early stages of the program, PEACE III experienced this challenge, when a data collector gave the feedback that “the phone went off on day one.” Another partner was unable to use the mobile app because the phone’s internal memory was full, so the app could not launch.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 99
When developing a strategy, it is important to involve all stakeholders, including those who will collect the data, use or analyze data, and manage the process. Chapter 2 of Pact’s MERL Module 4 discusses how to get started and what questions you should be asking stakeholders as you develop the strategic plan.
Use the following steps to help you begin to develop your organization or project’s mobile technology strategy.
1. Brainstorm the areas where mobile technology can improve your project, keeping in mind the sensitivity of the context and logistical limitations in which it operates.
2. Identify the data you would like to collect. 3. Identify and understand your mobile technology users. 4. Consider data-management processes. 5. Conduct a mobile technology feasibility scan. 6. Decide on the appropriate mobile technology platform. 7. Develop your mobile technology strategy document.
1. Brainstorm how mobile data collection can improve your project.
It is important to consider first how mobile technology will be useful for your project’s MERL system. Think about some problems you have and how mobile technology can solve them. For example:
Slow data processing: “Whenever we conduct surveys, data clerks are tasked with entering all the data. Given that this is in addition to their other work, it takes up to four months to obtain a completed dataset. This delays our ability to respond and make databased programming decisions.”
Poor data quality: “Too many times, our data collectors don’t follow skip patterns or questionnaire instructions. As a result, our data is poor.”
Lack of geographic data: “USAID has asked us to map services delivered to beneficiaries. We need to begin to collect GPS data.”
Learning Activity: Brainstorming on mobile technology for data collection
Referencing the brainstorming examples below, what type of organization or project data could be most efficiently collected using mobile technology? How might you integrate mobile technology into your current data collection activities?
Pact Myanmar Brainstorming
Goal: Pact Myanmar wanted to begin using mobile technologies to monitor and track women’s savings groups outcomes.
Problem: Every six months, data are collected from women’s savings groups on savings, loans, and dividends—a huge endeavor, given that there are more than 2,000 women’s savings groups in Myanmar.
Solution: Mobile technology made data collection much more efficient.
PEACE III Brainstorming
Goal: PEACE III partners with local implementers working in the cross-border locations to bring peace in these neighboring communities through trauma healing, cultural practices adaptation, and peace dividends.
Problem: To achieve its objectives, PEACE III needed to track partner activities and conflict incidences in real time. The program also wanted to collect GPS coordinates to enable mapping of partner activities and reported incidences.
Solution: Mobile technology enabled collection and reporting of real-time data and GPS coordinates.
(Learning activity continued on next page)
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 100
Goal:
Problem:
Solution:
2. Identify your data.
Based on your brainstorming, think about the data you would like to collect.
Indicators: If you are starting a new project, consider which indicators make the most sense to collect using mobile technology.
Tools: If your project is already underway, think about which data collection tool(s) could most appropriately be adapted for mobile data collection.
New information: If you could access data for analysis much more quickly after collection, as is possible with mobile technology, consider whether there are data that would be useful to have that you may not otherwise have collected (e.g., conflict data).
New data types: Remember that with mobile technology you can collect not only text and numerical information, but also photos, videos, GPS coordinates, audio, and biometric data.
Remember: Mobile technology should simplify your life. For example, think of the large volume of data that are waiting to be entered or data that people have to travel far to submit paper survey forms.
3. Identify and understand users.
As part of the process of deciding whether mobile technology will enhance your data collection and management processes, you need to identify and understand the end users of your mobile technology. Knowing your users will help you:
Obtain the right type and number of mobile devices and appropriate software: How many mobile devices are needed? What language will the software need to use? Under what conditions will the devices be used (e.g., access to power sources, rain or dust that might damage the devices)? What level of technology is appropriate for the end users? Also, consider their ages, experience with phones, literacy, and eyesight.
Tailor training to the users: The most effective training content and delivery is responsive to end users’ level of education, culture, and previous exposure to similar technology.
Get buy-in from the people who will be using the technology: Securing end users’ commitment to mobile technology during the design phase is the best way to
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 101
ensure the success of your rollout. Exposing end users to the technology early on will test any assumptions you have made about the appropriateness of mobile technology. Early user involvement also has the potential to greatly improve the level of participation and, ultimately, the speed and quality of implementation. It also is highly important to ensure gender equity when introducing technology. Obtaining buy-in from both male and female users is crucial. Additionally, sensitization to the technology may require different techniques with men and women, depending on existing usage levels.
4. Consider data-management processes.
Once you identify the relevant questions, data collection tools, and data types for your mobile initiative, consider your data management processes to determine which platform will be the most cost effective and best suited to your strategy for incorporating mobile technology into your work.
Frequency: How frequently will the data be collected (e.g., every month)?
Data collectors: How many people are collecting data and how many will require mobile devices (e.g., 10 field workers, all need mobile devices)?
The tool: How many items/fields are on the survey/data collection tool (e.g., survey is 15 pages long with 65 items)?
5. Conduct a mobile technology feasibility scan.
After you have a general idea of who will be using mobile data technology in your organization and how, you will want to conduct an initial feasibility scan to help you understand the mobile infrastructure and the cultural, political, and economic environment. This scan will help you ensure that your strategy is appropriate and realistic. Ask the following questions.
How widespread is network coverage? Which mobile service provider(s) make sense to use?
If coverage is poor in the targeted area, are Wi-Fi hotspots nearby, or is better coverage available elsewhere?
What data plans are available, and how much do they cost?
Are people able to keep their phones charged across the project area?
What are the data security and management issues to be aware of? Is privacy a consideration?
What are the legal and regulatory frameworks that might cause concern?
How will the use of mobile technology impact gender equity, ethnic relations, and access to power and resources? Will the use of mobile technology benefit some groups more than others?
Will using mobile technology bring unwanted attention to respondents or field workers?
Will privacy concerns or other factors bias the data in ways that a traditional approach would not?
6. Decide on the appropriate mobile technology platform.
After defining your mobile strategy objectives, identifying end users, and conducting a feasibility scan, the next step is deciding which mobile device and platform are most appropriate for your needs. Mobile platforms, the systems that receive data from mobile devices, provide a variety of features and functions, including data storage, verification, analysis, and reporting. See more in Chapter 3 of MERL Module 4.
With so many different mobile devices on the market, each with different plans and features, choosing a mobile device can be overwhelming. To quickly narrow your options, consider the
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 102
nature of your mobile initiative (one-off versus long-term), data requirements, method of transmission, power sources, and budget. Table 12.1 outlines the types of mobile devices you could purchase and the functionality of each.
Table 12.1: Mobile devices from basic to advanced
Basic Phone Feature Phone
Smartphone (Android) Tablets
Description Inexpensive cellphones with voice and SMS capabilities
Mid-range cell phones
Can access internet
Some have GPS capability and a camera
Advanced cell phones
Can access the internet
Most have GPS and a camera
Low-cost tablets
Come with Wi-Fi and may come with 2G/3G/4G
Pros Ubiquitous, cheap, known
Low power use
Variety of features available
Can transmit GPRS data
Moderate power use
Software can be updated and upgraded
Can use any language
Larger screens make for good visibility and ease of entering data
Low cost of some models
Cons Moving parts of flip phones and slider phones affect reliability and durability
Creating texts on basic keyboard can be challenging
Not all have GPS
Language compatibility issues
Small screen
Heavy power user, needs recharging frequently
Might be targeted by thieves
Heavy power user, needs recharging frequently
Might be targeted by thieves
Some are more expensive than other mobile devices
Manufacturers Nokia Nokia, Siemens, Sony Ericsson (Java phones)
Samsung, LG, ZTE, Vodaphone, HTC, Apple
Aakash, Ubistlate, Google Nexus, Samsung, Apple
A number of mobile platforms offer a broad range of features and different pricing plans; several open-source options come at no cost.
To determine which platform is right for your mobile initiative, consider your needs and overall project budget. To a certain extent, your project’s data needs and the frequency and type of data collection outlined in your mobile strategy will help you narrow your options.
Please keep in mind that platforms change rapidly and that features and price structures evolve over time; the latest information is always available on the platform’s website. When deciding among mobile platforms, consider phone requirements; the data entry interface and transmission methods permitted; data storage, analysis, and reporting features; miscellaneous features; and the pricing structure.
Table 12.2 shows the most commonly used mobile platforms on the market today; however, new options open daily. For additional details on the different mobile platforms, please see Chapter 3 of MERL Module 4.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 103
Table 12.2: Commonly used mobile platforms
Features Mobenzi Commcare Magpi DataWinners Frontline SMS iFormBuilder
Language Does not work in all alphabets yet
Not alphabet restricted
Not alphabet restricted
Not alphabet restricted
Not alphabet restricted
Not alphabet restricted
SMS No Yes Yes, at pro level Yes, costs more Yes No
Forms Yes Yes Yes Yes Yes, in old version Yes
Type of phones for forms
Java, Android, Blackberry, Windows
Java, Android Java, Android, Blackberry, Windows
Android Android and Apple
Web portal entry No Yes Yes Yes No Yes
Skip logic Yes, and repeat logic If you can code Yes No No Yes
Ease of survey setup interface
Low Low High Medium-high High High
Other features Images, GIS Audio, images, GIS GIS GIS -- Audio, Images, GIS and ESRI connector, attach files from phone memory, API, RFID
Survey Setup Support
Survey setup support Do it yourself (DIY; complicated) or set-up support (expensive)
DIY DIY DIY DIY
Price Price models differ, so which is more cost effective will depend on the size of the surveys, how frequently they will be conducted, etc.
To learn more Site: www.mobenziresearcher.com
Login: www.researchconsole.com
Phone list: http://www.mobenzi.com/researcher/Help/handset-support
Site: http://www.commcar
ehq.org/home/
Phone list: https://wiki.commcarehq.org/pages/viewpage.action?pageId=4718838
Site: https://www.magpi.com/login/auth
Site: https://www.datawinners.com/en/home/
Site: www.frontlinesms.org
Device list: http://www.frontlinesms.com/the-software/requirements/
Site: https://www.iformbuilder.com/
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 104
7. Develop your mobile technology strategy document.
The aforementioned examples highlight the need for projects to document procedures and
policies. A well-crafted mobile data collection strategy document should describe:
Purpose of the strategy document, including the challenge that mobile technology use will help you to solve
Goals and objectives of the mobile technology strategy
Data to be collected via mobile devices
Your planned mobile technology feasibility scan, including: - Available network coverage in the areas where you will be working - Comfort and experience level of mobile device users - Data plans and costs of data plans - Costs of mobile devices - Security issues and management - Internal and external technical support required
Tools and instruments selected and rationale behind selection
General data collection procedures
Reports/dashboards that will be generated
Integration of data into other systems
Data management and security plan
When You Should Continue Collecting Data on Paper
In some cases, it may not make sense to use mobile technology, such as follows.
Status quo satisfactory: When your data is not urgent and your current system is already meeting your standards for quality and efficiency, changing data collection methods may not make sense.
Security concerns: If stakeholders have concerns about the security of data leaving the country to an international data host, collecting data on paper may continue to make sense. The cost of having separate servers host data in country can be prohibitive for small data collection exercises.
For qualitative data collection: When you are collecting a lot of qualitative data, using mobile devices for numeric or even QWERTY-keypad input will limit the amount of information that can be entered quickly. If audio capture is possible, it may mitigate this issue if the survey is limited in size. Otherwise, paper data collection is most flexible.
Use of mobile devices impractical or unsafe: In some areas, using mobiles may make data collectors more vulnerable to violence or surveillance. Elsewhere, entering data on a mobile device as an interview is in process may be distracting for the respondent or even potentially bias responses, such as when the respondent is concerned for his or her privacy or security. Taking photos or recording video with phones can exacerbate these concerns.
It is important to consider the type of data you are collecting and who will be collecting it to make sure that the use of mobile technology is appropriate for the context.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 105
Develop a Budget
There are costs associated with mobile data technologies. The most cost-effective platform depends on the amount of data you collect, number of users, and number of survey items.
Costs of mobile versus paper data collection should be considered when deciding on the mode of data collection you will use. Table 12.3 outlines the considerations for both types of data collection mode.
Table 12.3: Data collection comparison: mobile technology vs. traditional paper
Cost of mobile data collection Cost of paper-based data collection
Number of mobiles (e.g., total number of data collectors plus backups or replacement mobiles)
Cost of submitting data (e.g., per SMS text message, or, if over the internet, per gigabyte)
Charging devices
Survey/database design
Technology support for survey setup (if needed)
Training of data collectors
Mobile platform (e.g., database, data verification, data analysis and reporting functionality)
Human resources (e.g., to manage mobile provision and data collection systems and processes)
Costs for printing, pens, and paper and for transportation for a limited number of paper backup survey materials
Printing costs
Pens and paper
Transportation of paper surveys
Training of data collectors
Human resources for, e.g., data entry, transportation, supervision of data collectors
Database design, data entry, and data verification
Storage of paper surveys (particularly if dealing with sensitive information)
Costs for each platform varies: some charge an annual fee for unlimited use (iFormBuilder), while others charge per data field collected (per survey question, for example—Mobenzi’s model). Of note, there are plenty more mobile software options than only Mobenzi and iFormBuilder, and projects are encouraged to investigate and produce cost estimates of a variety of platforms before making their decision.
Costs related to powering mobiles may need to be budgeted as well. Some options to ensure reliable power include:
Chargers that plug into car cigarette lighter sockets
External battery sources
Spare internal batteries
Solar chargers
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Chapter 12: Mobile Technology for Data Collection | page 106
Summary: Key points from this chapter
Use of mobile technology has proven to be a powerful and efficient tool for collecting data that can accelerate achievement of project objectives and, ultimately, of development goals.
Projects from diverse technical areas successfully have adopted mobile technology to perform survey data collection, routine M&E, patient tracking, crowdsourcing, and conflict early warning systems, among other data collection needs.
It is important to consider the type of data you are collecting and who will be collecting it to make sure that the use of mobile technology is appropriate for your context. Remember that mobile technology only should be used if it helps to make your data collection and reporting processes more efficient.
When planning to use mobile technology, it’s important to consider: - Your specific data needs and data management processes in order to decide which
platform will be the most appropriate - Identify and understand the users of your data, including to inform type of device - Conduct an initial feasibility scan to help you understand the mobile infrastructure
and the cultural, political, and economic environment - Develop a budget, train users, and pilot the technology before wide-scale roll out
Learning Activity: Evaluating experience with mobile data collection
For programs already using mobile data collection: In small groups, discuss your experience with mobile data collection over the past few months to one year. What is working well, and what is not working well? Recommend actions to address the aspects that are not working well in close consultation with your M&E and information technology teams.
Discuss what other opportunities there may be for your organization to use mobile technology beyond existing use. What strategy will you use to expand to these other projects/areas?
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 1: Success Stories Guide | page 107
Appendix 1: Success Stories Guide
Partners should report stories that either:
1. Show broad-scale sustainable peace activities/projects, i.e., no one-offs and nothing that is not being implemented at scale; the writer may focus on an individual beneficiary to highlight their work, but the greater project must impact more than an individual or small group of people
2. Relate to the peacebuilding project they work on
3. Highlight new, cutting-edge development innovations
4. Highlight any contributions to funder priorities, for example Value for Money (DFID) or localization of foreign aid (USAID).
For more information on the USAID Reform Agenda, visit http://forward.usaid.gov or other funder priorities. For the DFID Value for Money policy, visit https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67479/DFID-approach-value-money.pdf.
All stories should point to results!
How to Write Stories
Use a compelling title. Include an intriguing lead or hook, beneficiary quotes, and powerful statistics to communicate progress.
Include a character/subject that illustrates how the program has improved people’s lives or made a difference in the community.
Consider specific photographs that “communicate” to back up the story. Make sure to caption and attribute the photograph(s).
Keep the story short and concise: 350–500 words.
Consider your target audience. Different audiences will have different points of view and needs and will respond to different ways of sharing the story.
Focus on qualitative information, but with sufficient supportive quantitative information.
Have enough details to help unpack technical information for non-technical readers.
Avoid abbreviations/acronyms and jargon.
Use quotes from beneficiaries. Do not quote activity staff.
Photographs
The process of taking and quality of a photograph can make or break a story. The following should be considered when using a photograph in a success story.
Digital photos should be shot with at least a 3-megapixel resolution and, when possible, maintain at least 300 dpi (dots per inch).
Send only graphics files (preferably JPEG/.jpg). Do not attach an MS Word document with the photo pasted into it.
Do not alter, compress, or crop photographs. Send only original images.
Do not scan images from publications or other printed materials.
Generally speaking, the larger the file, the better the quality and final result.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 1: Success Stories Guide | page 108
Include a caption that briefly summarizes what is occurring in the photograph: who, what, when, and where.
The photo should be colorful, depict action, and feature the main story character.
Ask permission before taking a photo of someone. Check funder, country, and your organization’s requirements for taking and using photographs of human subjects.
Play with different angles and backgrounds instead of capturing the subject straight on. The most interesting and visually pleasing photographs are often those with that capture the subject at unusual angles.
Use available light instead of flash whenever possible.
Model Stories
Before sitting down to write, consider reviewing the model stories you have come across.59 This will give you a sense of what makes for a successful submission.
The Plain Writing Act of 2010 (http://www.plainlanguage.gov/index.cfm) requires the United States government and its contractors to write all new publications, forms, and publicly distributed documents in a “clear, concise, well-organized” manner that follows the best practices of plain language writing. The site also includes some great writing pointers!
59 Some examples of success stories can be found on Pact’s website: http://www.pactworld.org/features
Example of story from peacebuilding program in South Sudan: http://www.pactworld.org/blog/south-sudanese-displaced-war-peacemaker-always-ready-help
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 109
Appendix 2: MERL Plan Template
Place organization (and funder, if required) logo here
Name of Organization/Project
Monitoring, Evaluation, Reporting, and Learning Plan
Add period covered by the plan
Date: (Insert date finalized/updated)
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 110
Contents
Abbreviations and Acronyms………………………………………………………………………………………….. 1 1. Introduction……………………………………………………………………………………………………………… 2 1.1. Mission and Vision……………………………………………………………………………………….. 2 1.2. Purpose of the MERL Plan……………………………………………………………………………. 2 1.3. Overview of Programs…………………………………………………………………………………… 2 1.4. MERL and the Project Cycle………………………………………………………………………….. 3 1.5. MERL Team………………………………………………………………………………………………… 3 1.6. Audience Analysis………………………………………………………………………………………… 4 2. Conceptual Framework……………………………………………………………………………………………….5 2.1. Organizational/Project Theory of Change………………………………………………………..6 3. Indicator Definitions, Data Collection, and Reporting Plan…………………………………………….7 4. Reporting…………………………………………………………………………………………………………………. 8 5. Data Quality and Data Verification Procedures…………………………………………………………….. 9 6. Monitoring Tools………………………………………………………………………………………………………10 7. Deliverables Schedule……………………………………………………………………………………………….. 11 8. Evaluation Plan……………………………………………………………………………………………………….. 12 Annex 1. Work Plan……………………………………………………………………………………………………… 13 Annex 2. Indicator Targets…………………………………………………………………………………………… 14
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 111
Abbreviations and Acronyms
Add a list of the abbreviations and acronyms used in your MERL plan in alphabetical order
and defined as they are used in the text. Some standard abbreviations are included below.
CBO community-based organization
CSO civil society organization
MERL monitoring, evaluation, reporting, and learning
NGO nongovernmental organization
TOC Theory of Change
US$ United States dollar
page 1
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 112
1. Introduction
1.1. Mission and Vision
State your vision and mission here. Explain each as needed for better understanding by users of the MERL plan and to show how these inform the goal of this MERL framework.
1.2. Purpose of the MERL Plan
State the purpose of having this MERL plan for your organization/project (e.g., to outline the process for carrying out effective MERL in the organization/project, to inform decisions on achievement of organizational results).
1.3. Overview of Programs
Provide an overview of all the programs/projects the organization is currently implementing and for the one(s) for which you intend to apply this MERL plan.
page 2
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 113
1.4. MERL and the Project Cycle
Edit as needed the following matrix to demonstrate how your MERL system aligns with the project cycle and with the award “prime” organization, if you are an implementing partner. Fill in each cell with information relevant to your program.
The project cycle Key activities to integrate the MERL system into the project cycle
Clarify the organization’s vision and mission Clarify the goals of the MERL system
Select policy issues of interest Select policy issues of interest
Identify stakeholders Identify MERL audiences
Identify who needs information and data from the organization, why, and when
Conduct situation analysis Set up participatory monitoring and evaluation framework (with input from the community and other key audiences)
Develop peacebuilding approach/Theory of Change (TOC)/conceptual strategy
Establish peacebuilding approach
Develop TOC, goals, and objectives
Clarify conceptual model and identify assumptions
Identify what to monitor and evaluate Develop project indicators
Implement activities Monitor progress: Collect data, report on progress
Analyze results and evaluate project Design and implement project reviews/periodic evaluations
Use results to adapt and learn Report on project results/review project objectives/strategies/approaches
1.5. MERL Team
Identify members of your MERL team and explain their roles and responsibilities. Use position titles and not individuals names. Be sure to include all staff and volunteers that have responsibilities related to MERL, not just MERL and Program officers.
page 3
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 114
1.6. Audience Analysis
Use this tool to outline who your audiences/stakeholders are and what data they need. This exercise can help your organization see if there are audiences you are currently missing and help you to avoid collecting data that you do not need. Don’t forget beneficiaries and volunteers!
Audience WHAT data is required WHY the data is required
WHEN the data is required
How this audience can participate in the MERL program/system
External audience
Internal audience
page 4
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 115
2. Conceptual Framework
Use the either the results framework or logical framework to demonstrate your expected project results. These frameworks show the logical connections between what your organization/project hopes to achieve and the activities it conducts. It gives the reader of the plan an overview of why you do what you do.
Project Results Inputs Activities Outputs Outcome Impact
R1.1
R1.2
R1.3
page 5
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 116
2.1. Organizational/Project Theory of Change
Briefly explain the Theory(ies) of Change and key assumptions on which your conceptual framework is based. In other words, tell the reader how you got to the point of showing the relationships between the inputs, outputs, outcomes, and impact results.
page 6
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 117
3. Indicator Definitions, Data Collection, and Reporting Plan
Use the following template to describe each of the indicators your organization/project will report on.
Program/ project result
Indicator
(show reference number and indicator name as they appear in your targets table)
Indicator definition and disaggregates
(show definition, disaggregation, and unit of measure as they appear in your targets table)
Data source (name of data collection tools)
Frequency of data collection
Responsi-bility for data collection
Frequency of data analysis and reporting
EXAMPLE
Goal: Strengthen the horizontal and vertical linkages within and between local, national, and regional conflict management actors within the Horn of Africa
R1.1
R1.2
R1.3
page 7
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 118
4. Reporting
Describe the types of reports the organization/project is required to prepare periodically for your MERL audiences, such as weekly updates, monthly reports, quarterly reports, and annual reports. You can use the following template to develop a reporting schedule to show when each type of report is due, who is responsible for preparing it, and to whom the reports are submitted. Annex all relevant reporting templates to this plan (e.g., those from your current funders).
Audience Communication tool selected for reporting Schedule for reporting
Responsibility for preparing report
To whom is the report submitted
EXAMPLE
Funder: Pact
Quarterly report April 5, July 5, October 5, January 5
Program manager Pact regional manager and MERL officer
Soft copy of event and incident reports
Monthly by 5th of the following month
Program officer Pact regional manager and MERL officer
Financial reports Monthly by 5th of the following month
Finance manager Pact grants manager
Final report 30 days after end of grant Program manager Pact regional manager, grants manager, and MERL officer
Final financial report and closeout audit
30 days after end of grant Finance manager Pact grants manager
page 8
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 119
5. Data Quality and Data Verification Procedures
Insert a graphic showing how data flows in your organization from the field level to the national level using the data sources or data collection tools. The graphic should show the points at which data verification is done before it is forwarded to the next level.
After drawing the graphic, describe the procedures your organization is carrying out to ensure data is of good quality at each of the following data management stages: source, collection, collation, analysis, reporting, use, and storage. If no verification and data quality management procedures have been specified before, use this plan to develop some that will guide the organization/project going forward.
page 9
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 120
6. Monitoring Tools
Include a matrix of the monitoring tools that you will use to track the progress of your
organization/project. Remember that these are not the tools for collecting data, but are
related to monitoring the organizational and operational aspects of program/project rollout.
These tools include things like conflict/context monitoring tools, work plan implementation
monitoring, and assumptions monitoring. It is important to state in your matrix how often the
information captured in the tools is collected and/or will be updated. Remember that these
data collection processes must inform operational issues, and thus must be updated as often
as the information would be useful for management. Do not waste time, money, and effort
using tools that have no operational impact when their content is reviewed! Include blank
examples of your monitoring tools as an annex to this plan.
Operational area of concern
Name of monitoring tool
Data collection frequency Update frequency
Conflict monitoring
Project implementation and expenditure
Assumptions monitoring
page 10
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 121
7. Deliverables Schedule
Refer to your award document and identify all the contractual deliverables that are due to your funder(s). Then, complete the following matrix. Update it quarterly, especially to record the actual delivery dates.
Funder staff who will receive the deliverable Deliverable(s) Due date(s)
Actual date of delivery
Means of delivery
EXAMPLE
Grants manager
Monthly financial report
Quarterly progress report
Event reports
The 5th day of the month following the event or the close of the month or quarter
November 5, 2015 (confirmed by email and currier receipts)
Emailed and hand-delivered hard copy sent to funder
page 11
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 122
page 12
8. Evaluation Plan
Include a basic evaluation plan that enables you to evaluate why you have or have not achieved the goal/results that were set. This plan allows you to look at consequences (intended or unintended), effectiveness, efficiency, outcomes, and sustainability of initiatives. Remember that evaluation looks at the overall program/project, the operations, governance, and deliverables! Basically, it helps you identify the lessons learned and what you would do better next time. Use a simple tool such as the table below to help you evaluate your overall program/project.
Type of evaluation What do we need to evaluate?
What questions do we need to ask?
How will we obtain the data?
When will we get the data? Who will do this?
EXAMPLE
Outcome evaluation
Progress made in reducing violence in conflict areas
Have people changed their behaviors, the way they interact with other communities, and/or their underlying attitudes?
One-to-one interviews with key informants
Survey questionnaires
Semi-annually and annually
MERL officer
Program staff
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 123
Annex 1. Work Plan
Attach a copy of the organization or project work plan here.
page 13
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 2: MERL Plan Template | page 124
Annex 2. Indicator Targets
Attach a copy of your final approved indicator targets here.
page 14
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 3: References | page 125
Appendix 3: References
Argyris, C., R. Putnam, & D. McLain Smith, D. 1985. Action Science: Concepts, methods, and skills for research and intervention. San Francisco: Jossey-Bass.
CDA Collaborative Learning Projects. 2013. Reflecting on Peace Practice: Participant Training Manual. Available at http://cdacollaborative.org/publication/reflecting-on-peace-practice-i-participant-training-manual/.
Channel Research. 2008. Evaluation of Conflict Prevention and Peace Building. Handout for INCORE University of Ulster Summer School.
Church, C., & M. Rogers. 2006. Designing for Results: Integrating Monitoring and Evaluation in Conflict Transformation Programs. Washington, D.C.: Search for Common Ground.
Church, C., & J. Shouldice. 2002. The Evaluation of Conflict Resolution Interventions: Framing the State of Play. Londonderry: Incore.
---. 2003. The Evaluation of Conflict Resolution Interventions. Part II: Emerging Practice & Theory. Londonderry: Incore.
Conflict Sensitivity Consortium. 2004. Conflict-sensitive approaches to development, humanitarian assistance and peacebuilding. Available at ttp://www.saferworld.org.uk/resources/view-resource/148-conflict-sensitive-approaches-to-development-humanitarian-assistance-and-peacebuilding.
Dziedzic, M., B. Sotirin, & J. Agoglia (Eds.). 2008. Measuring Progress in Conflict Environments (MPICE) - A Metrics Framework for Assessing Conflict Transformation and Stabilization. Washington, DC: United States Institute for Peace. Available at http://www.usip.org/files/resources/MPICE%20Aug%2008.pdf.
Fitzpatrick, J. L., J. R. Sanders, & B. R. Worthen. 2011. Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson Education, Inc.
Herrington, R. 2015. Emerging Practices in Design, Monitoring, and Evaluation for Education for Peacebuilding Programming (1st ed.). Washington DC: Search for Common Ground. Available at http://www.dmeforpeace.org/educateforpeace/wp-content/uploads/2015/11/EP-guide-_-Final-B-1027.pdf.
Igbo, A. 2013, Sept. 3. Ten Tips for Developing a Data Management System for Monitoring and Evaluation (M&E). The Communication Initiative Network. Available at http://www.comminit.com/job_vacancies/content/10-tips-developing-data-management-system-monitoring-and-evaluation-me.
International Bank for Reconstruction and Development & World Bank. 2012. Designing A Results Framework for Achieving Results: A How-To Guide. Washington DC: Independent Evaluation Group. Available at http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/designing_results_framework.pdf.
Jaszczolt, K., T. Potkański, & S. Alwasiak. 2003. Internal Project M&E System and Development of Evaluation Capacity: Experience of the World Bank-funded Rural Development Program.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 3: References | page 126
Kusek, J.Z., & R.C. Rist. 2004. A Hand Book for Development Practitioners, Ten Steps to Results Based Monitoring and Evaluation System. Washington, DC: World Bank.
Lederach, J.P., et al. 2007. Reflective Peacebuilding. Mindanao, Philippines: John B. Kroc Institute for International Studies.
Norad, DAC Network on Development Evaluation International Network on Conflict and Fragility, & OECD. n.d. Chair’s Summary: Evaluating conflict prevention and peacebuilding activities: Lessons from the application phase. Available at http://www.oecd.org/dataoecd/8/48/48092571.pdf.
OECD. 2002. Glossary of Key Terms in Evaluation and Results Based Management. Available at http://www.oecd.org/dataoecd/29/21/2754804.pdf.
OECD Development Assistance Committee. n.d. DAC Criteria for Evaluating Development Assistance. Available at http://www.oecd.org/dataoecd/42/6/49756382.pdf.
---. 2008. Guidance on Evaluating Conflict Prevention and Peacebuilding Activities: Working Draft for Application Period. Paris: OECD. Available at: https://www.oecd.org/dac/evaluation/dcdndep/39774573.pdf.
Patton, M.Q. 1997. Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks: SAGE Publications, Inc.
Pact. 2008. Building Monitoring and Evaluation Systems in Civil Society Advocacy Organizations (3rd ed.). MERL Module 1. Washington, DC: Pact.
---. 2013. Mobile Technology Handbook. MERL Module 4. Available at http://www.pactworld.org/blog/pact%E2%80%99s-mobile-technology-handbook-debuts.
---. 2014. Field Guide for Data Quality Management. MERL Module 2. Available at
http://www.pactworld.org/sites/default/files/DQM%20Manual_FINAL_November%202014.pdf.
---. 2014. Field Guide for Evaluation. MERL Module 3. Available at http://www.pactworld.org/documents/field-guide-evaluation.
PEACE III Program technical proposal to USAID
Sartorius, R. & Carver, C. n.d. Monitoring, Evaluation and Learning for Fragile States and Peacebuilding Programs. Arlington, VA: Social Impact.
Segone, M. (Ed.). 2009. Country-Led Monitoring and Evaluation Systems: Better evidence, better policies, better development results. Available at http://mymande.org/sites/default/files/images/Country_ledMEsystems.pdf.
Scriven, M. 1991. Evaluation thesaurus. 4th ed. Newbury Park, CA: SAGE Publications, Inc.
Stave, S. E. 2011. Measuring peacebuilding: Challenges, tools, actions. Norwegian Resource Centre.
Social Impact. 2009. Strengthening Monitoring and Evaluation for Fragile States and Peace Building Programs [PowerPoint presentation].
U.K. Department for International Development (DFID). 1997. Guidance on Humanitarian Assistance. Available at
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 3: References | page 127
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/404342/Humanitarian-Response-Funding-Guidelines-2015.pdf.
---. 2009. Political Economy Analysis How to Note. Available at: https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/3797.pdf.
U.S. Government. n.d. Plain Language.gov: Improving Communication from the Federal Government to the Public [website]. Available at http://www.plainlanguage.gov/index.cfm.
United Nations Development Programme (UNDP). 2009. Handbook on Planning, Monitoring and Evaluating for Development Results. New York: UNDP. Available at http://web.undp.org/evaluation/evaluations/handbook/english/documents/pme-handbook.pdf.
University of Reading Statistical Services Center. 1998. Project Data Archiving: Lessons from a Case Study.
U.S. Centers for Disease Control and Prevention. 1999. Morbidity and Mortality Weekly Report (Vol. 48). Atlanta: Centers for Disease Control and Prevention.
USAID. 2011. USAID evaluation policy. Washington, DC: USAID. Available at http://www.usaid.gov/evaluation/policy.
---. 2016. USAID Forward [web page]. Available at http://www.usaid.gov/results-and-data/progress-data/usaid-forward.
World Bank. 2007. International Program for Development Evaluation Training (IPDET) Handbook. Washington, DC: World Bank.
Bibliography for Peace and Conflict Terms
Call, C. 2008. Knowing Peace When You See It: Setting Standards for Peacebuilding Success. Civil Wars 10(2): 173–194. Available at http://www.informaworld.com/smpp/content~content=a793148283~tab=send.
Collier, P., & A. Hoeffler. 2004. Greed and Grievance in Civil War. Oxford Economic Papers 56(4): 563–595.
European Union External Action. n.d. Conflict Prevent, Peace building and Mediation [web page]. Available at http://eeas.europa.eu/cfsp/conflict_prevention/index_en.htm.
Fearon, J., & D. Laitin. 2003. Ethnicity, Insurgency, and Civil War. American Political Science Review 97(1): 75–90.
Lawry-White, S. 2002. Review of the UK Government Approach to Peacebuilding. London: DFID. Available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67946/ev646.pdf.
Le Billon, P. 2005. Fuelling War: Natural resources and armed conflict. The Adelphi series edition no. 373. Available at https://www.iiss.org/en/publications/adelphi/by%20year/2005-1a3b/fuelling-war--natural-resources-and-armed-conflict-ebea.
Monitoring, Evaluation, Reporting, and Learning (MERL) for Peacebuilding Programs
Appendix 3: References | page 128
Leonhardt, M. 2000. Toward a Unified Methodology: Reframing PCIA. In Bergdorf Handbook for Conflict Transformation. Available at http://www.berghof-foundation.org/fileadmin/redaktion/Publications/Handbook/Dialogue_Chapters/dialogue1_leonhardt.pdf.
United Nations. 2008. United Nations Peacekeeping Operations Principles and Guidelines. Available at http://www.un.org/en/peacekeeping/documents/capstone_eng.pdf.
United Nations Development Programme. n.d. Bureau for Crisis Prevention and Recovery Overview [web page]. Available at http://www.undp.org/content/undp/en/home/ourwork/crisispreventionandrecovery/overview.html.
United Nations General Assembly and United Nations Security Council. 2007. Report of the Peacebuilding Commission On Its First Session, June 2006 - June 2007, A/62/137-S/2007/458, 4. Available at http://www.securitycouncilreport.org/atf/cf/%7B65BFCF9B-6D27-4E9C-8CD3-CF6E4FF96FF9%7D/PBC%20A62137-S2007458.pdf.
USAID. 2005. Fragile States Strategy. Available at http://pdf.usaid.gov/pdf_docs/PDACA999.pdf.
---. 2012. Peacebuilding and Reconciliation. Available at https://www.usaid.gov/what-we-do/working-crises-and-conflict/peacebuilding-and-reconciliation.
Williams, A.Z. 1999. Sierra Leone: The Political Economy of Civil War, 1991–98. Third World Quarterly 20(1): 143–162.
World Bank. 2008. State and Peace-Building Fund. Available at http://www.worldbank.org/en/programs/state-and-peace-building-fund.