Program Project Planning Lecture2.pdf

8/17/2019 Program Project Planning Lecture2.pdf

1/36

Planning, Monitoring and Evaluating

Programmes and Projects

Introduction to Key Concepts, Approaches andTerms

Working DraftVersion 1 – March 2000

Global Monitoring and Evaluation Initiative

Jim WoodhillIUCN M&E Facilitator forEast and Southern Africa


2/36

Introduction to PM&E Concepts, Approaches and Terms

Working Draft Version 1 – March 2000 IUCN Global M&E Initiative

IUCN – The World Conservation Union

Founded in 1948, The World Conservation Union brings together

States, government agencies and a diverse range of non-

governmental organizations in a unique world partnership: over

900 members in all, spread across some 138 countries.

As a Union, IUCN seeks to influence, encourage and assist

societies throughout the world to conserve the integrity and

diversity of nature and to ensure that any use of natural resources

is equitable and ecologically sustainable.

The World Conservation Union builds on the strengths of its

members, networks and partners to enhance their capacity and to

support global alliances to safeguard natural resources at local,

regional and global levels.

The IUCN Monitoring and Evaluation (M&E) Initiative

Through an approach which fosters questioning and reflectionand engages stakeholders at the regional and global levels, the

IUCN M&E Initiative aims to:

• develop a common understanding of M&E within IUCN

• develop a reflective culture within IUCN

• improve project/programme design and implementation

through the use of methods and tools in project, systems

and institutional assessments

• assess the relevance of the Union’s work against the

broader picture of ecosystem and human wellbeing

•

improve learning processes and reporting of lessonslearned

• put an overall M&E System in place for the Union.

Publications from the M&E Initiative are available on-line on the

IUCN website http://iucn.org/themes.html

Acknowledgements

Written by Jim Woodhill – IUCN Monitoring and Evaluation

Facilitator for East and Southern Africa.

Email – [email protected]

©: (2000) IUCN – International Union for Conservation of Nature and Natural Resources.

Reproduction of this publication for educational and non-

commercial purposes is authorised without prior permission from

the copyright holder provided the source is fully acknowledged.

Reproduction for resale or other commercial purposes is

prohibited without the prior written permission of the copyright

holder.


3/36

Introduction to PM&E Concepts, Approaches and Terms


Contents

1 INTRODUCTION ............................................................................................. .............................................. 1

2 APPROACHES TO PLANNING MONITORING AND EVALUATION ............................................... 2

3 KEY PLANNING, MONITORING AND EVALUATION CONCEPTS................................................. 3

3.1 A PARTICIPATORY LEARNING APPROACH ........................................................................................ ............. 33.2 THE MANAGEMENT CONTEXT .......................................................................................................... ............. 4

3.3 PROGRAMME AND PROJECT CYCLE ............................................................................................. .................. 6

3.4 PROGRAMME AND PROJECT LOGIC ................................................................................................... ............. 6

3.5 OBJECTIVE HIERARCHIES AND ASSUMPTIONS.............................................................................................. . 8

3.6 ACCOUNTABILITY AND CONTROL................................................................................................ ................ 153.7 K EY ASPECTS OF EVALUATION.................................................................................... ................................ 16

3.8 DEVELOPING AN OVERALL M&E STRATEGY/PLAN................................................................................ .... 173.9 DEVELOPING AND MONITORING EVALUATION QUESTIONS AND I NDICATORS ........................................... 193.10 OPEN E NDED EVALUATION ACTIVITIES.................................................................................................. 21

4 LOGICAL FRAMEWORK APPROACH (LFA) AND ZOPP................................................................ 23

5 RESULTS BASED MANAGEMENT ......................................................................................................... 27

5.1 CIDA RBM TERMINOLOGY......................................................................................... ................................ 28

5.2 USAID ROA TERMINOLOGY....................................................................................... ................................ 32


4/36


5/36

Introduction to PM&E Concepts, Approach and Terms 1


1 Introduction

This document has been developed to assist IUCN staff and partners

navigate their way through the terminology that surrounds the practice of

planning, monitoring and evaluation (PM&E) in environment and

development programmes and projects. It should be considered a workingdraft and will be revised and improved on the basis of feedback from those

who use it. A separate glossary of terms accompanies this document.

There is no question that planning, monitoring and evaluation are fields

littered with terminology that is often unclear and which is used with

different meanings by different groups. This problem can’t be solved by this

document, however, what it can do is to help explain the different

approaches to PM&E and how terms are used by different organisations.

Unfortunately confusion around terminology often makes PM&E seem

much more complex and difficult than is actually the case. In fact there are

really only a handful of key concepts that need to be understood aboutPM&E. If these concepts are understood then it is easy to make sense of the

different terminology and to translate terms between different approaches to

PM&E.

Monitoring and Evaluation (M&E) is often considered as a separate function

and responsibility from planning. However, if a project is poorly planned it

is very difficult to monitor and evaluate. Very often M&E staff and

specialists find themselves having to go back to basic planning principles

before they can assist with M&E. Hence the attention in this document to

planning and M&E.

Within IUCN it would simplify the difficulties of confusing terminology if a

standard set terminology could be used. The Global M&E Initiatives is

working towards such a set of concepts and terms. However, the reality is

that IUCN will always be working with different donors who demand the

use of their particular approach and terminology. This means that it will

always be necessary for staff to understand the underlying concepts of

PM&E and be able to translate between the approaches and terminology of

different donors and partner organisations.

The confusion about PM&E can also be reduced by understanding clearly

the way concepts and terms are used by different approaches and donors.

Consequently the later part of this document explores a number of

approaches in some detail.


6/36



2 Approaches to Planning Monitoring and Evaluation

Broadly there are three main approaches to PM&E in use by the major donor

agencies:

1. The logical framework approach (LFA) which is the most commonand widely used.

2. The German ZOPP, a close derivative of LFA. The acronym stands

for the German equivalent of objective oriented project planning.

3. Results Based Management (RBM) or managing for results, which

has become the favoured model of the Canadians and Americans in

recent years.

However, even within each approach there are often differences in the use of

terminology and many adaptations have been made as different groups put

the approaches into practice. Further, those within agencies who shouldunderstand the approach being used are often not as clear in their

understanding as would be ideal. This difficulty is compounded when

agencies are in a transition from one approach to another.

However, while there are certainly differences between the approaches, the

underlying principles of PM&E that they are each trying to promote are

remarkably similar. In essence, they are:

1. To develop programmes and projects based on a thorough

understanding of the situation in which an intervention is planned.

2. To involve stakeholders in a participatory process of programme or project design and evaluation.

3. To develop a set of clear logical objectives that can realistically be

achieved within a particular timeframe and within an allocated

budget and which will make a significant and sustained contribution

to a higher level development objective.

4. To make explicit the cause and effect (means ends) relationships and

external factors that underpin the programme or project and which

must hold true if planned activities are going to lead to desired

results and impacts.

5. To establish a monitoring and evaluation system, including

indicators, which will show if the objectives have been achieved and

provide information to support effective management and learning.


7/36



3 Key Planning, Monitoring and Evaluation Concepts

3.1 A Participatory Learning Approach

The approach to PM&E adopted by IUCN is one that emphasises the

participation of stakeholders in continually learning how to improve performance. Monitoring and evaluation is seen very much as a learning

process and not as an external top down ‘policing’ function. It is recognised

that given the complexity of conservation and natural resource management

and an extremely rapidly changing wider environment an adaptive approach

to programme and project management is essential. PM&E should be seen

as a process of helping people to learn how to do things better.

Consequently the theory and practice of adult learning is very important to

the monitoring and evaluation approach being developed within IUCN.

Given the breadth of this field in itself, the glossary will not attempt to cover

it in any detail. However, when using the glossary it will help to keep in

mind the idea of a participatory learning approach to monitoring andevaluation.

A participatory learning approach also means that there is much more to

M&E than just identifying and monitoring quantitative indicators. Learning

implies understanding, analysis, questioning, being critical and trying to

explain why things have worked or failed. Certainly quantitative indicators

are important and can be helpful but very often they provide only a small

part of the information needed for learning. Also for higher level objectives

or goals, such as improving protected area management or community well-

being it is just simply not possible to develop simple quantitative indicators

that have any real meaning. Unfortunately there is a widespread view thatdeveloping an M&E plan for a programme or project is primarily about

developing a set of such quantitative indicators. This document explicitly

challenges this perception. For example, well-facilitated review meetings

with staff, or the use of qualitative inquiry methods with beneficiaries, will

often provide much more valuable information for learning.


8/36



3.2 The Management Context

PM&E are essential functions of good management and should enable a

programme or project to achieve a high level of performance as illustrated in

the box below.

It is important to see monitoring and evaluation as tools to be integrated into

all aspects of programme and project management, as illustrated in the next

box. The starting point is to ask; “what information is required for effective

management and what sort of M&E system is required to provide it?”

Management Functions and M&E

Functions

Monitoring&

Evaluation

StaffingStaffing

OrganisingOrganisingControllingControlling

LeadingLeading

PlanningPlanning

PM&E

(learning)

Performance

PM&E - Critical Tools for Management

Management

should support

resulting in


9/36



Unfortunately M&E is often erroneously viewed as an annoying task of

simply providing donors with the information they require. Certainly

accountability to funding bodies is one function of an effective M&E system

but it is certainly not the only or the most important function. A list of

purposes for M&E is given below.

Purposes of Monitoring and Evaluation

Ensuring planned results are achieved

Improving and support management

Generating shared understanding

Generating new knowledge and support learning

Building the capacity of those involved

Motivating stakeholders

Ensuring accountability

Fostering public and political support


10/36



3.3 Programme and Project Cycle

The diagram below illustrates a generic programme or project cycle. It

emphasises the importance of starting with detailed scoping, situation

analysis and design stages. It also illustrates the importance of considering

M&E at all stages in the cycle. Importantly it also illustrates the need for

constant cycles of planning, acting, monitoring and evaluation (in other

words learning) during implementation.

3.4 Programme and Project Logic

There is an overall logic to any programme or project as illustrated below.

This logic can be described as follows. First, there is a situation that a group

of stakeholders wish to improve – the reasons for a programme or project.

This situation is defined, to a significant extent, by the problems and visions

of the stakeholder groups. There may or may not be common perceptions of

what the problems are or what would actually constitute an improvement,

which is why participatory approaches to planning are so important.

Developing a detailed and holistic situation analysis is a critical aspect of

programme or project planning.

The understanding of the situation will lead to a programme or project plan.

This will usually include:

The goal – a summary of what in the long term the programme or

project is contributing towards – related to impact.

The Programme/Project Cycle

Organisation

Mission

Financing and

contracting

Final Evaluation

Gov/Donor

Goals/Policies

Beneficiary

Needs

Plan

Act

Monitor

Evaluate

Implementation

Monitoring andEvaluation

Scoping

Formulation /Design

M&EStrategy

Mobilisation and

Implementation Planning

M&E

Plan


11/36



The Purpose – a summary statement of overall what the programme

or project should achieve given its timeframe and resources – the

overall outcome.

A set of Results – the main things that must be achieved for the

programme or project to realise its purpose, there may be several

levels of results. Results are also referred to as outputs and outcomes.

A set of Activities – what must actually be done for the results to be

realised.

A set of Inputs – the resources required for the activities to be

undertaken.

The programme or project is then implemented according to this plan

involving a process of inputs being used to undertake activities that lead to

actual results. Of course few projects go exactly as planned and there will

need to be constant cycles of planning, acting, monitoring, evaluating, re-

planning and so on. In some cases it may be necessary completely revise the

original plan.

The actual results should lead to a set of impacts that will improve theoriginal situation. In most projects there will also be unanticipated impacts

that may be positive or negative, which are also important to track.

Understanding this basic logic of a programme or project is the starting point

for understanding PM&E.

The Situation toImprove

Problems and Visions

Plan(Goal, Purpose,

Resultsand Activities)

Inputs Activities Actual Results

Reasonsfor Programme

Project

ProgrammeProjectand

ImplementatioProcess

IMPACT

Project/Programme Make Up and Logic


12/36



3.5 Objective Hierarchies and Assumptions

Any programme or project has an objective hierarchy, as shown below. This

maps out the way low level tasks or activities contribute to higher level

objectives and how meeting these objectives leads to the achievement of the

overall purpose or goal of a project or programme. Such a hierarchy of

objectives is also referred to as the intervention logic or the narrativesummary of a programme or project. It shows the cause and effect, or means

– ends relationships of an intervention. Developing a clear logical objective

hierarchy is fundamental to good programme and project design and

essential for M&E. It is called an objective hierarchy because any level

within it can be seen as being an objective. Higher level objectives are, or

should be, a consequence of achieving lower level objectives. Alternatively

it could be considered a results hierarchy. Although as will be shown later a

distinction is sometime made between an objectives hierarchy being what is

planned to be achieved and a results hierarchy being what is actually

achieved.

An objective hierarchy can be thought of as like the structure of a tree, with

the leaves or twigs being the detailed activities or tasks (low level

objectives) and the trunk being the overall goal (highest level objective) and

the branches being different intermediate level objectives. In theory therecan be many levels to an objective hierarchy, larger and more complex

programmes or projects require more levels than do small simple projects.

In practice, most planning approaches find four or five levels adequate. For a

programme one might imagine the trunk and main branches as being the

programme objectives and the sub-branches twigs and leaves as being the

projects that contribute to the programme objectives. In a very large project,

it may be helpful to think of having sub-projects. For a complex programme

Objective Hierarchy

Purpose

Goal

Key

Result

Key

Result

Sub

Result

Sub

Result

Sub

Result

A C T I V

I T Y

A C T I V

I T Y

A C T I V

I T Y

A C T I V

I T Y

A C T

I V I T Y

A C T

I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

Key

Result

A C T I V

I T Y

A C T I V I T

Y

A C T I V

I T Y

A C T I V I T

Y

SubResult

A C T I V I T Y

A C T I V I T Y

SubResult

Sub

Result

Sub

Result

Sub

Result

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

Assumptions(hypotheses)


13/36



with large projects, it is conceivable that it may be necessary to have a

programme, project and sub-project level of planning and each of these

levels having six levels of an objective hierarchy. In the end, planning needs

to be taken down to sufficient detail to enable day to day workplans for

activities and tasks. A well-developed objective hierarchy makes it clear

what must be done to achieve results and in reverse, along with indicators,

what results are achieved from completing activities.

An objective hierarchy also has a sideways logic. The outputs, products or

results from one strand of the hierarchy will often be critical inputs into

another strand. The conventional representation of an objective hierarchy

(as shown above) does not illustrate this sideways flow leading to the

criticism that such an approach to design is too linear and restrictive.

However it is equally possible to map out a project from a systems

perspective showing a series of interconnecting systems that have inputs and

outputs. It is important that the linear criticism is not used as an excuse for

poorly developed intervention logic.

Certainly, for complex programmes it will not always be possible to arrive at

a simple hierarchical logic, such as the one illustrate above, that adequately

expresses all the dimensions that need to be communicated. There may need

to be a number of ‘parallel logics’ or a matrix structure to the programme

framework. A detailed discussion of this is beyond the scope of this

document.

To assist in the planning process different levels within an objective

hierarchy are given different terms (goal, purpose, outcomes, outputs,

results, specific objectives, activities, etc). It is the lack of consistency in the

way terms are used for the different levels in an objective hierarchy thatcreates much of the PM&E confusion. However, as long as the concept of

different levels in a hierarchy is understood and the meaning of different

terms is understood by the stakeholders within a particular context, it doesn’t

really matter what they are called. Some examples of different objective

hierarchies that have a different number of levels and which use different

terminology are given on the next page.


14/36



Assumptions

An objective hierarchy should reflect cause and effect relationships between

lower and higher level objectives. The lower level activities/objectives are

the ‘means’ for achieving the ‘ends’ of the higher level objectives. There

will always be assumptions about the cause - effect (means – ends)

relationships in an objective hierarchy and these assumptions should be

made explicit in the design of a programme or project.

There are two types of assumptions, which are often confused in the logical

framework approach to planning. The first type of assumption is that related

to the internal logic of the programme or project. For example a project

design might be based on an assumption that by reducing poaching in an

area an endangered species will be preserved. This may be a correct

assumption or it may be incorrect because the main threat to the species

might in fact be reduced habitat or disease.

The second type of assumption relates to the external factors or external

environment that must exist for a project to succeed. For example, for a tree

planting project it may be assumed that rainfall will not be significantly

below average. For any project it will be assumed that there will besufficient political stability for the project to operate effectively.

For either type of assumption it is often helpful to identify, what are referred

to as, ‘killer assumptions’. These are the assumptions that if they are wrong

the project will fail completely.

Examples of Different Objective Hierarchies

Goal

Activities

Outputs

Purpose

Vision

Activities

Objectives

Goal(s)

Goal

Outputs

Outcomes

Purpose

Activities

Activities

Vision

StrategicObjectives

Mission

Goals

KeyResultAreas

Results

Goal

KeyResults

Purpose

Activities

SubResults


15/36



Good project design looks very carefully at assumptions and makes them

explicit. Likewise an important part of M&E is checking on the validity of

original assumptions.

Sorting Out Objective Hierarchy Terminology

Objective hierarchy terminology, as already mentioned, is the source of mostof the confusion around PM&E. To help sort this out it will be helpful to

discuss some basic ideas about different levels in an objective hierarchy

before getting too caught up in the terminology.

At the highest level of an objective hierarchy it is helpful to place the

programme or project within the context of some larger human endeavour.

For example an integrated conservation and development project may be

contributing towards the protection of the ecological, economic and cultural

values of a particular forest. However the project itself will not be able to

fully realise this highest level objective or goal, it can only make a

contribution. This level provides clarity about why a programme or projectis being undertaken. It helps to provide a sense of vision about the future for

those engaged with a programme or project. This level is commonly

understood as the goal for a programme or project. Some planning

frameworks call this level the vision and reserve the goal for the next level

down.

Impact is generally used to refer to the extent to which a programme or

project in fact does make a contribution towards the goal. However this may

not occur during the life of a programme or project and where others are also

contributing it may be difficult to desegregate the contributions made by one

initiative from those of another. This makes impact evaluation particularly

difficult, but nevertheless important.

Below this top level of an objective hierarchy is what can be considered the

purpose of a programme or project. This is overall what a programme or

project should achieve if it is successful. It is generally considered helpful to

try and summarise the purpose as a single statement to ensure focus and

clarity. For example “to develop the institutional frameworks and human

and organisational capacity for sustainable forest management”. This is

what a programme or project should be able to achieve within its available

resources and implementation period. If planning for an organisation this

level will usually be called the mission of the organisation.

Below this level are a series of major outcomes or results that need to be

achieved for the purpose to be realised. These should be the actual

observable changes in for example behaviour, institutions, economic

circumstances or physical conditions. Again these should be achievable

within the resources and timeframe of the programme or project. For

example, “staff within the forest department effectively carrying out their

responsibilities”. In the LFA this level is subsumed into the purpose level


16/36



and it is argued that there are outcomes at the purpose level. Other planning

frameworks refer to this level as the main objectives.

Each of these major outcomes or results arise from the logical consequence

of the programme or project delivering a set of products or services – outputs

– which arise directly from programme or project activities. For example

“forestry department staff trained in the skills required to carry out their

responsibilities”.

Not all planning frameworks make this distinction between outcomes and

outputs. It is this middle level of an objective hierarchy where terms and

concepts are most problematic. While outcomes generally refer to the higher

order changes or effects that arise from delivering outputs, which are

generally considered as tangible or concrete products and services, the

distinction is not always so clear cut. There is not a neat dividing line

between what can be considered outputs and are outcomes. In fact rather

than there being just two cause and effect steps there may be many, ie there

are also higher and lower level outputs and outcomes. While a slightlyartificial, the output and outcome distinction is still helpful in summarising

what can be expected to result at different levels within an objective

hierarchy. However, it needs to be remembered that it is a shorthand

summary and like all categorisations an approximation of reality.

This ambiguity around outputs and outcomes is why IUCN finds it less

confusing to talk of key-results and sub-results within the objective

hierarchy. The relationship between these terms is illustrated below.

Objective and Results Hierarchies Objective Hierarchy - what is

planned to be achieved

Results Hierarchy - what is

actually achieved

(also called impact or hierarchy and results

Actual Overall Result Outcomes

Actual Key ResultsOutcomes

Actual Sub ResultsOutputs

Goal

Purpose

(Planned)

Key Results

(Planned)

Sub Results

Activities

Impact

Outputs

Outcomes

Outcomes

Outcomes

Outputs

This is whereM&E terminology

can becomes veryconfusing

Inputs


17/36



The lowest level of an objective hierarchy is the activity level. For detailed

work planning it will often be necessary to break activities down the sub-

activities and/or tasks that will occur, for example over a particular year or

quarter.

From an M&E perspective it is very helpful to think in terms of impacts,

outcomes and outputs. However it is not necessary to limit impact questions

to the goal level of the objective hierarchy or outcome questions to the key-

result/outcome level of the hierarchy. This is easiest explained with an

example. Take the sub-result of a large integrated conservation and

development project - “sustainable livelihood activities adopted in target

villages”. An activity of this sub result may the promotion of bee keeping.

The real reasons for this may be to try and reduce the negative effects of

wild honey collection on a forest. The impact of such an activity would

relate to the extent to which a reduction of wild honey collection leads to an

improvement in the ecological values of the forest. The outcomes may

include the reduced level of actual wild honey collection, the level of success

adoption of bee keeping and extent to which bee keeping increases

household/village income. The outputs would be the number of farmers

trained in bee keeping, the formation of a bee keeping group, the provision

of hives or the development of a honey processing facility. In other words

even down to the activity level of an objective hierarchy it is possible to

examine impacts, outcomes and outputs.

In development work objective hierarchy thinking has been very much

influenced by the logical framework approach which has emphasised the

output level of planning and subsumed the outcome level into project

purpose. The idea was to focus on the tangible products and services that a

particular project management team should be directly responsible for

delivering. This has problems that manifest themselves in two ways. One,

the higher level results or outcomes that a project needs to achieve are not

made explicit and project management becomes very activity/output driven

and hence may not adapt its implementation strategy to achieve higher level

results when circumstances change. Two, in trying to overcome this

problem and attempting to focus on higher level achievements outputs come

to be used as essentially equivalent to outcomes. These issues relate to the

questions of accountability and control that will be discussed in the next

section.

The following table summarises the key terms discussed above.


18/36



Term Definition

Goal The longer term, high level improved situation that a programme or projects is contributing towards. The goal

helps explain why a programme or projects is beingundertaken. Generally the goal can only be achieved

through the combined efforts of others and a programme

or project cannot be solely responsible for the goal being

realised. Sometimes the term goal is used to refer to thehighest level direct results of a programme or project (ie it

is used interchangeably with what in this table is definedas the purpose). Vision is sometimes used in place off

goal. Example – The ecological, economic and cultural

values of XXX forest protected for current and future

generations.

Impact The extent to which a programme or project, or some part

of it, actually makes a contribution towards the goal.Impact is concerned with intended and unintended and

positive and negative contributions.

Purpose Overall what a programme or project, within the

timeframe and resources available, should achieve.Example – XXX forest being used and managed in asustainable way.

Outcome The observable changes in, for example, behaviour,

institutions, economic circumstances or physical

conditions that need to result from a programme or projectin order for it to realise its purpose and make a

contribution to the goal. Result or key result is also used

interchangeably with outcome. Example – Forestry

officers carrying out their responsibilities competently and

effectively.

Output The direct services or products that must be delivered forthe outcomes to realised. Result or sub result is also used

interchangeably with output. Example – Forestry officerstrained to develop collaborative management agreementswith local communities.

Result Used to refer generally to both outputs and outcomes.

There can be lower (sub) and higher (key) level results.

Example – as for outcome and output.

Objective Used generally to refer to anything that should be

achieved by a programme or project. Example – as for

purpose, outcome output depending on use.

Activity Specific actions that need to be undertaken for outputs to

be produced or outcomes/results/objectives to be realisedExample – Training workshop on collaborative

management conducted.

Sub Activity/ Tasks

A detailed breakdown of activities to the level required foryearly/quarterly/weekly/daily workplanning.


19/36



3.6 Accountability and Control

As you move up an objective hierarchy it becomes increasingly difficult to

directly control external factors that influence the achievement of results or

the goal of a programme or project. For example, a goal of a project might

be to improve the capacity of a government department responsible for

conservation. Part of this project might involve training. The project candirectly control the hiring of a training venue, the preparation of materials,

the provision of a trainer, and the notification of potential participants. It has

less control over whether potential participants will actually attend and

considerably less control, if any, over whether the skills the participants

learn will actually be used back in the organisational setting.

The issue of accountability and control relates very closely to the notion of

assumptions. For example assumptions will have been made that the

participants would use there skills back in the organisational setting

otherwise there would be not rationale for the project (or at least that

particular project activity).

There are two dimensions of accountability and control that are very

important to distinguish. The first relates to what a programme or project

management team should be accountable for if they have been given a plan

to implement. In this case they can really only be held accountable for

carrying out the activities and for the products or services that flow directly

from these activities - ie the outputs. The second dimension relates to the

overall accountability of a programme or project, which rests with those who

design, fund or have overall management responsibility. At this level there

should be accountability for the higher level results (outcomes) and the

purpose, even though the achievement of these is dependent on actions by

others over which there may be no direction control by the programme or

Within

project

control

Beyondproject

control

Goal

Activities

Purpose

Key Results(Outcomes)

Sub Results(Outputs)

Tasks

… what overall

the project can

reasonably be

accountable for

achieving

Ends … what the project

is contributing towards

Means Adapted from Materials Developed by ITAD

The Limits of Control and Accountability

… what is

within the

direct

management

control of a

project


20/36



project. However, the programme or project has been established on

assumptions that these actions by others will occur and the programme and

should be held accountable for the assumptions within the design.

Traditional logical framework approaches tend to emphasise the former and

suggest that the purpose of a project is outside the control of a management

team. Results based management approaches tend to emphasise managing to

achieve the higher level objectives (results).

3.7 Key Aspects of Evaluation

In developing any monitoring and evaluation system there are five aspects of

evaluation to consider as illustrated below. If you can provide information

on each of these you will be able to judge the overall performance of a

programme or project.

Relevance - Was/is the programme or project a good idea given the

situation to improve? Was the logic of the intervention logic correct? Why

or Why Not?

Effectiveness - Have the planned results been achieved? Why or Why Not

Efficiency - Have resources been used in the best possible way? Why or

Why Not?

Impact - To what extent has the programme or project contributed towards

its longer term goals? Why or Why Not? Have there been any unanticipated

positive or negative consequences of the project? Why did they arise?

Sustainability - Will there be continued positive impacts as a result of the

programme or project once it has finished? Why or Why Not?

The Situation to

Improve

Problems and Visions

Plan(Goal, Purpose,

Resultsand Activities)

Inputs Activities Actual Results

4. IMPACT

Key Aspects of Evaluation

3. Efficiency

1. Relevance

2. Effectiveness

5. Sustainability


21/36



3.8 Developing an Overall M&E Strategy/Plan

To effectively monitor and evaluate any programme or project it is necessary

to develop an overall M&E strategy or plan. A common failing for many

projects is that the only reference to M&E is the list of indicators and

monitoring mechanisms in the logical framework matrix table. This just

does not provide enough information to guide the actual implementation of aM&E system. The boxes below illustrate the process for developing and the

general content for an M&E plan.

Steps for Developing an M&E Plan1. Establish use and scope of M&E system

2. Check project objectives and logic

3. Establish overall evaluation requirements and questions

4. Establish requirements for regular monitoring of implementation

and progress towards desired results5. Test overall M&E strategy with potential users and refine 3 and 4

5. Establish the information and indicators needed for 3 and 4

6. Develop and test regular data gathering / monitoring mechanisms

7. Design open-ended and/pr periodic evaluation activities

8. Design information management system

9. Design a learning and feedback process

10. Decide how to evaluate the evaluation

- key evaluation questions

- focussing questions for learning lessons

- indicators and monitoring mechanisms

- open-ended evaluation activities

- participation and responsibilities

Contents for an Overall Project M&E Plan

Purpose and scope

Overview of approach (concepts, terminology, methods)

General project evaluation activities - eg ...☺ Annual internal reviews

☺ external reviews

M&E details☺ Goal level (impact)

☺

Purpose level☺ Results level

Appendices - eg ...

☺ Budget

☺ Details on indicators, monitoring mechanism, reporting

☺ Gnat chart of key M&E activities over project life


22/36



A few points are worth emphasising:

1. It is important to be clear about the overall purpose and scope of the

M&E system. In particular it should be clear who needs what sort of

information for what reasons, how extensive or minimal M&E needs to

be, and what resources are available. For example a project that has a

learning or action research focus will require a more comprehensive

M&E system than a project that is simply implementing a physical

works programme.

2. The overall system that will be needed for M&E to be effective must be

considered. This includes designing evaluation questions and indicators

that are relevant and practical, training staff in monitoring techniques,

developing monitoring forms and reporting processes, establishing an

information management system and establishing how information will

be analysed, reported and used.

3. It is particularly critical to design learning processes in which staff, beneficiaries, partners and donors participate. For example, annual

review meeting. Information from the M&E system should stimulate,

inform and support this learning process.

4. The whole M&E system must be developed around the use of

information. If information can't be used it’s collection is a waste of

time. Yet, it is remarkable how much fragmented and often unusable

data is collected by projects in the name of carrying out M&E.

5. It is important to recognise the difference between regular monitoring of

progress vs periodic and in-depth evaluation of some part of or the entire

programme or project. Regular progress monitoring will generally focus

more on output level indicators and the achievement of established

milestones or targets. Periodic in-depth evaluation examines whether

outputs are leading to expected outcomes and impacts, explores reasons

why and should assesses the effectiveness of the process of the

programme or project.

6. Responsibilities for M&E must be very clear and explicit in any terms of

reference, in job descriptions and be a core part of any staff performance

monitoring and appraisal system. If staff are asked to undertake M&E

work but it is not formalised as part of their core responsibilities it will

inevitably slip to the bottom of the work pile and never get done.

7. Make the M&E plan visual so everyone is aware about it and of their

responsibilities.


23/36



3.9 Developing and Monitoring Evaluation Questions and Indicators

A good M&E plan should clearly articulate the key evaluation questions that

need to be asked for each level of the objective hierarchy. To answer these

evaluation questions it will be necessary to identify information needs. The

necessary information may come from specific quantitative or qualitative

indicators, general project records, generally available information or from

specially designed evaluative or action research activities.

Traditionally a lot of emphasis has been placed on the development ofquantitative indicators as the key element in developing an M&E plan.

Starting at this point tends to narrow down and straitjacket an M&E system

and reduce its usefulness particularly in relation to supporting learning. For

good reason it is often very difficult or even impossible to develop sensible

quantitative indicators for the goal purpose and outcome levels of a

programme or project. Very often when quantitative indicators have been

developed for these levels they are either impractical to monitor or provide

relatively useless information in terms of overall evaluation of the result.

There is no question that indicators and in particular quantitative indicators

are an important part of an M&E system and wherever practical they should be used. However, an M&E system will be far more useful if it is designed

around the broad evaluation questions rather than narrowly focused

indicators.

In thinking about evaluation questions and indicators it is important to make

the distinction between evaluation and monitoring. It will often be necessary

and helpful to have some simple indicators that show regular progress

towards a result and which are monitored regularly. Output indicators are

Visualising an M&E Plan

Q1 Q3Q2 Q4

Year 1

Q1 Q3Q2 Q4

Year 3

Q1 Q3Q2 Q4

Year 2

Develop M&E

plan with

stakeholders

Preparation

for mid term

Mid termreview

Training in use

of reporting

system

Annual Review

and Planning

workshop

Annual Review

and Planning

workshop

PRA with

participating

communities

Preparation for

annual review

(performance and

lessons learnt)

Phase two

preparation

Key Meetings

Report Due

(Illustrative Only)


24/36



particularly useful in this regard. This is the ongoing monitoring of progress

that is required to manage a programme or project and which should show

early warning signs of problems. Evaluation is a more in depth and probing

assessment of the whole situation that should explore the reasons for success

or failure. This generally occurs less frequently. Different types of

information and indicators may be required for regular monitoring vs in

depth evaluation. These two different but related aspects should be reflected

in the description of an M&E plan for a particular result.

To effectively monitor and evaluation progress towards any particular result

(objective) in a programme or project the following steps will generally be

appropriate:

1. Identify the key evaluation questions for each level and result in the

objective hierarchy.

2. For each question identify what information or indicators will be

required to answer the question.

3. For each piece of required information or indicator establish:

• The methods and frequency for gathering the information or

monitoring the indicator.

• The baseline information required for comparison.

• What preparation and resources are required for the data to be

collected, collated and analysed, for example data collection and

analysis forms, training of staff, data base design, external

expertise.

• Who is responsible for carrying out each of the above and by

when.

4. For each question, or a set of questions, establish what overall analysis is

required and how the resulting knowledge will be used and what change

processes need to be in place to learn from and respond to the

knowledge.

5. Decide on an overall monitoring and evaluation plan for the particular

result. For example, how often will an overall evaluation of progress be

made and what indicators or information will be used to regularlymonitor progress and how often?


25/36



3.10 Open Ended Evaluation Activities

A good evaluation system should give adequate attention to what shall be

termed here open ended evaluation activities. These are all the aspects of

evaluation that complement an indicator based approach. Such open ended

activities (examples of which are given below) are necessary for the

following reasons:

1. There will often be unintended positive or negative results andimpacts from a project that will be missed by an evaluation that just

focuses on monitoring predetermined indicators.

2. Monitoring indicators alone often not provide an understanding of why

objectives have or have not been met. This requires discussion and

analysis with project staff and partners.

3. Monitoring indicators alone will not lead to understanding and learning

by programme or project staff and partners.

4. For complex or messy objectives it may not be possible to develop a

easily measurable indicator and the achievement of the objective mayhave to be demonstrated through more anecdotal information.

5. Monitoring indicators provide only limited capacity for evaluation of

the success or otherwise of the process of the project.

Detailed Result M&E Plan

EvaluationQuestions

RequiredInformation

andIndicators

DataGathering

Methods,FrequencyandResponsibilit-ies

BaselineInformation

RequirementsStatus andResponsibilit-ies

RequiredForms,

Planning,Training, DataManagement,Expertise,ResourcesandResponsibilit-ies

Analysis,Reporting,

Feedback andChangeProcessesandResponsibilit-ies


26/36



Examples of Open-Ended Evaluation Activities

• Annual Review and Planning Processes

• Monthly and/or Quarterly Review and Planning Processes

• Open Ended Impact Assessment

• PRA’s

• External Reviews

• Peer Reviews

• Stakeholder Meetings

• Regular Staff Meetings

• Analysing and Documenting Lessons Learnt

• Conference Presentations and Papers

• Advisory Committee Functions

•

Independent Assessments• Staff Performance Reviews

• Implementing Partner Performance Reviews


27/36



4 Logical Framework Approach (LFA) and ZOPP

The logical framework approach (LFA) and ZOPP have evolved from the

1960’s as methodologies for improving the systematic planning of

development projects. Over time, they have evolved from simply aframework for structuring project objectives to more sophisticated, process

orientated, approaches for involving stakeholders in project design and

management.

LFA and ZOPP are based around the following programme/project

development steps:

1) Systemic and participatory analysis of the situation in which some

intervention is anticipated

2) Clearly identifying the problem(s) to be addressed and identifying

the causes and effects of the problem(s). This is usually done bydeveloping a problem tree.

3) Using the situation analysis and problem identification steps to

consider intervention alternatives and to develop a logical hierarchy

of activities and objectives that will enable the problems to be

overcome.

4) Identifying the assumptions that underlie the logic of the objective

hierarchy ie being explicit about why it is assumed that particular

lower lever activities or objectives will lead to higher level ones.

Associated with this is identifying the external risks that may lead to

these assumptions not being realised and hence the project not

succeeding.

5) Establishing the indicators that will be used to verify that project

objectives have been achieved.

6) Developing the means by which information for the indicators will

be collected and analysed.

Various groups and facilitators have integrated an extensive range of

participatory planning methodologies and tools with the basic LFA/ZOPP

framework and quite sophisticated planning workshops have beendeveloped. There are numerous LFA/ZOPP manuals and documents.

The objective hierarchy for LFA and ZOPP usually has the following levels

and terms:

• Goal – the long term objective, change of state, or improved

situation towards which the programme or project is making a

contribution.


28/36



• Purpose – the immediate project objective ie the observable changes

in performance, behaviour or status of resources that should occur as

a direct result of the programme or project.

• Results (Outputs) – the products, services or results that must be

delivered by the project for the purpose to be achieved.

• Activities – the specific tasks that must be undertaken for the resultsto be achieved

The outcomes from such a planning processes are summarised in a project

planning matrix (PPM) or logframe table as illustrated below. It is important

to distinguish between the logical framework approach and the project

planning matrix. Often poorly planned projects, that in fact do not reflect an

LFA approach, are summarised in such a matrix.

Different terminology is used by different donors and other groups for both

the logframe objective hierarchy and the headings for the columns in the

project planning matrix. The main terminology used by the key donors is

summarised below. It’s also worth remembering that the staff of

development agencies aren’t always themselves familiar with the correct

definitions of some of the terms they are using. Different parts of the same

organisation may be using the same terms in different ways. Sometimes, theadoption of new terminology within these organisations takes some time to

reach all of the employees.

Outputs is the most commonly used term for the level between activities and

purpose, however the term results is now becoming more widely used, partly

reflecting the move towards results based management approaches an partly

because there is some confusion within the M&E terminology about the

meaning of outputs. IUCN has decided to use the term result rather than

Objective

Hierarchy

Indicators Monitoring

Mechanisms

Assumptions

and Risks

Activities

Goal

Purpose

Results(Outputs)

Project Planning Matrix (PPM)


29/36



output. The project planning matrix is usually only shown with one level of

results (outputs) however it is understood that there can be several levels of

results (ie key results and sub results) for a large and complex programme or

project.

Conventionally it has been understood that the inputs, activities and results

are within the direct control of a project, while the purpose and goal is

beyond direct project control. However this is a blurred rather than clear

line and depends on whether one is concerned with overall project

accountability (design, funding, eventual impact) or the just the

accountability for project implementation. With complex projects that need

to be adaptively managed even this distinction becomes blurred.

LFA and ZOPP have become widely accepted as useful and necessary tools

for project planning, however, they do have their weaknesses that include:

• focussing too much on problems rather than opportunities and

vision;

• being used too rigidly and leading people into a ‘blueprint’ approach

to project design and implementation;

• limited attention to problems of uncertainty where a learning and an

adaptive approach to project design and management is required;

• the tendency for poorly thought through sets of activities and

objectives to be entered into a PPM table giving the appearance of a

logical framework, when in fact the key elements of the analytical

process have been skipped;

• the simple logic of the LFA is often not appropriate to programme

level planning where it may be necessary to deal with a number of

parallel or cross cutting logics.

Despite these limitations and provided due attention is given to the

participation of stakeholders, and it is not used to rigidly the LFA/ZOPP

approach remains a very valuable tool for project planning and management.


30/36

Introduction to PM&E Concepts, Approach and Terms

Working Draft Version 1 – March 2000

Comparison of LFA Terminology Used by Different Donor Agencies

CIDA DANIDA DFID EC FINNIDA GTZ

Goal

Goal is

becoming

thestandard

term at this

level.

Goal Goal GoalOverall

Objective

Overall

ObjectiveOverall Goal

De

O

Purpose

Purpose or

ImmediateOutcome

are the

main

alternativesat this level

PurposeImmediate

ObjectivePurpose Purpose Purpose Purpose

O

Results

At thislevel, the

alternatives

are outputsor results.

Outputs Outputs Outputs Results Results Results

ActivitiesActivities

are used byall

Activities Activities Activities Activities Activities Activities A

(Source: ITAD Ltd Draft Glossary Developed for IUCN)


31/36



5 Results Based Management

Over recent years (Canadian) CIDA and USAID, in particular, have moved

to what has been called a results based approach and away from any explicit

use of the LFA. This development has arisen for two reasons. First because

it was recognised that more attention needs to be given to the actual

management of programmes and projects if planned results are to beachieved. Second because there has been growing pressure from donor

governments for donor agencies to demonstrate more explicitly the impacts

of development assistance.

While RMB and logical framework approaches do have slightly different

emphasis the underlying principles are quite similar. In essence they both

attempt ensure logical project design, that results are actually achieved and

that there are mechanisms for monitoring projects and demonstrating what

has been achieved.

Part of the reason for a movement away from the logical framework

approach was a perception that it was too ridged and did not provide for

enough flexibility in project implementation. Also the move to results based

approaches is an attempt to link development projects more explicitly to an

overall development strategy for the donor, the country or the region. Donor

agencies have become interested in showing the collective impact of their

entire portfolio of development assistance.

There is also a strong theme within the results based management of

managing a project to ensure higher lever results or project purpose are

achieved. This reflects an explicit recognition of the need for adaptive

project management.

CIDA defines Results Based Management (RBM) as:

“a management approach that centres on the establishment of a process and

environment where individuals work together to accomplish expected results.

The RBM process allows project managers to allocate or reallocate scarce

project resources based on performance information and incorporates lessons

learned into project management.”

USAID defines Results Orientated Assistance (ROA) or what is also referred

to as Managing for Results (MFR) as:

“A grant or cooperative agreement awarded to a Development Partner toachieve results that contribute to USAID’s performance goals.”

There are three principal elements of ROA:

1) a results orientated programme description

2) a performance measurement system


32/36



3) responsibility for performance

The ROA approach of USAID is designed to show how a particular project

contributes to the overall development assistance goals that have been set by

USAID and approved by the US Congress.

The main difference between RBM/ROA and LFA/ZOPP is that RMB/ROA

places as much emphasis on management and M&E as it does on the design,while LFA/ZOPP has tended to focus more on planning and design.

The RBM/ROA approaches are specifically designed to enable project

managers to cope with change and uncertainty and move away from ‘blue

print’ development planning. For example, USAID states:

“Overly prescriptive input-related detail should be avoided, in order to

preserve subsequent flexibility to adapt to changing circumstances “on the

ground” during implementation of the activity”

However, some experiences with USAID would suggest that this principle is

yet to be fully integrated into the various departments and processes that dealwith project approval. At times one will find different understanding

between the planning departments and the financial management and

contracting departments of agencies, the former saying flexibility and

adaptive management is fine while the latter demands much great rigidity.

In essence there is no particular conflict between LFA and results based

approaches, and LFA can be used in a perfectly complementary way within a

RBM context.

To further understand Canadian CIDA’s RBM and USAID’s ROA it will be

helpful to examine briefly some of the processes and terminology used by

each.

5.1 CIDA RBM Terminology

The basic framework of RBM is set out on the following page.

RBM is defined by the following characterists and processes:

• stakeholder participation;

• defining expected results;

• identifying performance indicators

• identifying critical assumptions and performing risk analysis

• organisational learning

• performance reporting

The monitoring and evaluation aspect of RMB is referred to as a

Performance Measurement Strategy (or Plan) (PMS).


33/36



RBM uses the following terminology and definitions:

Result: a describable or measurable change in state that is derived from a

cause and effect relationship

Developmental result: an actual change in the state of human development

that is the logical consequence of a CIDA investment in a developingcountry, measurable at output, outcome and impact levels.

Operational result: the administrative and management product of an

agency, its programs or projects.

Goal: the highest level objective that links a programme/project to a wider

set of strategies being undertaken to address a specific problem.

Purpose: the second level objective that defines specifically what the

programme, project or service is delivering and who are the beneficiaries.

Activities: the coordination, technical assistance and training tasks organised

and executed by the project personnel that transform inputs into results.

Inputs: the human organisational and physical resources contributed directly

or indirectly by the stakeholders of a project.

Impact (corresponds to project goal): a long term developmental result,

linked to the goal or vision, that is the logical consequence of achieving a

combination outputs and outcomes.

Outcome (corresponds to project purpose/component): a medium term

developmental result that is the logical consequence of achieving a

combination of outputs.

Output (corresponds to project activities): a short term developmental result

that is visible, concrete and tangible and is the logical consequence of project

activities.

Results chain: the sequence of results (or result hierarchy) from outputs to

outcomes to impacts.

Performance Indicators: qualitative or quantitative measures of project

inputs/activities (operational results) and outputs, outcomes and impact used

to monitor progress towards the achievement of expected result


34/36


35/36

Introduction to PM&E Concepts, Approach and Terms

Working Draft Version 1 – March 2000

RESULTS-BASED MANAGEMENTDefining Expected Results

Results Ch

Developmental

An actual change in the state of huthe logical consequence of a Cdeveloping co

Program / Project Management

Operational Results

The administrative and managementproduct of an agency, its programmesor projects

Inputs Activities Outputs Outcome

The humanorganisation andphysical

resourcescontributeddirectly by thestakeholders of aproject

The coordination,technicalassistance and

training tasksorganised byproject personnel

A short-termdevelopmentalresult that is the

logicalconsequence ofproject

activities

A medium-termdevelopmentalresult that is the

logicalconsequencesachieving acombination o

outputs


36/36


5.2 USAID ROA Terminology

USAID uses very similar terminology to that defined above in relation

RBM, however in the Results-Orientated Assistance Sourcebook there is no

clearly laid out set of key terms and definitions.

ROA uses the following hierarchy of results (objectives):

In relation to M&E ROA uses the following terms and definitions:

Performance: effectiveness in converting inputs to outputs, outcomes and

impacts.

Performance monitoring: a process of collecting and analysing data to

measure the performance of a programme, process, or activity against its

expected results.

Performance monitoring plan: a detailed plan for managing the collection

of data in order to monitor performance.

Evaluation: a relatively structured analytic effort undertaken selectively to

answer specific management questions regarding USAID-funded programs

or activities. In contrast to performance monitoring, which provides ongoing

structured information, evaluation is occasional. Evaluation focuses on why

results are or are not being achieved, on unintended consequences, or on

issues of interpretation, relevance, effectiveness, efficiency, impact orsustainability.

ROA Term Accountability forPerformance

Comments

Agency Goal USAID Agency

Agency Objectives USAID Agency

Mission/OperatingUnit StrategicObjective

Mission and/orWashington-basedOperating Unit – SOTeams

It is important recognisethat these top threelevels relate to USAID’sown objectives and sitabove those of afunded project.

Intermediate Result(s) DevelopmentPartner(s)

Outcomes DevelopmentPartner(s)

Outputs DevelopmentPartner(s)

Activities, Strategies,Processes

DevelopmentPartner(s)

These levels areestablished by thedevelopment partner (in

cooperation withUSAID) and define thefunded project.

Program Project Planning Lecture2.pdf

Documents

Transcript of Program Project Planning Lecture2.pdf