VAMESP II

Monitoring & Evaluation

Elementary

Opening and Introduction to the Course

VAMESP II

Getting to know each other?

Who am I ? What is my background? What do I do now? What is my involvement

with Monitoring and Evaluation

What are my expectations from this workshop / training activity

Illustration by Petra RÖhr-Rowendaal

VAMESP II

Objective of these sessions Review principles of M&E and methodologies of

implementation – Building awareness that M&E is more than just data

collection and progress reporting M&E is an integral part of good practice in project

management. Setting M&E in the context of regulatory and donor

requirements. Collaborative working on practical M&E issues

(logframes, developing indicators, preparing/planning for monitoring) as they affect course participants.

Introduce the planning tools required for robust M&E.

VAMESP II

Content – what will we discuss?

What is monitoring and evaluation? What must we do for GoV and Donors? What information do managers need? Planning for M&E using Logframe

approach? Selecting indicators for M&E? Developing an M&E Plan? Communicating M&E information.

VAMESP II

What method will we use? Advantages of participatory

learning

Brief presentations Discussion and information sharing Practical exercises Using the M&E Manual as a resource

Demands a collaborative and participative approach

Illustrations by Petra RÖhr-Rowendaal

VAMESP II

Monitoring & Evaluation

Elementary – National M & E System

Section 2

Legislation, Principles and Elements

VAMESP II

Framework for M&E of ODA investments

National Strategic objectives for ODA are based on: 10 year social and economic strategy A 5 year development plan The ODA Master Plan Comprehensive Poverty Reduction and

Growth Strategy Focal point for M&E of ODA is FERD in

the Ministry of Planning and Investment

VAMESP II

M&E Responsibilities Legal basis in Chap VI, Decree

131/2006/ND-CP Various levels of responsibility:

of Project management units of Line Agencies of Ministry of Planning and Investment

Various types of reporting Monthly, quarterly, annual and terminal

reporting

VAMESP II

Good Practice in M&E

Good practice principles of M&E Reporing and monitoring systems country owned

and led Donors aim to rely on country owned reporting and

monitoring systems Reporting and monitoring systems support country’s

own information needs M&E systems should be simple and harmonised Information should be shared and transparent Evaluation missions should be planned and

implemented in partnership

VAMESP II

Principles of the Vietnam M&E system

Usefulness Impartiality and independence Credibility Participation Harmonisation Scheduled monitoring and evaluation program Professionally designed evaluations Cost effectiveness Report communicate & feedback results Using results to manage

VAMESP II

Elements of the Vietnam M&E system Glossary of M&E terms Purpose and design Indicators that are S M A R T Measures Tools and pro-formas Analysis and interpretation Reporting Sharing information

VAMESP II

Monitoring flow chart

Chủ DA

VAMESP II

National Monitoring System-Progress

Quaterly Progress AMT thought Project Owner

PMU PMU PMU PMU

Quaterly Progress AMT thought Project Owner

PMU PMU PMU PMU

Quaterly Portfolio

PMT

Line Agency

Line Agency

Donor Donor

Quaterly Portfolio

PMT

Feedback & Banchmarking

Feedback&Benchmarking

National Agencies

VAMESP II

PMU

Annual PlanDisbursement & Outputs

Monitoring data:Actual disbursement

Monitoring data:Actual outputs

Ø Monthly progress report to MPI, MoF & line agency for Gp A projectsØ Quarterly progress report to line agency, MPI and donorØ Quarterly technical report to related technical ministriesØ Bi-annual national outcome indicators to line agency, MPI and donor

Monitoring data:National outcome indicators

M&

E R

epor

t

ODA monitoring - PMU

VAMESP II

Line

age

ncy

(Min

istr

y or

Pro

vinc

e)

Aggregate annual PlanDisbursement & outputs

Aggregated data:Actual disbursement

Aggregated data:Actual outputs

Ø Quarterly summary progress reports to MPI, MoF and donorsØ Quarterly feedback report to PMUsØ Bi-annual provincial or sector national indicators to MPI and donors

Analysed data:National outcome indicators

M&

E R

epor

t

ODA monitoring – line agency

VAMESP II

ODA monitoring – MPIN

atio

nal O

DA

Man

agem

ent A

genc

ies

Ø Monthly summary exceptions report to PM for cabinet meetingØ Quarterly summary progress reports to PM and donorsØ Quarterly feedback report to line agencies, PMUs and donorsØ Quarterly lessons learned to national M&E websiteØ Bi-annual summary of national indicators to PM and CG meeting

Dat

a Ex

chan

ge &

OD

A W

ebsi

te

National ODA PlanDisbursement & outputs

Consolidated data:Actual disbursement

Consolidated data:Actual outputs

Consolidated data:National outcome indicators

VAMESP II

Monitoring & Evaluation

Elementary – Foundations

Section 3

Communication

VAMESP II

Communication process

ChannelMessage ReceiverSender

Context

The Communications Process

Feedback

VAMESP II

Communication Process To get your message across to others

Communication process involves Sender Message Channel Receiver Feedback Context

Message is successful when the sender and the receiver perceive it in same way

VAMESP II

Communication process – The Sender

To be an effective communicator Establish credibility Display knowledge of the subject Display knowledge of the audience Know the context of message delivery

Know your audience Messages will be mis-understood if you

do not understand who you are communicating to!!

VAMESP II

Communication process – The Message

Written, oral and non-verbal communications are affected by:

Sender’s tone Method of organisation Validity of the argument What is communicated and what is left out Your individual style

Messages have intellectual & emotional parts

Intellect gives us ability to reason Emotion allows us to present motivational appeals, which

change minds

VAMESP II

Communication process – The Channel

Messages are conveyed through channels: Verbal face-to-face meetings Telephone and video-conferencing Written – letters, emails, memos and

reports

VAMESP II

Communication process – The Receiver

Messages are delivered to an audience Message will prompt actions or reactions

Audience brings ideas and feelings to communication process These influence understanding of the

message A successful communicator considers

these before delivering the message

VAMESP II

Communication process – Feedback & Context

Audience – provides feedback Verbal and non-verbal reactions

Monitor feedback It measures if the audience is

understanding the message Context – The situation in which the

message is delivered Surrounding environment or broader

culture

VAMESP II

Removing barriers Communication barriers can emerge

at any stage of the communication process

Aim is to lessen frequency of these barriers at each stage

Communication should be: Clear Concise Accurate Well planned

VAMESP II

Removing barriers at all stages If lengthy, disorganised or contains

errors: messages will be misinterpreted

Poor verbal or body language: Confuses the message

Barriers in context – come from too much information too fast

Understand your audience culture – converse and deliver your message to the particular audience

VAMESP II

Communication Models

What is a communication model Drawings Charts Diagrams Pictograms Schematics Cartoons

VAMESP II

Shannon & Weaver communication model

VAMESP II

Monitoring & Evaluation

Elementary - Planning for M & E

Section 4

The Logframe, Monitoring Planning and Logistics

VAMESP II

Logframe and M&E

The Logical Framework (Logframe) is a project planning matrix and a tool of analysis used at all stages of the investment cycle.

VAMESP II

Logical Framework Logical Framework – Logframe

A basic tool for investment design, implementation and monitoring and evaluation

Analyses existing situation Creates a hierarchy of actions to achieve outcomes Identifies potential risks to achieve outcomes Establishes how outputs and outcomes will be

monitored and evaluated Monitors investments during implementation

A number of names: PDM, LFM, LFA, ZOPP (but basic elements are the

same)

VAMESP II

Some Logframe Terminology Investment description Narrative summary Goal Purpose Component objective Objective Expected outcomes Objective hierarchy Hierarchy of results Intervention logic

Results Outcomes Outputs Activities Inputs Assumptions Indicators Performance

indicators Means of verification Frequency

VAMESP II

Logframe Matrix

Description Verifiable Indicators Means of

Verification

Assumptions

Goal

Purpose

Outcomes(Results)

Outputs

Activities

Inputs

Logical Framework Matrix

VAMESP II

Logical Framework Matrix

Description Verifiable Indicators Means of

Verification

Assumptions

Goal

Purpose

Outcomes(Results)

Outputs

Activities

Inputs

Logical Framework Matrix

VAMESP II

Developing a Logframe Matrix Is an iterative “step –by-step” process

The description (narrative summary) The assumptions The indicators The means of verification

Look back and test the logic Modify and test again

VAMESP II

Matrix Logic

VAMESP II

Linking LFA and Evaluation Criteria

VAMESP II

Developing a Monitoring Framework

Basic pre-conditions

Guiding the investment strategy for results

Create a learning environment

Ensure effective operations

Develop and use an M&E system

VAMESP II

The key steps in monitoring planning

Five steps Establish the purpose and the scope Identify performance questions, information

needs and indicators Planning information gathering and organization Planning critical reflection processes and events Planning the necessary conditions and capacities

VAMESP II

Contents of a monitoring plan Purpose and scope Approach Revised Logical framework and indicators Management information system and

reporting Monitoring work plan and timing of activities Establishing conditions and capacities Monitoring budget Annexes

VAMESP II

Summary monitoring framework

Logframe Reference

Indicator

What We will measure

How We will measure

Who will measue

Frequency of measurement

How results will be reported

Goal

Purpose

Outcomes

Outputs

Activities

Inputs

VAMESP II

Maintaining monitoring quality

Update the system to meet changing needs

Usefulness – M&E System provides practical information

Feasibility – methods and procedures are realistic and cost effective

Propriety – M&E activities are conducted legally, ethically and regard for welfare of those affected by the results

Accuracy – M&E outputs provide accurate information to support effective and efficient management

VAMESP II

Monitoring & Evaluation

Elementary - Indicators

Section 5

Types of Indicators

SMART characteristics

VAMESP II

Selecting indicators for monitoring

Indicators are “signposts” of change Indicators establish the measures on which

the monitoring system may be built. To select indicators it is necessary to identify

what information is needed to answer the performance questions.

What are performance questions??

VAMESP II

Performance Questions A performance question helps focus

monitoring activities on what is necessary to know if the investment is performing as planned and why that is so. Ask this question

“What question would we need to answer to know the extent to which an investment is achieving the objective and to explain the success or failure”

VAMESP II

Performance questions are not complex!

Logical Framework

Basic Performance Questions

Inputs What resources has the ODA investment actually procured and used?

Activities

What has the ODA investment actually done?

Outputs What has the ODA investment delivered as a result of its activities?

Outcomes

What has been achieved as a result of the outputs?

Purpose What has been learned from implementation of the ODA investment than can contribute to improved implementation or to building knowledge.

VAMESP II

Criteria and Indicators

provide the objective and measurable basis on which the assessment of performance and achievement can be analysed and allow a rigorous assessment on which judgements can be based in drawing conclusions and recommendations from the monitoring exercise.

they should be developed in conjunction with the stakeholders involved (including participation from the community) and may incorporate both quantitative and qualitative assessments.

VAMESP II

Principles in defining and selecting indicators

Do not remain static

Should complement existing systems

Developed in partnership with stakeholders

Simple and Cost effective to measure

Linked across scale Practical and objectively verifiable

VAMESP II

Characteristics of indicatorsTo be fully effective and usable

indicators need to be SMART:

Simple Measurable Attributable Relevant Timely

VAMESP II

Characteristics of Indicators

To be strategically meaningful indicators need to be clearly specified in terms of:

Inputs

Outputs

Outcomes

Impact

VAMESP II

Types (Signposts) of Change

Type of Change Example of ChangeThe presence of something

Number of restored district health centres

The type of access to a new innovation or service

The level of use

The extent of an activity or coverage

The relevance of the new innovation

The quality of an innovation

The effort required to achieve change

VAMESP II

Key types of indicators Quantitative

Easily quantified – measured by defined numerical values

kilometers of road constructed Number of commune primary schools provided

Qualitative Not as easy to measure with numerical values.

Narrative descriptions may be necessary Degree of participation of women Minutes of farmers meetings Perceptions of well being

VAMESP II

Categories of indicators

Leading indicators Measuring outputs/outcomes and performance drivers

Process indicators Routine data and measures used for attributing results

Lagging indicators Measuring final outcomes / impacts

VAMESP II

How to define indicators Decide on what to measure and why ?

Specify for each activity, intended result, or goal

Decide what is to be measured The quantity……..how much? The quality ………what? The target group…….who? The time/period……..starting when and for how long? The place………where?

Check whether the indicator(s) describe the overall purpose, results and activities accurately, If not, other indicators will have to be added, new ones found, or proxy measures used.

VAMESP II

What, How, Who, When

Logframe Reference

Indicator

What We will measure

How We will measure

Who will measue

Frequency of measurement

How results will be reported

Goal

Purpose

Outcomes

Outputs

Activities

Inputs

VAMESP II

The essential questions?? What to monitor and why?

How to obtain relevant and worthwhile information?

Who may be expected to use it and for what purpose?

VAMESP II

Types of Change

Type of Change Example of indicatorThe presence of something Number of restored district health centres

The type of access to a new innovation or service

Number of primary-aged school children with access to commune primary schools

The level of use The frequency of traffic use of new commune road

The extent of an activity or coverage

The proportion of mothers in the poorest 20% of households with access to village health workers

The relevance of the new innovation

Do seed banks resolve a production constraint or not

The quality of an innovation The quality of health services from restored District health Centres

The effort required to achieve change

The labour required for a new soil management technique

VAMESP II

Monitoring & Evaluation

Elementary – Data for Monitoring

Section 6

Baseline Data and Collection Methods

VAMESP II

The Data Journey

Source: IFAD 2002

VAMESP II

Data and Information

Data is raw material and has no meaning on its own

Synthesis and analysis of data produces information

Information explains and provides lessons which emerge as knowledge

VAMESP II

Types of Data and analysis Quantitative data

Analysis involves calculations – totals, average numbers, percentages compared to plan or targets. Can be more complex frequency, regression and cost-benefit analysis.

Qualitative data Analysis is very different and more difficult

than quantitative data. Can involve opinions, photo-points, stakeholder perceptions and non-standard answers

VAMESP II

Methods of comparison

“Before” and “after” Data gathered at the beginning, mid-term and

at completion

“With” and “without” (control) Compare data from locations with investment

activities and from locations without investment activities

“Variance from plan” Compare the actual performance or results with

the planned performance or results

VAMESP II

Sample analytical tool to transform data

GoV Participants Meeting Competency Criteria

05

1015

2025

3035

4045

H1-03

H2-03

H1-04

H2-04

H1-05

H2-05

H1-06

H2-06

H1-07

H2-07

H1-08

H2-08

Program half year period

GoV

Sta

ff P

artic

ipan

ts

(No.

)

-10.0%

10.0%

30.0%

50.0%

70.0%

90.0%

110.0%

GoV

Sta

ff P

artic

ipan

ts

(%)

Total participating 1/

No. meetingcompetency criteria 2/

% competent

Target

VAMESP II

Example – Comparing Activities

Activities Conducted Jan 2004 - Dec 2006

0

10

20

30

40

50

60

H1-2004 H2-2004 H1-2005 H2-2005 H1-2006 H2-2006

VAMESP Half-Year Period

Num

ber o

f Act

ivitie

s

Formal Training

OTJ Training Events

PCC Meetings

PMC Meetings

S H Consultations

TWG Meetings

Workshops

Study tours

Field Visits

VAMESP II

Monitoring & Evaluation

Elementary - Reporting

Section 10

Audience, Communication, Presentation & Interpretation

VAMESP II

Communication of findings

Why communicate M&E results Many potential audiences:

Government Funding agencies Steering Committees Cooperating institutions Implementing partners Primary stakeholders

Funders and managers need information on impact Partners and stakeholders want to understand problems

in order to find solutions

VAMESP II

Sharing Results of M&E

Build value throughout the M&E process to arrive at an informative, useful ‘story’ that makes an effective contribution to development cooperation

VAMESP II

Sharing results of findings

Develop a strategy for sharing results that optimizes the benefits to be derived from the monitoring & evaluation

Meet the information needs of key audiences while respecting the organisation’s agenda and context for development

VAMESP II

Communication opportunities

Carefully consider all opportunities for communicating results and be creative in devising a strategy that enhances the potential for making a ‘real’ contribution both within the organisation and externally.

VAMESP II

Communication principles

Identify the audience (use an audience matrix)

Integrate communication into M&E plan

Consider communication an investment

Communication requires feedback

VAMESP II

Communication and M&E Planning

Source: IFAD 2002

VAMESP II

Audience Matrix

Types of Monitoring Information

Types of

Audiences

Progress

toward

goal

Achieve-

ments

Economic

Impacts

Follow-up

Decisions

Actions

Community Members H M M L M

Project Management Unit Staff

M H L H H

Provincial Agency M H L H M

Ministry H L H H M

Funding Agencies H M H H M

Other groups L M L L L

Priority Code H = High M = Medium L = Low

Derived from: IFAD (2002)

VAMESP II

Practical Presentation of Information

Clarity of message for specific finding Frequency of communication Timeliness of feedback Consider location of the audience Effective use of graphics for analysis Keep feedback sessions focused

VAMESP II

Communication Media

Written reporting Qtrly, 6 mthly, annual & mid term reports,

newsletters Oral reporting

Direct discussion, radio programs Visual display

Graphs, charts, maps, photographs, video, drama

Electronic communication Email, electronic newsletters, websites

VAMESP II

Example – Interpreting data – Baseline in Time Series

GoV Participants Meeting Competency Criteria

05

1015

2025

3035

4045

H1-03

H2-03

H1-04

H2-04

H1-05

H2-05

H1-06

H2-06

H1-07

H2-07

H1-08

H2-08

Program half year period

GoV

Sta

ff P

artic

ipan

ts

(No.

)

-10.0%

10.0%

30.0%

50.0%

70.0%

90.0%

110.0%

GoV

Sta

ff P

artic

ipan

ts

(%)

Total participating 1/

No. meetingcompetency criteria 2/

% competent

Target

VAMESP II

Example – Interpreting data – Variance from plan

Plan VS Actual Disbursement - VAMESP II (A$)

$0

$200,000

$400,000

$600,000

$800,000

$1,000,000

$1,200,000

$1,400,000

$1,600,000

$1,800,000

$2,000,000

J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec J an Feb

Plan

Actual

VAMESP II

Example – Interpreting data – Comparing types of activities

Activities Conducted Jan 2004 - Dec 2006

0

10

20

30

40

50

60

H1-2004 H2-2004 H1-2005 H2-2005 H1-2006 H2-2006

VAMESP Half-Year Period

Num

ber o

f Act

ivitie

s

Formal Training

OTJ Training Events

PCC Meetings

PMC Meetings

S H Consultations

TWG Meetings

Workshops

Study tours

Field Visits

VAMESP II

Example – Interpreting data – Comparing progress against parameters

00.5

1

1.52

2.5

33.5

4Efficiency

Effectiveness

ImpactRelevance

Sustainability

VAMESP II

Monitoring & Evaluation

Elementary – Results Based Management

Section 12

Concepts and Definitions

VAMESP II

What is RBM

Results Based Management (RBM) is a management philosophy and approach that emphasizes development results in planning, implementation, learning and reporting.

VAMESP II

RBM needs participation Participation is an important ingredient

of RBM. Experience shows that participation improves the quality, effectiveness and sustainability of development actions.

RBM involves the participation of stakeholders in the: planning of projects & programmes; implementation of projects; monitoring and reporting of results.

VAMESP II

Results Based Management – Definition

Results Based Management is defined as: “A management strategyy focussing on

perfromance and achievement of outputs, outcomes and impacts “ (OECD-DAC 2002)

A “Result” is also defined by OECD-DAC as: “The output, outcome or impact [intended or

unintended, positive and/or negative] of a development intervention”

VAMESP II

Managing “Outputs” vs “Outcomes”

“Administrative” management Focuses on managing inputs,

activities and production of “outputs”

“Results based” management Focuses on management towards the

achievement of outcomes

VAMESP II

Comparing “administrative” and “results based” management

Administrative Results BasedDescription of situation before project start

Baseline data to define the situation before project start

Benchmarks for activities and outputs

Indicators prepared for outcomes

Collect data on imputs, activities and immediate outputs

Collect data on outputs – are they contributing to outcomes

Routine reporting on provision of inputs and outputs

More focus on perceptions of change among stakeholders

Directly linked to a discrete project

More qualitative & quantitiative analysis of progress to outcomes

Provides information on administrative, routine implementation and management issues

Project and strategic partners look at broader development effectiveness

VAMESP II

Good management Good management still requires:

The recording and monitoring of inputs and outputs; but

Recognises: That outcomes also must be monitored to

measure the success of a development activity. Outcome monitoring is a continual and systematic

process of collecting and analysing data to measure performance of ODA investments

VAMESP II

Key RBM Concepts RBM seeks to capture the process of change in the short,

medium and long-term. Development results are commonly linked together in a

results chain. Short-term results or outputs will contribute to medium-term results or outcomes which in turn will contribute to the long-term results or impact.

Completed activities are not short-term results or outputs. Outputs are the actual effects of completed activities.

RBM should identify the links between gender equality issues and the results the project seeks to achieve.

Beneficiary Reach refers to individual, groups or organizations affected by or benefiting from the results.

Results are influenced by the socio-economic context, gender analysis, the local capacity of stakeholders and organizations, the participation of stakeholders, available resources and the programme/project duration.