Monitoring Evaluation

Introduction

Monitoring Evaluation – relation to *Program management within an organization.

Evaluative objective – provides info to ensure that programs are working and they contribute to success

Monitoring Form- associated with the allocation of the resources

M& E is part of the total quality management and quality assurance thrusts.

Quality assurance is motivated by the need for governments to be seen to deliver high quality services.

Summary of Monitoring Evaluation (Form D)

Dimension Properties

Orientation Assessing Program processes and outcomes, for fine-tuning and to account for Program resources

Typical Issues • Is the program reaching the target population?• Is implementation meeting program objectives and

benchmarks?• How is implementation going between sites?• ….compared with a month ago?• How can we finetune this Program to make it more

efficient?• …to make it more effective?• Is there a Program site which needs more attention to

ensure more effective delivery

State of program Settled. Program plan is in place

Major focus Delivery and outcomes

Timing (vis-à-vis) Program delivery During delivery

Key Approaches Component AnalysisDevolved performance assessmentSystems Analysis

Assembly of evidence Meaningful use of valid performance measures• Quantitative indicators• MIS

Key Approaches to Monitoring Evaluation

Component Analysis - Senior management select a component of the Program for systematic analysis and review.

Devolved performance assessment- Senior management encourage all components of a Program to assess their performance on a regular basis.

Systems Analysis – A program which is centrally specified and disseminated for implementation to a large number of sites.

Component Analysis

Senior management select a component of the Program

Assess that component in terms of its own objectives and overall goals of the Program

The selection of the component is made on the grounds of concern*

Key Assumptions: Senior Management

Has sufficient overview of the organisation

Has the power to direct the evaluation unit to address the issue

Is a major audience for the evaluation findings

Devolved performance assessment

Senior management encourages all components of a Program to assess their performance on a regular basis.

Senior management receives these reports and, using appropriate criteria, makes judgments on the contribution of each component

Senior management provides guidelines, resources, and principles for judging

Systems Analysis

Applies to program which is centrally specified and disseminated for implementation to a large number of sites.

Program specification includes important goals.

Guidelines are provided for field staff; Field staff have little or no say in Program specification or implementation plans.

Evaluation scenario:

a set of important outcomes to be defined and made operational.

Using a centralized evaluation unit

Relating differences in attainment of the outcomes

Key Evaluation Questions

Is the program reaching the target population?

Is it being implemented in the ways specified

Ia it effective

How much does it cost?

What are the costs relative to its effectiveness

Monitoring Evaluation: Trends and Case

ExamplesEvaluands in program monitoring

Focus is evaluation within big “P” Programs which are ongoing.

Private sector- Ex: Training and development Program of a large regional bank

Public sector – Intellectual Disabilities Services Program

Common:

Contain mission statements

Designed to translate aspects of policy into tangible outcomes

Centrally planned or financed

Prime responsibility of Senior management

Ongoing, subject to modification

Elements of Program Management

A strategic plan for the implementation of relevant aspects of government.

Strategic planning – the process by which an organisation creates a vision of its future and develops the necessary structure, resources, procedures and operations to achieve that future.

A program structure

Management arrangements

The use of the MIS

MIS used as a basis for decisions

The use of MIS when considering fundamental issues

Large P Program evaluation

Unique characteristics:

1. There is a strong emphasis on outcomes.

2. Programs are ongoing and there is a need for evaluative information over time.

3. Many Programs are designed to provide goods or services rather than promote changes in behaviour

4. Evaluative data are often processed and reported in simple but logical ways.

5. Senior management, inparticular, may require gross or aggregated information

Assembling Evidence for Monitoring

Essential to use full range of data collection and analysis techniques

Indicators need to be at least part of the data collection

Indicators as Evidence

Key feature of indicators – used continually to inform decisions designed to alter the state of the social system affecting them..

Can be used as statements about the effectiveness of organisations.

Must be used in: comparing Program trends at different points in time (Monitoring)

To compare the performance of a Program to an acceptable set of standards or goals.

To compare the implementation of the same Program at different sites or locations

Types of indicators

Appropriateness – match between current community and government priorities and Program objectives;

Efficiency –the relative cost of achieving positive impacts via the program under consideration

Effectiveness – match between Program outcomes and Program objectives

Education for Sustainable

Development An Expert Review of Processes and

Learning Prof. Daniella Tilbury (University of Gloucestershire, United Kingdom) is the author of this publication commissioned by

UNESCO.

© UNESCO 2011

Section for Education for Sustainable DevelopmentDivision of Education for Peace and Sustainable Development UNESCO

7, Place de Fontenoy75352 Paris 07 SP France

Designed and printed at UNESCO Paris, France

Introduction

Monitoring and evaluation (M&E) is an integral part of education programme planning and implementation.

The United Nations Decade of Education for Sustainable Development (DESD, 2005-2014) is an endeavor that aims to reorient education policy, practice and investment to address sustainability.

This publication endeavors to identify which commonly accepted learning processes are aligned with ESD and should be promoted through ESD-related programmes and activities. It also seeks to examine which learning opportunities contribute to sustainable development.

Summary

The United Nations Decade in Education for Sustainable Development (DESD, 2005-2014) is a global movement which seeks to transform education policy, investment and practice. If it is successful, the DESD could change not only education but also the quality of life for many people across the globe.

Key Objectives:

i)  Which commonly accepted learning processes are aligned with ESD and should be promoted through ESD activities?

ii)  What are ESD and related learning opportunities contributing to sustainable development?

ESD learning frameworks and

processes The review has identified that certain key processes underpin ESD frameworks and practices. These include:

<  processes of collaboration and dialogue (including multi-stakeholder and intercultural dialogue);

<  processes which engage the ‘whole system’;

<  processes which innovate curriculum as well as teaching and learning

experiences; and,

<  processes of active and participatory learning.

Learning for ESD defined

Learning’ for ESD refers to what has been learnt and is learned by those engaged in ESD, including learners, facilitators, coordinators as well as funders. Often learning is interpreted as the gaining of knowledge, values and theories related to sustainable development but, as this review indicates, that ESD learning also refers to:

<  learning to ask critical questions;

<  learning to clarify one’s own values;

<  learning to envision more positive and sustainable futures;

<  learning to think systemically;

<  learning to respond through applied learning; and,

<  learning to explore the dialectic between tradition and innovation.

Critical Lesson through the review

It is difficult to access data on ESD processes and learning opportunities as these are rarely documented

There is a noticeable lack of data to show how these objectives and outcomes are achieved.

This relatively new field is only at the very earliest stages of generating the type of comparative and evaluative overview that provides a picture of effective processes and approaches.

The study recommends that during Phase II

i) data collection processes focus on actual experiences rather than reviews of the literature; and

ii) data collection tools are based on tightly- focused questions that will capture greater detail about learning processes

Critical Question

What is the extent and the depth of connection between the choice of processes in ESD initiatives and actual contributions to sustainable development?

the level of evaluative assessment within the literature is in its infancy,

the outcomes themselves are so varied and feature at multiple levels

external review of case study findings, anecdotal evidence from individual programme evaluations and the reflections of programme leaders seems to suggest that there are links that should be explored in more detail.

Contribution to sustainable

development this review presents a timely opportunity to consider the areas in which change

is emerging

The case studies reviewed in this document suggest that it is possible to map a wide range of contributions through ESD

The review unpacks and categorizes the range of potential contributions and some of the themes and priorities that are apparent across these key initiatives.

It has developed a template which could be adapted to serve as a data collation tool

ESD remains poorly researched and weakly evidenced.

This means there is not sufficient evidence to provide conclusive responses to the core questions that drive the present review and other similar investigations into the value of ESD as a field of research and practice.

These challenges will also confront the Phase II monitoring and evaluation report as it attempts to provide robust and meaningful evidence of the impact of the DESD initiative as a whole.