1

Monitoring and Evaluation of Social Programs

Kathy Lindert, World BankQualidade do Gasto Publico no Brasil

June 26-27, 2003

2

OutlineObjectives of M&E

Conceptual Framework, example

Monitoring (role, steps, etc.)

Evaluation (role, steps, etc.)

Dissemination and Uses

Institutional Issues and Sustainability

3

Objectives of M&EMonitoring and evaluation can help policy makers in many ways:

As planning and budgeting tool: to ensure effective allocations of government resources – planning, decision-making and prioritization

As a management tool by revealing the performance of ongoing activities at the sector, program or project level, to inform decisions about expansion, modification, or elimination of programs depending on their effectiveness

As an accountability mechanism, so that managers can be held accountable for the performance of their projects/programs and so that government can be held accountable for its performance

To obtain and maintain political support, institutional resilience (even across administrations), and financing for successful activities

4

Marco Conceitual

Resultados • Efeitos intermediários dos produtos sobre os clientes

Produtos • Produtos e serviços produzidos

Atividades• Tarefas empreendidos

para transformar insumos em produtos

Insumos • Financeiros, humanos, e recursos materiais

Metas(Impactos)

• Longo prazo, ampla melhoria na sociedade

Impl

emen

taçã

oO

bjet

ivos

5

Example: Secondary Education

Outcomes • Increased skills; more employment opportunities

Outputs • Number of youths completing secondary school

Activities • Schooling

Inputs • $, facilities, teachers, materials

Goal(Impacts)

• Higher income levels; increase access to higher skill jobs

6

Monitoring

7

Monitoring of Implementation and ResultsMonitoring is a continuous process of collecting and analyzing information to compare how well a program is being implemented against expected resultsMonitoring tracks both implementation and resultsEnfoque tradicional: Monitoring of activities Definir atividades e responsabilidades Monitoring activities is important

No entanto: É possível executar as atividades, cumprir as

responsibilildades, mas não fazer progresso até as metas

Question: how will you know when you have been successful?

“Se não se reconhecer o sucesso, vai se valorizar o fracasso”

8

Main Types of Monitoring

Output

Activity

Input

Outcome

Impact

Results Monitoring

Implementation Monitoring (enfoque tradicional)

Imp

lem

enta

tio

nR

esu

lts

9

Key Steps for Monitoring

Acordo sobre os resultados a monitorar

Seleção de indicadores de estos resultados

Medição de “baseline”

Eleição de metas quantitativas

Gerenciamento com relação às metas

10

1. Acordo sobre os resultados a monitorar

Outcomes make explicit the intended objectives of governmental action – the visionConsensus about these outcomes, and this vision, is important Participation in agreeing on outcomes is key Key stakeholders should be involved (e.g., government, civil society,

donors, etc.)

Desenhar para sucesso Evitar resultados impossíveis

Motivar pessoal com uma visão inspiradora Criar linha de vista (“line of sight”) para todos que trabalham no

sistema entre as suas ações e o resultado desejado

“Se você não sabe para onde vai, todos os caminhos estão certos”

11

2. Seleção de indicadoresOutcome indicators are not the same as outcomes

Indicadores devem medir resultados mas tambem processos e productos

Um resultado corresponderá típicamente a mais de um indicador

Respondem à pergunta:

“Cómo conheceremos o sucesso, se o vermos?”

12

2. Seleção de indicadores, cont.

A good performance indicator must be:

Clear (Precise and unambiguous)

Relevant (Appropriate to subject at hand)

Economic (Available at reasonable cost)

Adequate (Must provide a sufficient basis to assess performance)

Monitorable (Must be amenable to independent validation)

13

3. Establishing Baseline Data on Indicators

A performance baseline is…Information (quantitative or qualitative) that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used to: Learn about recent levels and

patterns of performance on the indicator; and to

Gauge subsequent policy, program, or project performance

14

Data Sources May Be Primary or Secondary

PRIMARY data are collected directly by the organization, for example, through surveys, direct observation, and interviews.SECONDARY data have been collected by someone else, initially for a purpose other than that of the program. Examples include survey data collected by another agency, a Demographic Health Survey, or data from a financial market. Secondary data often can save money in acquiring

data, but be careful!

15

Dados de “Baseline”

Conversascom

pessoasafetadas

Análisesdos

registrosofficiais

Métodos informais/menos estruturados

Visitas de

campo

Entrevistasdas

comunidades

Observação de participantes

Entrevistas cominterlocutores

chaves

“Gruposde foco”

Pesquisasúnicas

Questionários

Pesquisas de painel

Experiências controladas

Census

Métodos formais/mais estruturados

Observação direta

16

Practicality

Are the data associated with the indicator practical?

Ask whether…Quality data are currently availableThe data can be procured on a regular and

timely basisPrimary data collection, when necessary, is

feasible and cost-effective

17

4. MetasTargets are the quantifiable levels of the indicators that a country wants to achieve at a given point in timePrecondições Clara compreensão do passado recente Conhecimento dos recursos disponíveis

Considerações políticas Talvez o mais importante sendo o custo de não cumprir a

meta… Gerenciamento para sucesso

Seja realista! Faixas? Nota: gestão por resultados não depende críticamente das metas

18

5. Monitoria e gestão por Metas

“Abertura” faz parte do enfoque Desenvolver estratégias de divulgação para o

público, internet, etc.

Monitoramento deve ser contínuoInformações devem correr “verticalmente” e “horizontalmente”A nenhum ponto do sistema devem ser coletados dados sem ser analisados Cria incentivos em favor da qualidade dos dados

19

5. MonitoriaFidelida

de

Validade

Pontualidade

Metodologia estávele consistente

Medida clara e diretado desempenho relevante

Dados atualizados esuficientemente frenqüentepara influenciar decisões gerenciais

20

5. Monitoria: FreqüênciaQuanto mais observações tem, tanto mais os diagnósticos serãoSegurosSofisticados

?Tempo

Melhorar acesso aos mercados

Acc

esso

TempoMelhorar acesso

aos mercados

Acc

esso

21

Evaluation

22

O papel das avaliaçõesResults-Based Evaluation

• An assessment of a planned, ongoing, or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process.

Evaluation Addresses

“Why” Questions: What caused the changes we are monitoring

“How” Questions: What was the sequence or processes that led to successful (or not) outcomes

“Compliance/ Accountability Questions”: Did the promised activities actually take place and as they were planned?

23

Uses of EvaluationTo make resource decisionsTo re-think about the causes of a problemTo identify issues around an emerging problem, i.e. children dropping out of schoolTo identify alternativesTo build support of public sector reform / innovationTo help build consensus among stakeholders on how to respond to a problem

24

Evaluation Means Information on:

Strateg

y

Whether we are doing the right thingsRationale/justificationClient satisfaction

Operation

Whether we are doing things right Effectiveness in achieving expected

outcomes Efficiency in optimizing resources

Learning

Whether there are better ways of doing it

AlternativesBest practicesLessons learned

25

General Types of Evaluations & PERs

Evaluations can occur at many levels: Evaluations of individual staff performance

(including project managers) Evaluations of organizational units Evaluations of programs (focus of this

presentation) Evaluations of sectors/policy areas (e.g., cross-

sectoral issues, fiscal policy, overall education policy)

Public expenditure reviews generally evaluate the broadest level (evaluations of sectors/policy areas), but they also draw on evaluations of programs

26

Types of Program EvaluationsProcess Implementation Evaluations: Provides detailed information on whether program is

operating as intended (are we doing things right?) Provides continuous feedback to assist managers Provides detailed information to those interested in

scaling up or replicating Identifies bottlenecks

Cost-Benefit Evaluations: Calculates the cost of the intervention relative to

results Are we doing things efficiently?

27

Types of Program Evaluations, cont.

Impact Evaluations: Measure net effect of program to determine whether or

not objectives are being achieved Tries to establish causality between activities and impacts Impacts include both those that were intended and those

that were not Impact evaluations require a clearly established counter-

factual: What would have happened without the project? Generally need to compare those “with” the project (beneficiaries,

“treatment group”) and those “without” the project (“control group”) both before and after project

28

Stages of EvaluationClarify objectives of evaluationExplore available informationDesign evaluationForm evaluation team (capacity?)Collect information (sample, questionnaire, pilot, data collection, entry, quality control, etc.)AnalysisReporting, disseminating, discussing resultsUsing results: feedback to policies, budgets, planning

29

Dissemination and Uses of Information

30

Divulgação

Informar sobre o estado dos projetos

Prover índices sobre problemas

Explicar problemas

Criar oportunidades para considerar melhoras o alternativas

Prover informação sobre tendências

31

Usando as ConclusõesAs a planning and budgeting tool: Formular/justificar orçamentos Ajudar alocação operacional de recursos

As a management tool: Melhorar os serviços

As a performance/accountability tool: Responder às demandas para “accountability” Motivar pessoal Monitorar fornecedores Provocar debate sobre problemas

Reforçar confiança pública

32

Institutional Issues and Sustainability

Demand for M&E is crucial Structural requirements (ex. national law that mandates M&E of all

federal programs) Budgetary and planning links

Papéis claros Papel formal para organismos de planejamento e finanças

Credibilidade Sistema deve entregar boas e más notícias! Proteção dos gerentes contra conseqüências gravas

Responsabilidade (“Accountability”) Papéis da sociedade civil, mídia…

Capacidade técnica