Improving Public Policies in Brazil through Results: Developing

Capacities and Systems of Monitoring and Evaluation

CLEAR-FGV-IPEA

September 28-29, 2015

Orlando Gracia Fajardo

Monitoring and Evaluation System in Colombia: SINERGIA

Information for proper decision-making

Sistema Nacional de Evaluación de Gestión y Resultados-Sinergia

• Institutional framework: Sinergia as a building block for development planning

• Sinergia components: monitoring, evaluation and subnational capacity building

• Rating Sinergia

Institutional framework: Sinergia as a building block for development planning

Sinergia components: monitoring, evaluation and subnational capacity building

Rating Sinergia

Sinergia institutional evolution

EL PND es un instrumento más del esquema de planificación del país

POAI -PGN (Anual)

Visión

Plan de Gobierno

Plan Nacional de Desarrollo

Planes Sectoriales

Planes Indicativos / Institucionales

Programas

Proyectos

MFMP (10 años)

MGMP (4 años)

INSTRUMENTOS DE PLANEACIÓN

MFMP

(10 años)

Define y ejecuta la política

económica y fiscal del Estado.

Impulsar el crecimiento económico de

la Nación y controlar los gastos de

funcionamiento del Estado.

Prepara y coordina, políticas, planes,

programas y proyectos

Relacionados con la actividad productiva

y la inversión.

Presupuesto de Inversión

Gastos de Funcionamiento

La planificación en Colombia FUNCIONAMIENTO VS INVERSIÓN

Formular el PND cada 4 años

(pensamiento corto plazo)

•Adelantarse a la realidad actual y liderar la planeación para el largo plazo. •Apoyar el diseño de formulación de políticas de información pública y administrar el sistema de información gerencial del Estado.

Apoyar la formulación de políticas públicas, planes y programas alineadas al PND. Ejercer la Secretaria Técnica del CONPES. Hacer seguimiento y evaluación a las políticas planes y programas

Planear, articular y liderar cuando se le

asigne, las respuestas a temas puntuales y urgentes del Estado

Apoyar la formulación de proyectos Programar la Inversión Pública Hacer seguimiento al desarrollo de los proyectos

Facilitar la articulación de la Planeación Regional con el PND y los objetivos de desarrollo de LP

Institucionalizar los temas una vez estos se hayan diseñado

Liderar la construcción del portafolio de proyectos del País e impulsar la estructuración de proyectos

La planificación en Colombia EL ROL DEL DNP

DNP – Estructura orgánica

Dirección General

Subdirección Territorial Subdirección Sectorial Secretaría General

Dirección de Seguimiento y Evaluación de Políticas

Públicas

Dirección de Estudios Económicos

Oficina de Informática

Oficina Asesora Jurídica

Control Interno

Dirección de Desarrollo Territorial Sostenible

Dirección de Regalías

Dirección de Inversión y Finanzas Públicas

Dirección de Infraestructura y Energía

Sostenible

Dirección de Desarrollo Empresarial

Dirección de Desarrollo Social

Dirección de Desarrollo Rural Sostenible

Dirección de Justicia, Seguridad y Gobierno

Dirección de Desarrollo Urbano

Constitutional mandate

• Decreo 2167 de 1992 • Ley 42 de 1993 • Ley 152 de 1994. Ley organica

de planeación • Decreto 1633 de 2000 • Decreto 195 de 2004 • Decreto 1290 de 2014

Artículo 343 y 344 Constitución

Política de Colombia, 1991

• Conpes 2688 (1994) • Conpes 2790 (1995) • Conpes 2917 (1997) • Conpes 3294 (2004) • Conpes 3515 (2008)

Documentos Conpes 1994-2010

Institutional framework: Sinergia as a building block for development planning

Sinergia components: monitoring, evaluation and subnational capacity building

Rating Sinergia

Información efectiva para el mejoramiento de las políticas públicas

SEGUIMIENTO

SINERGIA

EVALUACIÓN

SINERGIA

TERRITORIAL

SINERGIA

RENDICIÓN DE CUENTAS

INNOVACIÓN + INVESTIGACIÓN

MONITORING EVALUATION

ACCOUNTABILITY

INNOVATION AND RESEARCH

Effective information for better public policies

The National System for Evaluation of Public Sector Management and Results

SINERGIA TERRITORIAL

The National System for Evaluation of Public

Sector Management and Results– Sinergia-

DSEPP (Technical secretariat)

National Planning Department (Technical Directions)

National Government (Presidency + sectors)

Citizenship (Civil society+ Academy)

Stakeholders (Part of the intervention)

Decision makers (Public manager)

Ge

ne

rato

rs

Use

rs

SINERGIA’s model is based on the understanding of the public policy value chain, which is oriented to:

1) Organize the logical sequence in the design of public policy,

2) Identify indicators for strategic monitoring and

3) Identify bottle necks to define the evaluation required

Goal Inputs Procedures Outputs Outcomes Impacts

Productivity, opportunity costs and

efficiency (Financial monitoring)

Efficacy (Strategic monitoring) Effectiveness (Evaluations)

Value chain

Sinergia's monitoring process is oriented to determine the achievement of governmental goals in terms of outputs and outcomes, providing continuing and reliable information about the achievement of the National Development Plan.

MONITORING SYSTEM

The System evolution

• TARGET: strengthen the management capacities of central government. The system allow the monitoring and the management of President´s priorities.

• STRATEGIC ANALYSIS: Use procedures indicators and the analysis does not have a strategic vision.

• TOOLS: A set of sectorial indicators.

• PERIODICITY: Emphasis on the medium-term analysis (biannual and annual).

• USE: Provides information on administrative and management topics, in contrast to broader aspects of effectiveness of public policy.

• TARGET: Continuously verify the compliance of PND goals, through early alerts and the use of effective information for improving decision making.

• STRATEGIC ANALYSIS: Use the Results-based management in public administration approach, and take into account the perception of change among stakeholders.

• TOOLS: The model use transversals, sectorial, and procedure indicators. Also, it use dashboards as a managerial tool.

• PERIODICITY: Emphasis on the short-term analysis (monthly and quarterly)

• USE: It generates systematic documents with qualitative and quantitative information about the compliance of PND goals based on the analysis of dashboards.

Monitoring as a system

SEGUIMIENTO

System administrators

Presidency

Directorate of Monitoring and

Evaluation of Public Policies -NPD-

Sou

rce

s o

f in

form

atio

n

Use

rs

Sector (ministry or

administrative department)

Report information

Validates information and analyze the progress in goals

Government

Decision making

Citizenship

Control bodies

MONITORING

The system is based on a set of indicators classified according to the value chain of public policy and that respond to a pyramid scheme based on roles and responsibilities of public administration.

Monitoring has a pyramidal structure of indicators

Strategic

Sectorial

Administrative Management

Related to topics that require intersectorial coordination.

Related to topics that are developed by sector authorities.

General performance indicators.

Monitoring outputs

Monthly report

Balance results

President report to Congress

Citizen perception survey

Dashboards definition identifying each sector contribution

https://sinergia.dnp.gov.co/portaldnp/

Sinergiaseguimiento.mp4 TableroDeControl.mp4

Today, all SINERGIA information, data and analysis are available at

internet ….

… the system has some views according to users …

Evaluation results are also available

It is necessary make coherent “what we want” with “what we do”.

National Planning: define objetives and aligns and coordinates government instruments interventions.

Accounting and Budgeting: stipulate production costs and define the allocation of resources for the creation of public value.

Monitoring and evaluation: allow to structure the information to verify the achievement of objectives in terms of delivery of goods and services, results achieved, and impacts attributable to government intervention. Besides, generates feedback to the process of public value creation.

Issues and challenges: Articulation of the National Budget and the Results-

Based Monitoring System

Monitoring Innovation Tools: The Delivery System (DS)

We have adapted the UK’s Delivery System methodology to the needs of our country. Is now conceived as an ideal tool to identify the reasons and bottlenecks of the policies or programs identified by our monitoring system. The experience of its implementation in the Land Restitution Policy: 1. The results: used by the leader of the policy. 2. The key to success: the active participation of the actors involved in the policy

implementation. 3. An ideal monitoring tool: it highlights bottlenecks, pitfalls and innovative practices within

the policy process and allows to deliver more efficiently. 4. The acquisition of know-how and goodwill: Sinergia has been changing the

understanding of the public policy towards the results-oriented model.

Monitoring Innovation Tools: The Perception Survey

The Survey aims to know the citizenship perception about the governmental performance and the achievement of the goals established in the national development plan.

Number of surveys done: 8 Sample size: 7,170 surveys (May – June 2014) Representativity: National, urban/rural, regional Error rate: 1,4% Sample design: probabilistic, multistage Components: Good governance Growth and competitiveness Opportunities equality Peace consolidation Environmental management Government’s balance Inputs for the new NDP (Needs and expectations)

A good understanding of the way how citizens are perceiving the results of the national government policies is vital for improve the quality of the governmental actions.

EVALUATION SYSTEM Sinergia understands the evaluation as a systematic and objective research applied in one of the links of the value chain whose goal is to improve designing, implementation, execution and effects of a public intervention through the enhancing of the policy making process

Strategic evaluations as a system

EVALUATIONS

Technical Direction at NPD

Supports prioritization, design and execution

Sector (Ministry)

Government

Citizenship

Use

rs

Control bodies

Head of the evaluated

intervention

Consultancy firms

Develop the evaluation execution

Coordinates, designs and facilitates the evaluation execution

Decision making

Implements recommendations

Propose recommendations

Supports the implementing of recommendations

In order to be effective, evaluations need to:

Be a result of a standardized process

Include the participation of all stakeholders

Answer decision maker questions

Be in line with the government agenda

Evaluation Schedule

• Design 3 Months

• Procurement 3 Months

• Development 8 Months

• Use of Results 6 Months

TOTAL: 20 Months

Selección de la política a evaluar

Diseño de la

Evaluación

Ejecución

Uso de

resultados

Area del Gobierno

Contratación

Firmas

externas

Selection of policies to

be evaluated

Evaluation Design

Procurement

Government Area

Evaluation Development

Implementing Results

External firms

Monitoring is enhanced with evaluation … However, Evaluations have to be effective …

Impact

29,6%

Outcomes

41,7%

Processes

23,5%

Executive

14,8%

Inputs Processes Outputs Outcomes Impacts

Institutional

18,3%

Sinergia has a wide evaluations portfolio, so that each type of

evaluation responds to specific bottle necks identified.

SINERGIA’s results – Type of evaluation

Since 2002 Sinergia has made more than125 evaluations

Evolution of the System

Additionally to social evaluations, Sinergia evaluates other sectors of public administration Sinergia has increased the types of evaluations Sinergia has generated methodological documents about evaluation

2 1

2 3

5

9

11 11

26

13

11

14

0

5

10

15

20

25

30

2002 2003 2004 2006 2007 2008 2009 2010 2011 2012 2013 2014

Evaluaciones 2002-2014 Tipos de evaluaciones 2002-2014

Ejecutiva

Operaciones

Institucional

Resultados

Impacto

Síntesis de evaluación

Quality check: Monitoring Committee of the evaluation

In order to guarantee the technical and methodological quality of the evaluation, this committee validates:

– The evaluation methodology

– Instruments and fieldwork methods

– Information gathering

– Data analysis

– Hypothesis validation

– Conclusions and recommendations

The Monitoring Committee is formed by:

Technical Dictatorate -

NPD

Sinergia - NPD

Entity in charge of the policy

For achieving the seal of quality of our evaluations, Sinergia works together with technical and methodological experts and the entity in charge of the policy…

Issues and challenges:

Implementation of recommendations Plan

(PTIR)

PTIR as a tool to strengthen the use of evaluations results and recommendations:

The objective is to present the recommendations as detailed routes for improving the evaluated program and be used as such.

Specific objective of the evaluation

Result/Finding

Recommendation

Actor responsible for the implementation

of the recommendation

Effects of implementation of

the recommendation

How does it work?

Design of a first draft:

consultancy firms

Revision of its technical and

methodological consistency: Monitoring

Committee of the evaluation

Improvement of the PTIR: consultancy

firms

Dissemination and

implementation of the PTIR: NPD

and evaluated entities

SUBNATIONAL CAPACITY BUILDING

What is it?

Goal

Process

Timeline

2012

It is a strategy to strengthen local authorities, through the support and technical assistance, in the design, implementation and development of monitoring and evaluation systems .

Adoption of monitoring capacities Capture of territorial information for national analysis

1 •Development of monitoreable plans

2 •Planning and implementation of monitoring systems

3 •Design and implementation of evaluation systems

INITIATION EXPANSION CONSOLIDATION SUSTAINABILITY

2012 2013 - 2015 2016

January February March April May June July August September October November December

Stage 2. Monitoring & Evaluation of Territorial

Development Plans Stage 1. Elaboration, debate and approval of

Territorial Development Plans

Sinergia’s Territorial approach

Institutional framework: Sinergia as a building block for development planning

Sinergia components: monitoring, evaluation and subnational capacity building

Rating Sinergia

Evaluation

Evaluation

Monitoring

Orlando Gracia Public microeconomics consultant

o.gracia@me.com