Open Days, Rome 3 July

Evolution of Uval's work program: from support to evaluation

Italian Evaluation Unit Open Days3 – 5 July 2006

Laura Raimondo

Open Days, Rome 3 July

Until April 2006

Ministry of Economy and Finance

Open Days, Rome 3 July

Department for Development Policies

CSF MA

Open Days, Rome 3 July

UVAL’s tasks

• UVAL founded in 1982• Reformed in 1998• Re-reformed in 2002• Main tasks:

– Promotion of data, information and knowledge for public policies

– Support to administrations to modernize and innovate public action,

– Evaluation of programs and capacity building for evaluation

– Ex-ante assessement of projects

Open Days, Rome 3 July

UVAL’S work program

• Activities (not staff) are organized in four areas– Methods and studies for public policy– Modernization and administrative innovation– Program evaluation – Project appraisal

• 104 activities• Support to other parts of the Department and to

the Network of the Italian Evaluation Units is still a large part of work (e.g., project appraisal, support to programming for 2007-2013)

Open Days, Rome 3 July

The Evaluation Unit (UVAL)

Area 1 Area 2 Division 3 Division 4

Metodi e studi per le politiche pubbliche

Trasferimento metodi e sostegno attuativo

Program Evaluation Project Evaluation

Tendenze e analisi delle politiche di

sviluppo territoriale

Sostegno attuativo ai servizi DPS e alle

amm.regionali e centrali

Coordination CSF evaluation system

Development of guidelines and standards for the

assessment of infrastructure and integrated projects

Rapporto territorialeIntese e APQ,

impostazione e performance.

Valutazione QCS e controllo qualità valutazioni dei PO

Progetto conti pubblici territoriali e

finanza pubblica

Attuazione Premialità QCS.

Metodi e indirizzi per la valutazione e  l'utilizzazione

della valutazione

Support for project cycle enhancement

Supporto e formazione Rete dei nuclei

Gruppi di lavoro (GdL) del QCS.

Conduzione e committenza di valutazioni su strumenti, piani e

programmi

Certified assessment of specific sectorial programmes

Comunicazione e pubblicazioni.

Contributo indirizzi e raccordi attività DPS

 

Metaevaluation of projects

     

Area 1 Area 2 Area 3 Area 4

Methods and studies for public

policies

Modernization and innovation of

administrative actionProgram Evaluation Project Evaluation

Trends and analysis of territorial

development policies

Support for implementation for DDPservices and for

regional and central adm.

Coordination of CSF evaluation system

Development of guidelines and standards for the

assessment of infrastructure and integrated projects

Annual reportInstitutional agreements

and FPA, support to design and

implementation

CSF evaluation and control of evaluation quality of OP

Project of territorial public accounts and

public financesImplementation of CSF performance reserve.

Methods and guidelines for evaluation and use of

evaluation

Support for project cycle enhancement

Support and training to the Network of regional and central evaluation units

CSF working groups Conduction and commissioning of evaluation on instruments, plans

and programmes

Certified assessment of specific sectorial programmes

Communication and publications Input to DDP guidelines

 

Metaevaluation of projects

     

Open Days, Rome 3 July

Program evaluation

• Evaluation capacity development– Started with Mezzogiorno & structural funds– Now expanding to whole country & all development

program– Huge task: guidelines, groups, observation of

practice & processes, common work

• Evaluations– Slowly growing – Legitimity (who asks for evaluations)– Resources (financial and staff)

Open Days, Rome 3 July

Evaluation: where we started from

• A “continental” public service: No orientation to results

• Absence of social partners and policy makers

• Push for evaluation from EU Structural Funds:

– No national reinterpretation of general framework

– Predominance of EC evaluative needs

– Rigid framework• Compliance at best purely formal

• No interactions

• Low quality of evaluations

• Little (if any) use of evaluations

Open Days, Rome 3 July

Evaluation capacity strategy initially relied on:

• EU evaluation requirements

+• New context of policy making

– More ambitious development policy– More emphasis on local knowledge– More emphasis on change– More responsibilities to regional governments– Greater importance to administrative capacity

Open Days, Rome 3 July

Close links between administration capacity and development policy

• Development policy requires administrative capacity

• Evaluation capacity is part of administrative (and political) capacity

• Evaluation results improve development policies if useful for key actors

• More than one way to get there

Open Days, Rome 3 July

A piecemeal strategy

• developed over time

• gradually introducing innovations

• knowledge from day-to-day work

• taking into account achievements and disappointments

We are still working on it

Open Days, Rome 3 July

Some evaluation capacity building activities UVAL worked on

• Drafting national Guidelines (2001-2002)

• Efforts to open up the market (2001)

• Setting up networks and discussion groups (from 2001)

• Creation of evaluation units (2000-2002)

• Meta-evaluation activities (from 2002)

• Self assessment exercises (2002-2003)

• Utilization of evaluation, evaluative knowledge, and implicit knowledge in discussing strategic lines for next programming period and for nat’l interventions (2005)

• Orientation to ex-ante evaluation process and products and organization of training and discussion groups (2006)

Open Days, Rome 3 July

UVAL role in the evaluation of Ob.1 CSF

•Organizing the evaluation:

• Suggested four segments (main contract, environment, cultural resources, macro model)

•Evaluation questions:

• Collected MA’s point of view• Drafted Terms of Reference• Dealt with EC, evaluator, MA

•Steering Group:

• Read and commented report drafts and reports• Informal dealings, formal meetings and feedback• Protected evaluators’ independence

Open Days, Rome 3 July

A part of the Network of Italian Evaluation Units

• Not ALL evaluation units:

– Evaluation Units dealing with Development Policies– Within executive branch – Some Ministries and all Regions– There are others: Regional elected assemblies,

Universities, other Ministries.

• Mix of tasks (actually evaluation only a small part)

– ex ante analysis, project appraisals, feasibility studies– direct evaluations (only one so far)– management of mid-term evaluations – Various roles in programming activities– Interaction with local authorities and coalitions on projects

• UVAL plays a proactive role in the Network, but has no formal “central” role.

Open Days, Rome 3 July

The National Evaluation System

– UVAL – Ministry of Economy and Finance (Coordinator)

– INEA – Ministry for agricultural policies– ISFOL – ESF Evaluation Unit – Ministry of

Labour– Evaluation Units of MA’s

Initially built to deal with evaluation of Objective 1 regions’ strategy and OPs

Now on the way to be extended to all Italian Regions’ regional policies

Open Days, Rome 3 July

External : Mid-term evaluation of CSF and OPExternal : Mid-term evaluation of CSF and OP

Internal:Internal:

Evaluation of the performance reserve Evaluation of the performance reserve systemsystem

Thematic evaluations (social inclusion, urban Thematic evaluations (social inclusion, urban development, research)development, research)

Self-evaluationSelf-evaluation

Different ways to organise evaluation

Open Days, Rome 3 July

EVALUATION OF EVALUATIONS, MID-TERM EVALUATION, UP-DATE

•Design•Early Reports

•Fieldworks•Intermediate Products and in-depth studies

•Fieldworks•Evaluation Questions•Organization of evaluation update•EC reads Evaluation reports•SNV reads reports for lessons and utilization

•Fieldworks•Updating Mid-Term Evaluation Reports

•SNV reads intermediate evaluation products •Working in team with Ob.1 Units (Questionnaire and Meeting in Oct).

EV

AL

UT

ION

O

F

EV

AL

UA

TIO

NS

MID

-TE

RM

E

VA

LU

AT

ION

OF

OP

an

d C

SF

Mid-Term Evaluation Report

20062005200420032002

Up-dating Mid-term :Strategy of CSF

•SNV reads preliminary products•Work with Ob.1 Units •Methodological talks SNV – Evaluators – MA’s – Units•SNV makes reports to CSF’s MC