A project financed by the European Commission. The views expressed are those of the MWH Consortium and do not necessarily reflect those of the European Commission

Interim Evaluation of EU Pre-Accession Programmes in Turkey

(Service Contract No. 2007/142-454)

Kick-off for 2008 Monitoring and Evaluation

25 February 2008

IE Contract - The MWH Consortium

2

IE Turkey - Team and Contact Details

Contract start/finish

6 September 2007 – 5 December 2008 (15 months)

Team Leader Ms. Liz Cunningham

Experts Mr. Steven O’Connor

Ms. Sevil Geveci

Ms. Ayla Hekimoğlu

Mr. Kamil SorgunContact details Bogaz Sokak, 27/5,06700 G.O.P.

Ankara.

Tel: 0312 467 97 873

DG Enlargement - Unit E4

Task Manager – Jan BehrensDG Enlargement - Unit E4

Task Manager – Jan Behrens

Contract Methodology

Quality ControlSystem Oversight

MWH ConsortiumInterim Evaluation Team

MWH ConsortiumInterim Evaluation Team

Project Stakeholders

IE Conclusions & Recommendations

Interim Evaluation Input

PlanningCoordination

Feedback

ConsultationSystem Feedback

Follow-up of IE Recommendations

Delegation of the European

Commission

Secretariat General for EU

Affairs

Interim Evaluation Framework

4

Interim Evaluation – What it is NOT

X Interim Evaluation is not audit....

X Interim Evaluation is not control.....

X Importantly, Interim Evaluation is not

monitoring by the Commission

services

5

Interim Evaluation – What it IS

Part of a general policy of evaluating all Community

expenditure

Helps to improve relevance, effectiveness, impact and

accountability of EU funds

Provides stakeholders with independent assessments

of the state of implementation

Participative and capacity building exercise

6

Interim Evaluation – WHY we do it

To assess the state of implementation of the projects

To assess the state of implementation of initiatives in the

sector

To provide information to improve implementation of

ongoing projects and sector performance

To provide lessons learned for future projects

To provide lessons learned in relation to the project

cycle and its implementation

7

How do monitoring and evaluation differ?

They are different, but interrelated functions, as they both contribute knowledge as a basis for accountability and enhanced performance;

–Monitoring: “Are we doing things right”? while

–Evaluation, in addition, “Are we doing the right things?” and “Are there better ways of achieving the results?”

Definition of monitoring,evaluation and audit

The Process – The 5 Criteria

10

Different Kinds of Impact

Criterion Definition FocusImmediate impact

The extent to which outputs are being used by the beneficiaries(Output: what is now in place that was not there before)

Direct beneficiaries

Intermediate impact

The extent to which the outputs are used by others not directly targeted by the project (‘spin offs’; planned/unplanned or positive/negative)

Direct and indirect beneficiaries

Wider/global impact

The contribution of the outputs, immediate, intermediate impacts to socio-economic developments

Society in general

11

Example of Impacts

Example: Laboratory equipment for testing hazardous chemicals

12

Ratings – Six levels

Summary Definition Explanation

HS Highly Satisfactory All specified objectives expected to be achieved within lifetime of the project

S Satisfactory The project is expected to largely achieve its objectives

MS Moderately Satisfactory

Not fully satisfactory but not weak enough to deserve a bad rating

MU Moderately Unsatisfactory

Not fully unsatisfactory, but not good enough to justify a good rating

U Unsatisfactory Most of the objectives will not be achieved

HU Highly Unsatisfactory

The project is not expected to achieve any of its objectives

13

Interim Evaluation – Rating Model

CriterionProject Relevance Efficiency Effectiveness Impact

Sustainability

Overall rating

Sub-sector 1 title

Project title 1

Project title 2

Project title 3

Total for sub-sector ↓→ ↓→ ↓→ ↓→ ↓→ →•Sub-sector 2 title

Project title 4

Project title 5

Total for sub-sector ↓→ ↓→ ↓→ ↓→ ↓→ →•

Total for the Sector → → → → → →•14

Interim Evaluation - Information Base

Documentation (Accession Partnership, Regular Reports, national strategies, policy documents, Monitoring Reports, project and programme level etc.)

Financial data (up-to-date status)

Structured Interviews with stakeholders (policy makers, management, EUSG, ECD, CFCU, experts, contractors, Twinning Advisers, beneficiaries etc.)

Surveys/questionnaires: depending on project

Other analysis as required (e.g. databases)15

Information Base – Documents Required

• Monitoring Reports

Good quality Monitoring Reports (on time, completed correctly, accurate and up to date information)

• Other documentation/Per Project1.Project Fiche

2.Technical Assistance – ToR, Inception Report, Progress Reports, important output documents (TA reports etc.), Minutes of meetings etc.

3.Twinning – Contract, Quarterly Reports, output documents, Minutes of meetings etc.

4.Supplies/works – tender dossier, reports of evaluation committees etc. as required

5.Grant schemes – guidelines, list of applicants/grantees16

IE Reporting Process – The Results

1. Findings are presented in a draft Sectoral Interim Evaluation Report containing:

Conclusions and Recommendations of the independent Evaluators

Sectoral recommendations Operational recommendations aimed at specific

projects ‘Early warnings’

2. The draft report is finalised taking into account the comments as and where Evaluators judge necessary

3. The finalised report is debriefed where stakeholders discuss the recommendations and agree a plan of action

17

Interim Evaluation Reports 2007-2008

18

Interim Evaluation – Linked to SMSC Sector

Current Status

1 Internal Market Completed

2 Political Criteria At final stages

3 Environment/Transport Ongoing

4 Public Finance, Statistics, Support to European Integration

Ongoing

5 Cross Border Cooperation Starts early March 2008

6 Regional Competitiveness Starts early March 2008

7 Human Resource Development Starts mid April 2008

8 Civil Society Starts late April/early May 2008

9 Rural Development Starts mid May 2008