Post on 22-Dec-2015
agrir
egio
nie
urop
a
Exploring the perspectives of a mixed case study approach for the evaluation
of the EU Rural Development Policy 2007-2013
Ida Terluin and Petra BerkhoutAgricultural Economics Research Institute LEI
122nd European Association of Agricultural Economists Seminar
Evidence-Based Agricultural and Rural Policy MakingMethodological and Empirical Challenges of Policy Evaluation
February 17th – 18th, 2011, Ancona (Italy)
associazioneAlessandroBartola studi e ricerche di economia e di politica agraria
Centro Studi Sulle Politiche Economiche, Rurali e AmbientaliUniversità Politecnica delle Marche
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Common Monitoring and Evaluation Framework (CMEF)
CMEF is designed by the EU Commission for the
evaluation of the EU Rural Development Policy 2007-
2013
Comprehensive approach: about 160 indicators and 140
common evaluation questions
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Concerns on the CMEF
large number of indicators is experienced as a cumbersome
requirement of Brussels; data are not always available
simplification of the CAP: wish to have a simpler and more popular kind
of evaluation framework
evaluation questions are not always relevant in Member States/regions
emphasis on what has happened and not how or why it happened
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Aim of our paper
To explore whether there is an alternative approach to
the CMEF for the evaluation of the EU Rural
Development Policy 2007-2013 and to asssess the
perspectives of this alternative for EU wide use
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Plan of this presentation
1. Comparative analysis of evaluation methods of rural development
policy
2. Identification of an alternative approach for the CMEF
3. Testing of this alternative approach in the midterm evaluation of the
RDP 2007-2013 in The Netherlands
4. Assessment of the perspectives of this alternative approach for EU
wide use
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Comparative analysis of evaluation methods of rural development policy
Literature study: 22 recent evaluation methods
Classification of these methods according to applied methodology
into five goups:
1. the CMEF type approach: hierarchy of indicators combined with evaluation questions
2. the tally approach: counting whether a quantified objective has been achieved
3. the econometric approach: use of econometric methods
4. the modelling approach: use of models
5. the mixed case study approach: use of quantitative and qualitative analyses, often in case studies
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Main properties of evaluation methodsApproach
CMEF type
tally econo-metric
model-ling
mixed case study
What happened in quantitative terms
x x x x
What happened in qualitative terms x
How and why did it happen x
Unintended effects of policy x
Why actors participate in policy x
Easy to apply for evaluator x x
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Identification of an alternative approach to the CMEF
Starting point for identification: different properties of the
groups of evaluation methods
Given differences of CMEF and mixed case study
approach, this last approach might be a promising
alternative for the CMEF
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
The mixed case study approach
analysis consists of different stages, in which different methods are
applied
by combining evidence from these stages, it is tried to find exploring
patterns
approach has been applied for the ex post evaluation of the EU Rural
Development Policy 2000-2006 in Wales by Midmore et al. (2008):
1. analysis of secondary data of the case study region
2. in-depth interviews with representatives of different interest groups
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Design of a protocol with instructions for the various steps in the evaluation
1. Analysis of the baseline situation in the case study region
2. Analysis of the objectives of the EU Rural Development
Programme (RDP) 2007-2013 in the case study region
3. Analysis of the financial input and the output for each rural
development measure
4. Questions for in-depth interviews with representatives of the
different interest groups
5. A template for writing a report
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Testing alternative in midterm evaluation RDP in 2 Dutch provinces:(1) Zeeland and (2) Gelderland
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Experiences in case study in Zeeland (1)
Baseline situation in the case study region: easy to describe, as data
and literature were readily available
Objectives of the RDP: difficult to assess, due to complex way the
RDP is implemented in The Netherlands
– there is one RDP for the whole country
– national government is responsible for axes 1 and 2
– province is responsible for axes 3 and 4 and has integrated them
with other rural policies
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Experiences in case study in Zeeland (2)
Data on financial input and output indicators for each rural development
measure are collected by two institutes, which have different data systems
Data for input and output indicators for measures under axes 1 and 2 are
only available at national level, and it is very time-consuming to extract
regional data
Data on result and impact indicators in Zeeland are not collected
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Experiences in case study in Zeeland (3)
Progress in the absorption of the EAFRD budget 2007-
2009 in Zeeland:
Group 3 focuses on tourism activities, basic services,
village renewal and rural heritage, measures which fit
rather well into the rural development needs in Zeeland
1 Axes 1 and 2 no information
2 Measures 311, 312, 41 moderate
3 Measures 313, 321, 322 and 323
substantial
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Experiences in case study in Zeeland (4)
According to the six interviewed persons:
the landscape and quality of life have been improved in qualitative
terms by the RDP
these projects fit in the long tourist tradition in Zeeland as well as
the need to maintain or improve the supply of basic services in rural
areas
economic diversification has been boosted
they had no idea of the impact of the measures of axis 1 on the
development of the agricultural sector
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Experiences in case study in Zeeland (5)
actors participate in the RDP measures as this generates funds for
their projects
most of the projects would also have been conducted without
EAFRD financing, but often at a smaller scale
the requirement of 50% national cofinancing for each project is not
always feasible
interviewees appreciated the easy communication and the
openness of the province of Zeeland, which stimulates the
implementation of the measures of axes 3 and 4
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Concluding remarks (1)
1. Based on a comparative analysis of 22 evaluation methods of rural
development policy, we have identified the mixed case study approach
als an alternative to the CMEF
2. The results of testign this alternative in the province of Zeeland are
promising: the analysis of inputs and outputs of the measures in
combination with interviews provided sufficient information to get a good
overview of the performance of the RDP
agrir
egio
nie
urop
a
122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)
Concluding remarks (2)
3. The interviews yielded useful insights into processes within the
region
4. When experiences with the mixed case study approach in other
regions are also satisfying, then it could be considered to use it EU
wide as an alternative to the CMEF