1

Agricultural Policies Evaluation Area

MGAP strategy forEvaluation of Agricultural Policies

Bureau of Programming and Agricultural Policy (OPYPA)Ministry of Livestock, Agriculture and Fisheries (MGAP)

URUGUAY

2

Schedule

I. Institutionalizing agriculture policies evaluation at MGAP

a) Why evaluate at MGAP? b) Building internal capabilitiesc) Evaluation Scope and methodologies.

II. Evaluation agenda prioritized by authoritiesa) Progress in evaluation strategyb) Information System for evaluation.

III. Lessons learned

3

The MGAP is making progress in institutionalizing evaluation of policies and programs, with emphasis on impact assessment and evaluation of results.

Why?

a) Why evaluate at MGAP?

Need evidence on what works► Justification for program ($$)► Scale up or not – did it work?► Compare different policy options

11

Improve program/policy ► Design (eligibility, benefits)► Implementation (efficiency & targeting)

22

Be accountable and transparent to the people (stakeholders) for whom we are working 33

Impact evaluation for evidence based policy making

4

b) Building internal capabilities

► Agricultural Policies Evaluation Area has a specialized technical team.▪ Evaluators are responsible for the technical aspects: the evaluation

methodology, sampling design, data collection, analysis, and reports.

► Institutionally, the Agricultural Policies Evaluation Area is located within OPYPA but is external to the implementing programs units

5

b) Building internal capabilities

Why internal Evaluations?

Advantages Disadvantages

● To build a "culture of evaluation"

● Evaluation needs a deep understanding of the theory of change and the context.

● Evaluation reports and policy recommendations would be more accepted if they came from inside.

Challenges to maintain objectivity and credibility.

Agreement OPYPA-AGEV: Key partner who gives technical guarantees on the independence of evaluation (an external view).

6

c) Evaluation Scope

What was the effect of the program on outcomes? (before-after, no causality)

How much better off are the beneficiaries because of the program/policy (attribution)?

Is the program cost-effective? Traditional

Monitoring

cannot answer these.

7

c) Methodologies

► Quantitative methods: Matching, Difference-in-Difference, Instrumental variables, Regression discontinuity design, others.

►Qualitative methods: including semi-structured interviews and focus group discussions

► A mix of methods Do what works best in

the context.Be flexible, be

creative.

8

Schedule

I. Institutionalizing agriculture policies evaluation at MGAP

a) Why evaluate at MGAP? b) Building internal capabilitiesc) Evaluation Scope and methodologies.

II. Evaluation agenda prioritized by authoritiesa) Progress in evaluation strategyb) Information System for evaluation.

III. Lessons learned

9

1. Adaptation of production systems to climate variability▪Building resilience to climate change and variability in vulnerable smallholders (GFCC)

▪Water for Animal Production (APA)

▪Agricultural insurance

2. Rural development: competitive insertion of family farming in value chains

▪Institutional Strengthening Plans (PFI)

▪Family, Integral and Sustainable Production (PFIS)

a) Progress in evaluation strategy at MGAP

Baseline

Final Report

Baseline

Design

10

•1 • Idea•2

Impact Assessment

APA (Efects medium and

long term)

Inputs for Design and Redesign

Results Evaluation PFI

(End of implementation)

Baseline GFCC-PFIS

(start of implementation)

Process Evaluation DID-AGEV

(during implementation)

Contexto

a) Progress in evaluation strategy

11

National Level

National Level

Beneficiary Beneficiary

Producers /Rural organizations

Producers /Rural organizations

Project Level

Project Level

Farms Level

Farms Level

Programs application forms DB

National Agricultural Census 2011

b) Information system

ApplicantsApplicants

Data panel SNIG-Dicose

GFCC SurveyPFIS Survey

Family Farmers Registration DB

PFI Survey and Qualitative data

12

Schedule

I. Institutionalizing agriculture policies evaluation at MGAP

a) Why evaluate in MGAP? b) Building internal capabilitiesc) Evaluation Scope and methodologies.

II. Evaluation agenda prioritized by authoritiesa) Progress in evaluation strategyb) Information System for evaluation.

III. Lessons learned

13

1. It is important to plan for impact (the ‘start point’). Some changes in MGAP interventions would result in better evaluation and strengthen impact.

►Encourage the development of an explicit Theory of Change, measure target population and indicators.

►Embed impact assessments in organizational learning and planning.

►Ensure input from policy makers and implementers at all stages of design and development of impact assessment processes.

Lessons learned

14

2. MGAP information systems need further refinements to maximize the use of administrative data and statistics, which have great potential in Uruguay and are less expensive than surveys.

3. It is convenient to deploy both quantitative and qualitative assessments to better understand causes and effects, which leads to better policy recommendations. Main challenge: develop an adequate “methodology toolbox”.

4. Focus on producing inputs for program design and redesign. Use findings widely and creatively for improved impact of impact assessments!

Lessons learned

15