Approach To and Findings From Farming Practices Survey

October 17, 2008

Approach To and Findings From Farming Practices Survey

October 17, 2008

Overview of Presentation Overview of Presentation

I. How an Impact Evaluation Works

II. Evaluation Design for Water-to-Market Training

III. About the Farming Practices Survey

IV. Findings from the Farming Practices Survey

I. How an Impact Evaluation Works

II. Evaluation Design for Water-to-Market Training

III. About the Farming Practices Survey

IV. Findings from the Farming Practices Survey

Impact EvaluationImpact Evaluation

Was the program effective?

Estimating impacts involves comparing: – Outcomes with the program with outcomes if there

were no program

Counterfactual: What participants would have experienced if there were no program– True counterfactual is not directly observed

Goal of impact study is to identify a comparison group to approximate the counterfactual

Was the program effective?

Estimating impacts involves comparing: – Outcomes with the program with outcomes if there

were no program

Counterfactual: What participants would have experienced if there were no program– True counterfactual is not directly observed

Goal of impact study is to identify a comparison group to approximate the counterfactual

Importance of the Counterfactual:An Illustrative Example

Importance of the Counterfactual:An Illustrative Example

Rural poverty has decreased in recent years, prior to MCA-Armenia programs

In this case, we want to see how much more did poverty decline because of MCA-Armenia?

Rural poverty has decreased in recent years, prior to MCA-Armenia programs

In this case, we want to see how much more did poverty decline because of MCA-Armenia?

How Much More Did Poverty Decline Because of MCA-Armenia?

How Much More Did Poverty Decline Because of MCA-Armenia?

Poverty With and Without MCA

2005 2006 2007 2008 2009 2010

Po

ve

rty

Ra

te

Impact

Random Assignment: Often Considered the “Gold Standard”

Random Assignment: Often Considered the “Gold Standard”

Randomly assign into two groups, similar to a lottery

Program and comparison groups are the same on average, except one group has access to program – Subsequent differences in outcomes can be attributed

to the program

Random assignment most feasible when there are limited resources/excess demand

Randomly assign into two groups, similar to a lottery

Program and comparison groups are the same on average, except one group has access to program – Subsequent differences in outcomes can be attributed

to the program

Random assignment most feasible when there are limited resources/excess demand

II. Evaluation for Water-to-Market Training

II. Evaluation for Water-to-Market Training

Water-to-Market ActivitiesWater-to-Market Activities

Focus on evaluation of training programs– Water management techniques– Transition to higher value agriculture

ACDI/VOCA providing training – Primarily in Compact Years 2-5

Focus on evaluation of training programs– Water management techniques– Transition to higher value agriculture

ACDI/VOCA providing training – Primarily in Compact Years 2-5

Evaluation Design for Water-to-Market Training

Evaluation Design for Water-to-Market Training

Ideally, would want to randomly assign farmers who apply for training

Not practically feasible or politically viable– New program starting, so no excess demand

– Intervention at the village level (potential for

spillover/contamination)

We randomly assigned villages to program or “control” group– Use a phased-in approach

Ideally, would want to randomly assign farmers who apply for training

Not practically feasible or politically viable– New program starting, so no excess demand

– Intervention at the village level (potential for

spillover/contamination)

We randomly assigned villages to program or “control” group– Use a phased-in approach

Shortcomings of the Approach Shortcomings of the Approach

Less statistical power to detect impacts than random assignment at the individual level – Not all farmers in a village may want to participate in

training – Clustering effects (village specific effects)

Challenge can be overcome to some extent if:– Take-up rates of training high – More villages included

Less statistical power to detect impacts than random assignment at the individual level – Not all farmers in a village may want to participate in

training – Clustering effects (village specific effects)

Challenge can be overcome to some extent if:– Take-up rates of training high – More villages included

Random Assignment Design Random Assignment Design

Randomly assign when training will start in each village, with three groups:– Compact Year 2 – Compact Years 3 and 4– Compact Year 5

Compare Compact Year 2 villages to Compact Year 5 villages

Randomly assign when training will start in each village, with three groups:– Compact Year 2 – Compact Years 3 and 4– Compact Year 5

Compare Compact Year 2 villages to Compact Year 5 villages

Random Selection of Villages (Implemented in August 2007) Random Selection of Villages

(Implemented in August 2007)

Started with a list of villages with “good water” and which could benefit from training – Excluded villages served in pilot phase

Random selection conducted publicly to: – Ensure transparency – Allow for greater accountability and

implementation fidelity

Started with a list of villages with “good water” and which could benefit from training – Excluded villages served in pilot phase

Random selection conducted publicly to: – Ensure transparency – Allow for greater accountability and

implementation fidelity

Selection of Villages Selection of Villages

Total of 277 village clusters were assigned– Year 2: 120 clusters– Years 3 and 4: 77 clusters– Year 5: 80 clusters

Selection of clusters stratified by WUAs– In proportion to the number of villages in the WUA

To ensure balance across treatment and control villages – Also for political reasons

Total of 277 village clusters were assigned– Year 2: 120 clusters– Years 3 and 4: 77 clusters– Year 5: 80 clusters

Selection of clusters stratified by WUAs– In proportion to the number of villages in the WUA

To ensure balance across treatment and control villages – Also for political reasons

Selection of Villages (cont’d) Selection of Villages (cont’d)

No. of Villages By Year of Training and Ag. Zone No. of Villages By Year of Training and Ag. Zone All Zones Ararat Valley Pre-Mountainous Mountainous Subtropical

Year 2: treatment 120 44 58 12 6Years 3 and 4: nonresearch 77 18 19 38 2Year 5: control 80 28 38 10 4Total 277 90 115 60 12

Overall Approach to the Impact Evaluation

Overall Approach to the Impact Evaluation

Compare outcomes for farmers in treatment and control villages

Examine impacts on key outcomes– Participation in training and adoption of new practices– Changing crop patterns, improved yields, and

increases in income

Data and Sample – Survey of farmers in the treatment and control

villages– Sample of 5,000 farmers to detect impact on poverty

of 5 percentage points

Compare outcomes for farmers in treatment and control villages

Examine impacts on key outcomes– Participation in training and adoption of new practices– Changing crop patterns, improved yields, and

increases in income

Data and Sample – Survey of farmers in the treatment and control

villages– Sample of 5,000 farmers to detect impact on poverty

of 5 percentage points

About the Farming Practices Survey

About the Farming Practices Survey

Farming Practices Survey Farming Practices Survey

Baseline survey conducted by AREG

Implemented during Nov 2007- Feb 2008 – In 223 communities– Target sample of 5,000 farmers

Topics covered in the survey – Demographic characteristics– Agricultural practices and productivity – Income and consumption

Baseline survey conducted by AREG

Implemented during Nov 2007- Feb 2008 – In 223 communities– Target sample of 5,000 farmers

Topics covered in the survey – Demographic characteristics– Agricultural practices and productivity – Income and consumption

Identifying the Sample Frame Identifying the Sample Frame

Sample frame is critical as it defines what population the study represents – Need comparable frame for treatment and control

villages– Want persons who farm as their main occupation

No viable sample frame exists for the surveys – Originally attempted to use WUA member lists– Worked with WUA heads to get names of farmers – During pretest, AREG found some lists were bad – All lists had to be verified by AREG in the field and

developed as necessary

Sample frame is critical as it defines what population the study represents – Need comparable frame for treatment and control

villages– Want persons who farm as their main occupation

No viable sample frame exists for the surveys – Originally attempted to use WUA member lists– Worked with WUA heads to get names of farmers – During pretest, AREG found some lists were bad – All lists had to be verified by AREG in the field and

developed as necessary

Implementing the Survey and Checking Data Quality

Implementing the Survey and Checking Data Quality

Interactive interviewer training (Q by Q, etc)

Piloting of both survey instrument and sample list – Revisions to questionnaire– Sample verification process

Regular generation of field reports

MCA did field visits to review processes

Independent review of initial batches of data

Interactive interviewer training (Q by Q, etc)

Piloting of both survey instrument and sample list – Revisions to questionnaire– Sample verification process

Regular generation of field reports

MCA did field visits to review processes

Independent review of initial batches of data

Planned Improvements to the Upcoming Survey

Planned Improvements to the Upcoming Survey

More use of photos in training interviewers

on farming practices

Pretesting both survey instrument and sample list

Double key data entry

More use of photos in training interviewers

on farming practices

Pretesting both survey instrument and sample list

Double key data entry

Baseline FPS Survey Baseline FPS Survey

Completed interviews with 4,855 farmers

Sample focuses on households farming for five years or more in these communities

Sample not representative of all farmers in Armenia

However, sample internally valid for study– Will show comparison of farmers in treatment

and control villages later

Completed interviews with 4,855 farmers

Sample focuses on households farming for five years or more in these communities

Sample not representative of all farmers in Armenia

However, sample internally valid for study– Will show comparison of farmers in treatment

and control villages later

IV. Baseline FindingsIV. Baseline Findings

Farmer and Household Characteristics

Crop Cultivation and Sales

Income and Poverty

Farmer and Household Characteristics

Crop Cultivation and Sales

Income and Poverty

General Approach to Analysis and Data Cleaning

General Approach to Analysis and Data Cleaning

Descriptive analysis – Averages and distributions– Examined conditional and unconditional measures– Used weights to adjust for treatment-control balance

Data checks – Missing values and outliers – Logical skips – Internal consistency

Measuring income and consumption is challenging – Sensitivity tests – Looking at distributions critical

Descriptive analysis – Averages and distributions– Examined conditional and unconditional measures– Used weights to adjust for treatment-control balance

Data checks – Missing values and outliers – Logical skips – Internal consistency

Measuring income and consumption is challenging – Sensitivity tests – Looking at distributions critical

Farmer and Household Characteristics

Farmer and Household Characteristics

Farmer and Household Characteristics

Farmer and Household Characteristics

Characteristics (Percentages) Characteristics (Percentages)

<40 18.7 40–49 35.5 50–59 27.7 60 or older 18.1 (Average) (49.2)

Less than secondary 13.6 Secondary 40.8 Secondary (vocational) 28.0 More than secondary 17.6Female Respondent 11.9Household Size 5.3 Children 1.4

Respondent’s Age

Respondent’s Education

Farmer and Household Characteristics (cont’d)Farmer and Household Characteristics (cont’d)

Land Cultivated by Respondents (Sq. Meters) Land Cultivated by Respondents (Sq. Meters) All Zones Ararat Valley Pre-Mountainous Mountainous Subtropical

Average 19,846 14,274 22,445 30,776 23,638Median 10,000 7,000 14,000 20,000 13,000

Average 1,722 1,700 1,630 2,377 1,247Median 1,200 1,200 1,200 2,000 1,000

Area of Land Cultivated

Size of Kitchen Plot

Few Farmers Use Water Management Practices

Few Farmers Use Water Management Practices

Respondents’ Irrigation Practices (Percentages)Respondents’ Irrigation Practices (Percentages)

Furrow row spacing 7.4

Scientific scheduling 0.1

Water meters 0.0

Non-pressure/pipe irrigation 0.6

Pressure irrigation 0.3

Respondents Using:

Crop Cultivation and SalesCrop Cultivation and Sales

Crop Cultivation and SalesCrop Cultivation and Sales

Respondents Growing and Selling Crops (Overall Percentages)Respondents Growing and Selling Crops (Overall Percentages)

Respondents Growing

Respondents Selling

Grains 44.6 7.0Fruits 77.0 36.4Vegetables 32.4 16.5Herbs 5.1 2.1Potatoes 25.7 5.8Nuts 6.3 0.7Grass 38.1 6.6Other 13.8 4.7

Crop Cultivation and Sales Vary by Agricultural Zone

Crop Cultivation and Sales Vary by Agricultural Zone

• Farmers in Ararat Valley grow more fruit, and much more likely to sell their fruits and vegetables

• Farmers in the Mountainous Zone grow much more grain and potatoes

• Mountainous Zone farmers sell very little

• Farmers in Ararat Valley grow more fruit, and much more likely to sell their fruits and vegetables

• Farmers in the Mountainous Zone grow much more grain and potatoes

• Mountainous Zone farmers sell very little

Income and PovertyIncome and Poverty

How to Measure Agricultural Income?How to Measure Agricultural Income?

• Crop sales tell us monetary income

• Much of production is consumed by household; not reflected in sales

• Assign monetary value to crops consumed by household

• Subtract agricultural costs to get profit

• Crop sales tell us monetary income

• Much of production is consumed by household; not reflected in sales

• Assign monetary value to crops consumed by household

• Subtract agricultural costs to get profit

Crop Sales and Value of ProductionCrop Sales and Value of Production

Respondents’ Average Crop Sales and Values (AMD)Respondents’ Average Crop Sales and Values (AMD)Sales Value (Production x Price)

Grains 27,136 110,458Fruits 261,839 307,462Vegetables 73,214 126,970Herbs 9,585 12,020Potatoes 34,653 62,010Nuts 590 3,534Grass 18,745 63,852Other 23,785 32,702Total 449,822 707,133

Measuring IncomeMeasuring Income

• Survey is relatively short (30 minutes)

• Focus on agricultural production, but learn about non-agricultural income/consumption as well

• Income measures include employment income, pensions, remittances, other benefits

• Consumption measures include groceries, household products, utilities, transportation

• Survey is relatively short (30 minutes)

• Focus on agricultural production, but learn about non-agricultural income/consumption as well

• Income measures include employment income, pensions, remittances, other benefits

• Consumption measures include groceries, household products, utilities, transportation

Household Income, Only Considering Monetary Income

Household Income, Only Considering Monetary Income

Annual Household Monetary Income (AMD)Annual Household Monetary Income (AMD)

Monetary agricultural profit (crop sales – costs)

Total Monetary Income

344,000

165,983

818,520

461,930 300,000

627,913

Nonagricultural income

Monetary agricultural profit (crop sales – costs)

Total Monetary Income

-35,950

Method 1 (Monthly Salary x 12) MedianAverage

384,000652,538

-35,950

451,000

Nonagricultural income

Method 2 (Monthly Salary x 6) Average Median

165,983

Household Income, Including Value of Crops Consumed

Household Income, Including Value of Crops Consumed

Annual Household Economic Income (AMD)Annual Household Economic Income (AMD)

113,667

885,224 522,577

423,293

Method 1 (Monthly Salary x 12) Average

Total Economic Income

461,930

Total Economic Income

Nonagricultural income

Nonagricultural income

Economic agricultural profit (crop value – costs)

Economic agricultural profit (crop value – costs)

300,000

Method 2 (Monthly Salary x 6) Average

652,000

Median

652,538 384,000

423,293 113,667

Median

1,075,831

Measuring PovertyMeasuring Poverty

• Consumption-based measure

• Adopt similar approach as NSS/World Bank

• Food poverty line: Meets minimal caloric needs

• Complete poverty line: Allowance for other basic necessities

• Consumption-based measure

• Adopt similar approach as NSS/World Bank

• Food poverty line: Meets minimal caloric needs

• Complete poverty line: Allowance for other basic necessities

Estimates of Poverty RatesEstimates of Poverty Rates

Respondent Households in Poverty (Percentages)Respondent Households in Poverty (Percentages) Food

PovertyComplete Poverty

Excluding consumption of own crop production 11.5 26.1Including consumption of own crop production 7.5 18.3Average household consumption relative to poverty line 349 237

Poverty Varies Appreciably by ZonePoverty Varies Appreciably by Zone

Respondent Households in Poverty by Zone (Percentages) Respondent Households in Poverty by Zone (Percentages)

0

5

10

15

20

25

30

Ararat Valley Pre-Mountainous Mountainous Subtropical

Food Poverty

Complete Poverty

0

5

10

15

20

25

30

Ararat Valley Pre-Mountainous Mountainous Subtropical

Food Poverty

Complete Poverty

Many Households are Near Poverty Line

Many Households are Near Poverty Line

Consumption Relative to Complete Poverty Line (CPL) Consumption Relative to Complete Poverty Line (CPL)

Including own crop consumption

0

5

10

15

20

25

30

35

40

45

Below CPL 1-2 timesCPL

2-3 timesCPL

3-4 timesCPL

4-5 timesCPL

5 or moretimes CPL

Including own crop consumption

0

5

10

15

20

25

30

35

40

45

Below CPL 1-2 timesCPL

2-3 timesCPL

3-4 timesCPL

4-5 timesCPL

5 or moretimes CPL

Values of Variables byTreatment Status

Values of Variables byTreatment Status

Demographic Characteristics and Land Size of Sample Members

Demographic Characteristics and Land Size of Sample Members

Treatment Control

Respondent’s Age (years) 49.2 49.2 Female Respondent 13.0 10.8 Respondents Education

Less than Secondary 13.8 13.4Full Secondary 40.6 41.1Secondary Vocational 28.9 27.1More than Secondary 16.7 18.4

Area of land cultivated (ha) 1.99 1.98Kitchen plot size (ha) .171 .173

Use varying furrow spacing 7.5 7.4

Crop Cultivation and Sales and Income for Sample Members Crop Cultivation and Sales and Income for Sample Members

Treatment Control

Percent Cultivating Grain 45.7 43.5Fruit 76.3 77.6 Vegetable 33.1

31.7

Revenue from Crops Sold (AMD) Grain 21,711 32,531Fruit 317,619 206,355 Vegetable 88,858

57,653

Market Value of Harvest Grain 112,784 108,144 Fruit 332,365 282,692 Vegetable 110,924

142,928

Household Income and Poverty Household Income and Poverty

Treatment Control

Household Income (AMD) Nonagricultural Income 674,780 630,414 Total Monetary Income 929,592 708,041** Total Economic Income 1,115,105 1,036,766

PovertyComplete Poverty Rate 17.9% 18.7% Consumption Relative to CPL 238% 237%

Lessons Learned and Plans for the Future

Lessons Learned and Plans for the Future

Modifications to the questionnaire – More detailed measure of employment income– Consistent units for crop production

Enhancements to data collection procedures – Use pictures of irrigation practices – Focus on high response rates

More focus on longitudinal analysis – Useful to factor out preexisting differences

Final impact report (2011) – Possibility of interim report

Modifications to the questionnaire – More detailed measure of employment income– Consistent units for crop production

Enhancements to data collection procedures – Use pictures of irrigation practices – Focus on high response rates

More focus on longitudinal analysis – Useful to factor out preexisting differences

Final impact report (2011) – Possibility of interim report