Model based approach for estimating and forecasting crop statistics:

Update, consolidation and improvement of AGROMET model

“AGROMET Project”Working Group Meeting on Crop Statistics

October 25th, Luxembourg

DevStat

Agenda

1. Introduction: Project Goals and current AGROMET model.

2. Solutions: a new model.A. Key features and ImprovementsB. Imputation of missing dataC. Model selectionD. Model validationE. Set-Aside effect of A

3. Results and examples.

4. Conclusions and comments.

DevStat

Project goals1. Introduction

• To provide forecasts for crops, vegetables and fruit production on every country member at the EU, based on historical data on both observables provided to Eurostat periodically: harvest production (H) and arable area (A).

• To generate a SAS application for the statistical

analysis, and use the SAS Enterprise Guide to create its interface.

• To generate documentation on the application and form users.

DevStat

Project Implementation1. Introduction

• DEVSTAT and UMH-CIO are the members of the consortium providing services to Eurostat under the Framework contract number 6001. S008.001-2009.065.

• Within this framework contract, the consortium received the request for services ESTAT E0/24 for the updating, consolidation and improvement of current AGROMET model, according to the goals stated in the previous slide.

• This presentation shows the main features of the project and the current state of the development and implementation of the new model.

DevStat

Current Agromet Model: main features

• Eurostat has been using the AGROMET MODEL, programmed within FAME, to provide forecasts for areas, yield and production on crops, on twelve of the countries at the EU, and also on its aggregate.

• Yield is calculated by dividing the observable vaiables Production and Area.

• Estimates of Yield and Area are produced based on the last 10 years, by using:• Linear regression • Quadratic regression • ARIMA(1,1,1)

• Production estimate is obtained by multiplying Yield and Area estimates.

1. Introduction

DevStat

Current Agromet Model: limitationsThe current AGROMET model:• Constraints forecasts from fixed estimation models for each

product, whatever the country and prediction year.• Constraints some forecasts to predictions based on lines or

parables, also assuming independency of the historical available data series (inconsistent assumption on time correlated data series)

• Excludes forecasting in all these cases (combinations product-country) with 2 or more missing values in the data series from the last 10 years.

• Focuses on forecasting the non-observable variable YIELD to provide forecasts on the observable HARVEST PRODUCTION.

• Does not provide forecasting error measures.• Only covers crops on 12 from the 27 current UE members.

1. Introduction

DevStat

Key features of the new model2. Solutions

• HARVEST PRODUCTION (H) and AREA (A) are the relevant observable variables in the analysis and model fitting.

• Prediction data: 10 years time series. IMPROVEMENTS

• An Imputation procedure for missing data values has been developed.

• Trend models reasonably substitute fixed Agromet models.• Relationship between the observables A and H is used for H

prediction whenever possible. • Inclusion of an automatic model selection criteria for different

fitted models.• Error measures are provided on the forecasts.• Set-aside specifications on A can be used to predict the effect

on H.

DevStat

Imputation procedure

1) Regression/Inverse regression of H on A.

2) Moving Average Estimate of lag 1

2. Solutions

DevStat

Imputation procedure2. Solutions

HARVEST

AREA

1) Regression/Inverse regression of H on A.

DevStat

Imputation procedure2. Solutions

2) Moving Average Estimate of lag 1

DevStat

Imputation results2. Solutions

DevStat

Model especification

Case 1

Case 2

Case 3

4 or more NA data Average of last 3 years

3 or less NA data ARIMA (p,d,q)

no NA data on H and A

A: ARIMA (p,d,q)Regression models forH on A with ARMA (p,q) errors

2. Solutions

DevStat

Model selection

• Automatic selection rule for the best model among the ARIMA(p,d,q) and ARMA(p,q) models.

• Flexible and optimal adaptation to data, whatever the product, country and behaviour along time.

• Intrinsic validation mechanism: the best model provides minimum deviation between observed and estimated.

These facts are distinctive improvements with respect to the previous AGROMET model.

2. Solutions

DevStat

Validation

The validation index VAL is defined as the “percentage of cases whose confidence or credible interval does get the observed value at the prediction year”. This validation index is assessed at two levels:

• COUNTRY-TYPE-PREDICTION YEAR (country VAL index)

• PRODUCT-PREDICTION YEAR (product VAL index)

2. Solutions

DevStat

Set-aside on Area2. Solutions

In order to produce forecasts of HARVEST under land restrictions, two possibilities are considered:

• When a regression model is available, to predict HARVEST based in the regression model, with AREA, as explanatory variable.

• If not, to predict HARVEST on a proportional basis, by applying the same percentage of reduction for arable land.

DevStat

Interface in SAS E-G3. Results

Selection of Country

DevStat

Interface in SAS E-G3. Results

Selection of Type of product

DevStat

Interface in SAS E-G3. Results

Selection of product

DevStat

Interface in SAS E-G3. Results

Selection of Prediction year

DevStat

Interface in SAS E-G3. Results

Selection of Variable to predict

DevStat

Estimation results: example case 1

Case 1 4 or more NA data on A and H Average of last 3 years

3. Results

DevStat

Estimation results: example case 2

Case 2 3 or less NA data on A and H H: ARIMA (p,d,q)A: ARIMA (p,d,q)

3. Results

DevStat

Estimation results: example case 3

Case 3 no NA data on A and HARIMA(p,d,q) on ARegression models for H on A with ARMA (p,q) errors

3. Results

DevStat

Validation Results: product VAL index3. Results

DevStat

Validation Results: country VAL index3. Results

DevStat

Conclusions4. Conclusions

The AGROMET model has been improved:• It uses more appropriate time series models• It adapts the available data, whatever the

country, product or prediction year, always with the optimal model selected.

• It provides forecasting errors.• Its interface allows for selection on type of

product, product, country, prediction year, set-aside.

• It accommodates a set-aside restriction on A• It has been programmed under SAS Enterprise

Guide.

DevStat

Comments4. Conclusions

THANKS FOR YOUR ATTENTION