Importing and translating crop model data

C. H. Porter, C. Villalobos University of Florida

CIMMYT (Nepal) – March 18-22, 2013

ICRISAT (India) - March 25-29, 2013

AgMIP IT Products

• Websites

– Public site www.agmip.org

– Research site research.agmip.org Documentation of Tools

– AgMIP toolshed tools.agmip.org Download of Tools

– AgMIP connect site connect.agmip.org

– AgMIP data: data.agmip.org (under construction)

– Open source repositories github.org/agmip

• Listservs

– http://lists.agmip.org/cgi-bin/mailman/listinfo/agmip

– http://lists.agmip.org/cgi-bin/mailman/listinfo/software-releases

– http://lists.agmip.org/cgi-bin/mailman/listinfo

• Databases

– Crop Experiment / Sentinel site (operational)

– Crop model outputs (operational)

– Crop scenarios (operational)

– Gridded data (conceptual)

Web-based data search

• http://data.agmip.org

AgMIP harmonized databases

• Model data stored in a

consistent format

• Filters allow import of data

from various model-ready

files and other data sources

• Translators allow export of

data to various model-ready

formats

The DOME:

Data Overlay for Multi-model Export

Fill in missing data prior to data translation so that all models use the same assumptions. Examples:

• Planting densities

• Initial soil conditions

Overlay a management regimen, for applying RAPs to baseline conditions. Examples:

• Fertilizer rates

• Cultivars

ACE-ACMO-DOME Desktop utilities

• Historical simulations (survey data) – Typically single year

– Field measured data plus field_overlay DOME

• Baseline seasonal simulations – Multi-year using baseline conditions

– Add Baseline_seasonal_stragegy DOME to create multi-year simulations with automatic planting dates.

• Future scenario seasonal simulations – Multi-year using future weather and (optional) adaptation

– Use Future_seasonal_strategy DOME to create multi-year simulations

– Adaptation consistent with RAPs applied through DOME

– Future weather data

7

3 Types of Simulations for

Integrated Assessments

Steps to data preparation and

translation

1. Gather, assemble and enter field data.

2. Harmonize field, soils and weather data using Survey_data_import template.

3. Apply field assumptions in the field_overlay template using best knowledge of agronomic practices in the region.

4. Save all files in csv format using the ADA utility.

5. Use QuadUI desktop application for data translation to crop model formats.

6. Run models for historical conditions.

7. View error and log reports if simulations were not successful.

8. Use ACMOUI to combine output data with metadata in ACMO*.csv files for input to economic models.

9. Compute bias using simulated and observed yield data.

Repeat steps 2 – 9 as needed to adequately characterize field production under survey conditions.

Survey Data Simulations

Steps to data preparation and

translation

1. Use a seasonal_strategy template to create a multi-season simulation for baseline conditions.

2. Save all files in csv format using ADA utility.

3. Use QuadUI desktop application for data translation to crop model formats.

4. Run models for baseline multi-year conditions.

5. View error and log reports

Repeat steps 1 – 5 as needed to generate simulated results for multi-year simulations.

6. Use ACMOUI to combine output data with metadata in ACMO*.csv files for input to economic models

Baseline Seasonal Strategy Simulations

Future Scenario Seasonal Strategy Simulations

Repeat the steps for a baseline seasonal strategy simulation, except that future

climate weather data are used and management adaptation strategies may be

applied using the seasonal_strategy template, consistent with RAPs.

Step 1 – Collect raw data

Use a spreadsheet to convert units to AgMIP formats.

Step 2 – Prepare crop model data

• Two ways to input data:

– Reverse translators from model data currently

available for DSSAT only. We are working on

an APSIM reverse translator.

– Use spreadsheet templates to fill in data with

proper ICASA variable names and units.

• multiple templates will allow different kinds of data

– User Interface for data entry (future)

Step 2 – Survey data template

(continued)

• Excel spreadsheet for calculations,

conversions, date formats

– Field, soils and weather data

• Example: Survey_data_import.xlsx

• One line per site for metadata, field, and

management data

• Separate sheet for soil data

• Separate sheet for weather data

Step 2 – Survey data template

Machakos data as an example: (Survey_data_import.xls)

ICASA Master Variable List

• Harmonized database uses ICASA Master Variable List to interpret data definitions and units.

• See ICASA_Master_Variables_10Sep2012.xlsx or online (always up-to-date) at https://docs.google.com/spreadsheet/pub?key=0AiUhvGN3ZVgXdGJud3lDNThFSnlGWGlkTjgxWXRwbWc&output=html

– living document – go to web version for the latest

– adding variables to spreadsheet templates: use code_display column from ICASA list

• Recent copy in Data directory on USB drive.

Step 3 – Apply field overlay

• Excel spreadsheet templates for applying

DOME data

– Field Overlay

• Example: Field_Overlay.xlsx

• Separate sheet for each field overlay

• Supplies data which are missing from field data

Step 3 - DOME template

Step 4 - Save csv format files

• Use ADA to create separate csv file for each data type: – soil

– weather

– management / field / metadata (initial conditions, observed data)

– Field overlay

– Seasonal strategies

• ADA versions: – ADA_classic for Excel 2003, 2007

– ADA_2010 for Excel 2010

• Beware of date formats when you re-open a csv file with Excel!

Step 5 - QuadUI

– Desktop application for conversion of spreadsheet templates to AgMIP harmonized format (JSON), APSIM or DSSAT formats.

– If DOME data are included, these are applied to the baseline data

– Near future: web-based application to upload and archive data, search and query, download in multiple formats.

Step 5 – QuadUI (continued)

• Start by double-clicking

the quadui.bat file.

• Select the zip file

containing field, soils

and weather data.

• Select the field overlay

file(s), zipped, if multiple

files.

• Select the seasonal

strategy file(s)

• Select the location for

output files.

Step 5 – QuadUI (continued)

• Select format(s) for conversion • DSSAT

• APSIM

• JSON

• Currently working on additional translators for:

• STICS

• Aquacrop

• InfoCrop

• SarraH

• EPIC

• CropSyst

• Salus

• WOFOST

• RZWQM

• ORYZA

• Hit “Convert” button

Step 6 – Run Models

• DSSAT – run.bat file provided which will

run the batch of experiments created.

• APSIM – double-click the .APSIM file to

run experiments through the APSIM UI.

Step 7 – QuadUI

troubleshooting

• Look at crop model files for notes made by the translators

• Look at QuadUI log file (where the QuadUI jar file is located) for notes on invalid variables or DOME functions

• Application requires that headers are spelled exactly as in code_display of ICASA Master Variable List (not case-sensitive)

• Check list of changes in upcoming slides. The REG_ID name is a common mistake.

• Variables must appear in correct file (e.g., no weather data in soils file)

Step 8 - Crop model outputs

Use ACMOUI to create ACMO_model.csv file which can be used for analysis, aggregation, and input to economic models.

Steps 9 – Compute bias

• 𝐵𝑖𝑎𝑠 = 𝑚𝑒𝑎𝑛 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑦𝑖𝑒𝑙𝑑

𝑚𝑒𝑎𝑛 𝑠𝑖𝑚𝑢𝑙𝑎𝑡𝑒𝑑 𝑦𝑖𝑒𝑙𝑑

• There are now two types of DOME files:

– field_overlay is used to fill in data that

required to run the crop models, but were not

recorded in the field.

– seasonal_strategy is used to create

hypothetical simulations for multiple years and

/ or imposed management regimens

• Multiple field_overlay files may be used for

management data or field metadata (but

not for soils or weather data).

25

Important changes since the

regional workshops!

• New or updated tools:

– QuadUI has been updated to allow two

different kinds of DOME files

– ACMOUI reads model output data and

combines with simulation metadata to

produce the ACMO_model.csv file, which is

used by the economists.

– ADA utility helps convert data from

spreadsheet format to csv and zip files.

26

Important changes since the

regional workshops!

• Variable and function names: – In the INFO section at the top of all DOME files, the

variable “Region” should be “REG_ID” for the economists to recognize the data. This is critical, as it prevents old templates from working!

– OM_TOT – total amount of organic matter added in a season (previous template incorrectly used OMAMT).

– function ROOT_DIST() was previously labeled ROOT_DIS in template.

– PLDP - planting depth should be entered in mm, not cm, per the ICASA definitions.

– FL_LOC1, FL_LOC2, FL_LOC3 should be added to the field overlay data for country name, sub-country name and local name.

– ENDAT variable added for end of simulation date.

27

Important changes since the

regional workshops!

• New DOME functions: – TAVAMP() - calculates the average annual temperature

and temperature amplitude for a weather station.

– ICN_DIST() - distributes a user input value of kg/ha mineral Nitrogen through a soil profile as a constant concentration

– AUTO_PDATE() - Automatic planting algorithm for seasonal strategies

• New DOME operator: – REPLACE_FIELD_ONLY - used for CO2Y, the static

atmospheric CO2 value. For historical simulations, the value should be the actual values for the year of the experiment. For seasonal strategy simulations, the value should be constant at the midpoint of the time slice being simulated.

28

Important changes since the

regional workshops!