MeteoIO

A meteo data I/O library

2

Before MeteoIO...

• Each application doing its IO– Insufficient robustness– IO functions spread out all over the code– Input data very strictly formatted, several

steps in preparing the data– Hard to adapt to new protocols/formats

→IO a source of weakness...

3

MeteoIO: goals• Remove all IO functions out of the

physics engine• Make IO easy to use• Make IO robust• Make IO flexible• Greatly simplify data preparation• Allow unmonitored, automatic operation• Allow end user to fine tune his IO...

4

MeteoIO: ini files[Parameters]

DEMSRC = ARC

DEMFILE = ../input/surface-grids/zwischberg.dem

LANDUSESRC = GRASS

LANDUSEFILE = ../input/surface-grids/zwischberg.lus

METEOSRC = A3D

METEOPATH = ../input/meteo

SPECIALPTSSRC = A3D

SPECIALPTSFILE = ../input/surface-grids/zwischberg.pts

OUTPUT = ARC

OUTPATH = ../output/

plugins

5

MeteoIO: primary data structures

• MeteoData:– Date object

– Ta

– Iswr

– Vw

– Rh

– Lwr

– Nswc

– Ts0

– P

– ...

– Supports equality comparison

• StationData:– Longitude

– Latitude

– Altitude

– Northing (metric grid)

– Easting (metric grid)

– Name

– Supports equality comparison

6

MeteoIO: example of use

7

What is it doing?

• Parsing io.ini

• Loading plugins supporting requested data sources

• Reading data (across files, network, protocols and supporting comments, different EOL, etc)

• Converts stations' coordinates if necessary

• Filtering data, marking invalid data as “no data”

• Conversion of units to SI

8

Example: output

[i] meteoio/A3DIO.cc:86: Loading dynamic plugins:

[i] Number of 2D meteo stations: 15

---------- Station: 1 / 16

Name:

Air Temperature: 266.25

---------- Station: 2 / 16

Name: ANET-ZER

Air Temperature: 266.45

---------- Station: 3 / 16

Name: ANET-VIS

Air Temperature: 268.75

(etc)

This station has no name

9

Data sources

• Currently supported:– Meteo data: Alpine3D's Ascii, Boschung's

XML, Gsn, Imis Oracle database, GeoTop– Grids: ARC, GRASS

10

Writing a new IO plugin• Provide implementation of some or all of:ReadMeteoData (const Date_IO& dateStart, const Date_IO& dateEnd,

std::vector< std::vector<MeteoData> >& vecMeteo, std::vector< std::vector<StationData> >& vecStation, const unsigned int& stationindex=IOUtils::npos)

Read2DGrid (Grid2DObject& grid_out, const string& parameter="")

ReadDEM (DEMObject& dem_out)

ReadLanduse (Grid2DObject& landuse_out)

ReadAssimilationData (const Date_IO& date_in, Grid2DObject& da_out)

ReadSpecialPoints (CSpecialPTSArray& pts)

Write2DGrid (const Grid2DObject& grid_in, const string& name="")

11

Filters• Available:

– Min, max– Max observed changed– No observed change

• To be added:– Median Average Deviation

• Special processing– Resampling– accumulation

• Easy to add new filters

12

MeteoIO: example of use II

13

What does it change?

• IO are now buffered -> efficient on slow access

• Automatic resampling for missing data– Using linear interpolation between two

neighbors– To be added: cyclic interpolation, etc

14

Userland additionIn the user's code: add a function checking

that the input requirements are met(ie: nb of lwr, nb of ta, etc)

15

Data structures/methods

• Array, Array2D, Array3D– Using vectors for a safe and efficient

implementation

• Grid2DObject, Grid3DObject– Geolocalized array

• DEMObject– Computes slope, aspect, normal's

components (Corripio, Hicks, etc)

16

MeteoIO: example of use III

17

MeteoIO: example of use IV

18

TA, 0 to 16° RH, 62% to 90%

Gotthard, 2008­10­01 12:00, 19 stations

19

Spatial interpolations• Available methods:

– Std pressure

– Constant value

– Constant with lapse rate

– IDW

– IDW with lapse rate

– For Rh, Td IDWK with lapse rate and back to Rh

– Terrain influenced VW and DW

• One method defined for only 1 data input, one for multiple data input (fallback)

• Regression:– Linear regression for the moment, removes the worst data point if the

regression is too poor

20

Spatial interpolations 2

Planned additions:– Physical wind interpolation (quick Bernoulli

resolution?)– More regression models (dynamically

adjusted to the data)– Smarter choice of which method to use

(dynamic, including data quality)

21

Geographic projections• Conversion to/from WGS84 from any

coordinate system– Through own implementation (CH1903, ...)– Through proj4 library

• Conversion to/from local grid MapProj local_proj(lat_ref, long_ref);

local_proj.convert_from_WGS84(latitude, longitude, easting, northing);

22

MeteoIO: on the side• Date handling (Date_IO class)• Config files handling (ConfigReader

class)• Data storage (Arrays & grids classes,

sparse matrix (coming))• Ported to Pop-C++ for parallel execution• Application specific IOs classes

23

Thank you!Mathias Bavay (SLF)

Thomas Egger (Egger Consulting)Laurent Winkler (HES-SO)

Florian Hof (SLF)Moustapha Mbengue (SLF)

Gael Rosset (HES-SO)Julien Jeanneret (HES-SO)

Nora Helbig (SLF)