Ecmwf observations data handling

WMO BUFR training course Langen 17-20 April 2007

Slide 1

Slide 1

Ecmwf observations data handling

Milan Dragosavac


Slide 2

Slide 2

MDB RDB PDBFDB

ODB

IFS

MARS

General applications

Analysis Forecast

DisplayPrintPlot

Data transport

Datatransport

Data acquisition


Slide 3

Slide 3

Data acquisition

Acquire data

Process data

Archive data ( not MARS)

Outgoing ( send data outside Ecmwf)


Slide 4

Slide 4

Acquisition process

GTS data

ftp

Other data


Slide 5

Slide 5

Acquisition outgoing

Qscat satellite data in BUFR form

SSMI brightness temperatures

TOVS 1B calibrated and converted into BUFR format


Slide 6

Slide 6

MEB feed

Restores any old data

Join segmented bulletins

Write bulletins in MEB

statistics


Slide 7

Slide 7

Message data base

“STREAM”_YYYYMMDD

AIRC_20070216

AIRS_20070216

AMAP_20070216

GRIB_20070216

75 streams in total

One stream per day


Slide 8

Slide 8

Message data base attributes

TIME character( 4,1)

OBSTYPE character( 4,1)

TTAAii character( 6,1)

BBB character( 3,1)

CCCC character( 4,1)

_SOURCE character(80,1)

_CHECKSUM longinteger

_TIMESTAMP character(20,1)


Slide 9

Slide 9

Message data base attributes continued

_data_size longinteger

_bulk_size longinteger

_owner character( 3,1)

_last_access time(0)

_access_number longinteger

_compress character( 1,1)

_data bulk(20,0,512,1)


Slide 10

Slide 10

Message data base indices

UNIQUE I_timestamp_SYNO_20070215 on (_TIMESTAMP)

UNIQUE I_comp_SYNO_20070215 on (TIME, TTAAii, BBB, _CHECKSUM)

NORMAL (2,62) I_OBSTYPE_SYNO_20070215 on (OBSTYPE)


Slide 11

Slide 11

Message data base

100 Gbytes allocated + 150 Gbytes for METOP data

Over 22 Gbytes / day

Saved on ECFS according to OBSTYPE

All data can be reprocessed if Data base is lost

HP cluster 4 CPU 64 bit processors

4 operational machines + 1 for development

Raid disks 1.2 Tbytes


Slide 12

Slide 12

MEB data arrival

Monitoring arrival times of different data types

Analysis cycle

Average number of days


Slide 13

Slide 13

Message data base data availability

Data availability according to synoptic hour


Slide 14

Slide 14

MEB x-interface

Allows MEB search

Find corresponding BUFR observation in REB

Expand BUFR and GRIB data

Write data into file


Slide 15

Slide 15

MEB x-interface

REB BUFR observations corresponding to MEB bulletin


Slide 16

Slide 16

Preprocessing suite

MEB – message data base feed task

deqc – decoding, quality control and BUFR creation, writing into REB

ex00 – extraction for analysis

bc00 – extraction for boundary condition suite

wmo – ftp WMO station list


Slide 17

Slide 17

Decoding

Decoding

Quality control

BUFR creation

Global grib fields creation

Writing into REB


Slide 18

Slide 18

REB extraction for analysis

14 files created

Conventional observations

Satellite data

Sea surface temperatures

Ice analysis


Slide 19

Slide 19

REB – report data base

One table per observation type per day

AIRC_20070215

ATOV_20070215

B002_20070215

CREX_20070215

TEMP_20070215

SSBT_20070215

QSCA_20070215

statistic


Slide 20

Slide 20

REB attributes

TIME character(6,1)

IDENT character(9,1)

_BUFR_TYPE integer

_BUFR_SUBTYPE integer

_LATITUDE1 float(6)

_LONGITUDE1 float(6)

_COR1 integer

_QC integer


Slide 21

Slide 21

REB attributes continued

_TIMESTAMP character(20,1)

NR_SUBSETS integer

_data_size longinteger

_bulk_size longinteger

_owner character(3,1)

_last_access time(0)

_access_number longinteger

_data bulk(20,0,512,1)


Slide 22

Slide 22

REB indices

UNIQUE I_ts_SYNO_20070215 on (_TIMESTAMP)

UNIQUE I_comp_SYNO_20070215 on (TIME, IDENT, _BUFR_SUBTYPE, _LATITUDE1, _LONGITUDE1

NORMAL (2,15) I_extract_SYNO_20070215 on (TIME, _BUFR_SUBTYPE, IDENT)


Slide 23

Slide 23

REB reports data base

Just bellow 22 Gbytes / day

100 Gbytes allocated + 150 Gbytes for METOP data

All data can be re-decoded at any time

Data can be accessed through MARS request and Metview

Archived in MARS


Slide 24

Slide 24

REB x-interface

Search REB

Find original bulletin in MEB

Expand BUFR and GRIB data

Write data to file

Access WMO station list data base


Slide 25

Slide 25

BUFR x-interface

Expands BUFR message

Any subset can be viewed

Filtering features


Slide 26

Slide 26

WMO Data Base

WMO_a_master

WMO_a_master_manual

WMO_a_master_tape


Slide 27

Slide 27

WMO_a_master attributes

Region_country integer not null

Block smallint not null

Station_index integer not null

Station_subindex smallint not null

Principal _station character(1,1)

Station_name text(20,20,30,1)

Latitude real

Longitude real


Slide 28

Slide 28

WMO_a_master attributes continued

Pressure_elevation integer

Pressure_elevation_flag character(1,1)

Ground_elevation integer

Ground_elevation_flag character(1,1)

Pressure_level smallint

Surf_prog_00 character(2,1)






Slide 29

Slide 29

WMO_a_master attributes continued




Ua_type_00 smallint

Ua_type_06 smallint

Ua_type_12 smallint

Ua_type_18 smallint

Modify_flag character(1,1) not null


Slide 30

Slide 30

WMO_a_master indices

UNIQUE I_wmo_a_master ON (block,station_index, station_subindex)


Slide 31

Slide 31

WMO station list


Slide 32

Slide 32

ODB Observational data base

Interface to analysis

Data base contains unpacked bufr data and original bufr data

Loading done in parallel with load balance achieved on supercomputer due to bufr features

Analysis updates ODB

Analysis bufr feedback files created

Archiving of feedback files in MARS


Slide 33

Slide 33

Loading ODB with observations

Tasks running in parallel

2-16 processors used per task

Load balancing achieved


Slide 34

Slide 34

ODB to BUFR

odb2bufr family


Slide 35

Slide 35

Odbviewer


Slide 36

Slide 36

Odbviewer


Slide 37

Slide 37

Odbviewer


Slide 38

Slide 38

Odbviewer


Slide 39

Slide 39

Odbviewer


Slide 40

Slide 40

Odbviewer


Slide 41

Slide 41

Odbviewer


Slide 42

Slide 42

Future

Sharp increase in volume of observations expected

IASI data with 8641 channels per observation

Data bases have crucial role in observations handling

Binary data representation allows easy handling of observations as bulk objects

Ecmwf observations data handling

Documents

Transcript of Ecmwf observations data handling