AQD IPR pilot study programme on e-Reporting - … · AQD IPR pilot study programme on e-Reporting...
Transcript of AQD IPR pilot study programme on e-Reporting - … · AQD IPR pilot study programme on e-Reporting...
AQD IPR pilot study programme on e-Reporting Up-to-date data flow
Jaume Targa 4th pilot meeting
Copenhagen 15-16 March 2012
European Environment Agency
European Environment Agency
Introduction
• Aim
• Timeline for reporting
• Example from a data provider (data model, schema & codelists)
• Testing phase at EEA
• Q&A
European Environment Agency
Aim
• To continue sharing “near real time” (now, UP-TO-DATE) air quality information across Europe
• UTD data flow is now managed by Article 10 of the 2011/850/EC
• Provision of UTD data + update of UTD following further quality control (Re-submission!)
European Environment Agency
Timeline for reporting AQD dataflow E2
Continuous sharing of UTD
On-going throughout year X, at hourly frequency appropriate to the assessment method
Continuous updates on UTD
European Environment Agency
The AQD dataflow E2
Purpose: Provision for reporting of NRT information exchange for public covering information and alert thresholds
Timeline: On-going throughout year X, at hourly frequency appropriate to the
assessment method Legal instruments: Dir. 2008/50/EC Art. 26 expanding upon EoI decision 97/101/EC
NATIVE
UTD-XML
European Environment Agency
AQD data model for dataset E – primary data
• It is based on the O&M Observation type
• It contains relevant info on measurement data + result values
• It links to both • AQD Process -> methodology used • AQD Feature of Interest -> exact location of measurement
European Environment Agency
AQD data model for dataset E – primary data
Let’s work through an example
European Environment Agency
Country examples - UK
• Aprox. 136 sites
• Stored as MySQL database
• Most stations measure several pollutants
European Environment Agency
• What are we exchanging? Latest 24-48 hours of AQ data!!
• How? Data blocks within XML as…
What information are we sharing?
2012-02-28T13:00:00,1,0,23
European Environment Agency
How is the data encoded?
2012-02-28T13:00:00,1,0,23
• Corresponding “time” of the result • YYYY-MM-DDThh:mm:ss
European Environment Agency
How is the data encoded?
2012-02-28T13:00:00,1,0,23
• Corresponding “time” of the result • YYYY-MM-DDThh:mm:ss
• Validity 1: valid 2: valid, but number replaced by 0.5*detection limit 3: not valid due to station maintenance or calibration not valid or missing 4: not valid or missing
European Environment Agency
How is the data encoded?
2012-02-28T13:00:00,1,0,23
• Corresponding “time” of the result • YYYY-MM-DDThh:mm:ss
• Validity
• Verification
1: verified 2: preliminary verified 3: no verification
European Environment Agency
How is the data encoded?
2012-02-28T13:00:00,1,0,23
• Corresponding “time” of the result • YYYY-MM-DDThh:mm:ss
• Validity
• Verification
• Value
European Environment Agency European Environment Agency
RE
SU
LTS
2012-02-28T13:00:00,1,0,23
What does it look like in the XML?
Number of hourly values included for this sampling
point
European Environment Agency European Environment Agency
RE
SU
LTS
2012-02-28T13:00:00,1,0,23
What does it look like in the XML?
Number of hourly values included for this sampling
point
• What else? -> how do we tell where the measurement was taken!
European Environment Agency
AQD data model for dataset D - Station
• Provider (data type ‘Contact Details’) • Change documentation • Zone code • Zone name • Zone type • Zone delimitation (data type ‘Spatial Extent’) • Zone history: application start and end date • Predecessors (link to B) • Resident population • Resident population reference year • Code of designated pollutant • Protection Target • Exemption or postponement according to Article 22 of Directive 2008/50/EC
European Environment Agency European Environment Agency
Unique process ID
Unique Sampling Point + Unique feature of interest (INLET)
Number of hourly values included for this sampling
point
RE
SU
LTS
2012-02-28T13:00:00,1,0,23
European Environment Agency
Time period of data included
TimeInstant -> time file has been generated
European Environment Agency
Unique process ID
Unique Sampling Point + Unique feature of interest (INLET)
Number of hourly values included for this sampling
point
RE
SU
LTS
2012-02-28T13:00:00,1,0,23
European Environment Agency
General view on XML
European Environment Agency
XML file including observations from different sampling points (measured at several monitoring stations)
Observation gml:id uses the EoI code where SamplingPoint (Inlet) belongs to
The station name is followed by a simple sequence to diferenciate between different Processes (i.e. Instruments)
In this example, GB is using a combination of EoI code and number for the unique id required for gml:id
European Environment Agency
Interim xlink conventions
• In the pilot testing programme there will be a need for interim xlink conventions
• Metadata cited by xlink conventions in one data flow may not exist elsewhere (yet) e.g. within UTD data flow
</om:parameter><om:observedProperty xlink:href="#PM10"/><om:featureOfInterest xlink:href="SampleFeature_GB0729A_sp2"/>
• So in this case you will need to refer to the gml:id and location where you expect it to be when you have finished the job
Doesn’t exist yet !
European Environment Agency
Interim xlink conventions - assumptions
• FeatureOfInterest SampleFeature_EoICode_INLET sp1 = gases inlet sp2 = PM10 automatic inlet sp3 = PM2.5 automatic inlet
• Process Process_UniqueValue process_6 = PM10 TEOM FDMS process_9 = NO2-NOx (general) process_5 = O3
(No. of processes will increase as GB meta data is re-structured)
PROCESS ID need to be re-used when necessary (i.e. same measurement config. will use same ID)
European Environment Agency
Interim xlink conventions - assumptions
• SamplingPoint SamplingPoint_EoICode_INLET sp1 = gases inlet sp2 = PM10 automatic inlet sp3 = PM2.5 automatic inlet
European Environment Agency
Naming conventions proposals for UTD
• Interim file naming conventions for the pilot
• Data flow deliveries should be prepared for with a 3-part notation
• Using this convention
dataFlowSuffix_countryCode_genericTimeReference.xml
E2a_GB_2012022912.xml - relates to UTD dataflow for the 24-hour period specified
European Environment Agency
Transformation mechanism
• TB??
Communication
• Extraction of AQ data from stations
Database
• MySQL
AQD schema
• Load to AQD.xsd with GOPublisher
Transmission
• FTP upload of UTD data
European Environment Agency
Testing phase at EEA
UTD XML delivered by pilot MS
Operational delivery via FTP
Upload to database (EEA)
Perform QA/QC checks & calculate initial statistics (EEA)
Publish via AQ viewers (EEA)
European Environment Agency
Common issues – issues to discuss
• Metadata cited by xlink DO NOT EXIST
• Current database not structured in relation to Dataset D
• For current NRT, we only use the EoI code to allocate station
European Environment Agency
Important issues – issues to discuss
• Agreement of generation of gml:id
• Need to agree common field and record separator
• Agreement on generating Process_IDs ?? Useful question from BE: the Process_ID for NO2 is 9, for PM10 it is 6. What are the id's for the other pollutants ?