A QA/QC protocol for modelling applications

P. Thunis & L. Tarrasón

Berlin, February 2020

Current situation

No prerequisites on the actual model to be used as long as the model complies with the

Modelling Quality Objectives (MQO)

FAIRMODE &

CEN TC264 / WG43

Is this sufficient to ensure quality of modelling applications?

1. Do you have a documented protocol (or set of consistency checks) that you currently apply to

your modelling chain (emissions, meteorology, air quality modelling...) to ensure that the final

assessments are of sufficient quality? If yes, what type of protocol (or actions) do you

apply?

2. According to you, which are the key elements that are currently missing (or elements where

harmonization is highly needed) that should be considered in priority in this protocol?

3. To support the elaboration of a common QA/QC protocol, we plan to use the composite

mapping platform as a main tool to identify the main issues, identify the inconsistencies and

assess if the proposed solutions are efficient. In this context, do you have proposals on the

suggested usage of this platform?

Questionnaire sent to NCP (summer 2019)

1. What type of protocol (or actions) do you apply?

• Some protocol exist for very specific situations (e.g. industrial plant) but are not general enough

• Lots of statistical analysis and plots but no real documented protocol

• Model inter-comparison whenever possible

• QA/QC procedures (e.g. checks on emissions) are existing but not a real protocol

• Input data are fixed by law

2. Which are the key elements that are currently missing?

• Summary performance plots and statistics

• Checks on emissions

• Checks on meteorology

• Structuring and automation of the process

3. Use of the composite mapping platform – looking for suggestions…

Responses to the questionnaire

1. What content for the protocol? (Documentation and/or assessment oriented?)

2. Which process for elaborating the protocol? (collect best practices?)

Outcome: A draft QA/QC protocol to be prepared.

It should outline best practices for QA/QC of modelling applications with an initial

scope on assessment purposes (forecast and planning will be considered at a

later stage)

Madrid discussions and outcome (autumn 2019)

Draft document

• For which pollutants? NO2, PM and O3

• For which scales? All relevant to the AAQD

• Models? All types (Gaussian, Eulerian, Lagrangian,

CDF…) relevant to the AAQD

Scope (assessment purposes)

Documentation (overall)

• Model details

• Input data

• Intended applications

• Evaluation databases

• Code verification

• …

Documentation (purpose)

• Model resolution

• Model calibration

• Data requirement

• …

Evaluation

(QA/QC tests)

• Performance evaluation

• Sensitivity analysis

• Input data analysis

• Model inter-comparison

• Benchmark tests

• …

Scope: ensuring quality through documentation and evaluation tests

Overall aspects Specific aspects of implementations

Modelling results originate from fit-for-purpose models

We will follow the recommendations from CEN TC264/43 (reflected in

the FAIRMODE guidance document on “MQO and benchmarking”) on

specific aspects of the evaluation process. E.g. which measurements to

use? how frequent should the assessment be?, what to do when

sufficient measurements are not available?...

We will apply the station classification and station types currently

proposed by the IPR for evaluation (QA/QC tests)

Requisites and assumptions

Three main sources:

1. Documentation required in the context of the “Implementing Provisions

for reporting” mechanism

2. Documentation filed within the former “Model Documentation System”

3. National / regional / local protocols

a. French audit for the AASQAs (presented by F. Tognet in Madrid)

b. Any other available source of information?

Documentation

Composite mapping platform

MDS descriptive topics Sub-topics

Basic information

Model name

Model version and status

Latest date of revision

Contact information

Level of knowledge needed to operate model

Intended field of application

Model type and dimension

Model description summary

Model limitations/approximations

Resolution

Temporal resolution

Horizontal resolution

Vertical resolution

Schemes Advection & Convection

Turbulence

Deposition

Chemistry

Solution technique

Input Availability and Validation of Input data

Emissions

Meteorology

Topography

Initial condition

Boundary conditions

Data assimilation options

Output quantities

User interface availability

User community

Previous applications

Documentation status Ranking levels 1 – 5

Validation and evaluationRanking levels 1 – 5

Model inter-comparison

Frequently asked questions

Extract from Model Documentation SystemExtract from French AASQA audit

Documentation (examples)

• All sources of documentation to be merged

according to most relevant and link to

composite mapping

Documentation examples IPR and composite mapping

Main sources:

1. Outcome of the Madrid’s discussions

2. QA/QC protocol available from modelling groups (currently only from the

CHIMERE group!)

QA/QC proposed evaluation process

Main QA/QC protocol steps:

STEP 1: Passing the Modelling Quality objective (MQO)

STEP 2: How well is my model capturing the spatial variability?

- Fulfilling CEN spatial Model Performance Criteria (MPC)

- Incremental assessment (rural vs. urban vs. street)

STEP 3: How well is my model capturing the temporal variability?

For each station type, assess the seasonal, day/night and week/week-

end behavior when appropriate

STEP 4: Input consistency (meteorology, emissions…)

STEP 5: Ex-post assessment (repeat step 1 on multiple cases + delta)

STEP 6: Multi-pollutants

QA/QC proposed evaluation (main steps)

MQI(i) =Oi −Mi

)β𝑈(Oi

FAIRMODE QA/QC evaluation (indicators)

STEP 2: Spatial indicators STEP 3: Temporal indicatorsSTEP 1: MQO

FAIRMODE QA/QC evaluation (indicators)

STEP 5: Ex-post assessment STEP 6: Multi-pollutants aspectsSTEP 4: Input consistency

Emissions

Meteorology

Bound. Cond.

Learn from the

compared behavior of

different pollutants

• Is the proposed approach to QA/QC modelling protocol appropriate? Is

there something missing?

• Documentation - What actions should be put in place for identifying the

most appropriate documentation requests? Should FAIRMODE use the

composite mapping metadata for modelling as a test? How do you

recommend to link to the IPR?

• Evaluation tests – Are the proposed tests adequate? Is there something

missing? Should FAIRMODE initiate a series of feasibility exercises for

the evaluation tests to be included in the protocol?

Discussion in plenary

• Outcome from today’s plenary discussion to be incorporated in the draft

protocol

• First draft document of the QA/QC protocol to be circulated by March

2020 -

• Review from FAIRMODE participants suggestion on documentation,

QA/QC tests… June 2020

• Selection of documentation requirements Next technical

• Selection of evaluation tests based on information received Next

technical

Timing and follow-up

Thank you

© European Union 2020

Unless otherwise noted the reuse of this presentation is authorised under the CC BY 4.0 license. For any use or reproduction of elements that are not owned by the

EU, permission may need to be sought directly from the respective right holders.

Slide xx: element concerned, source: e.g. Fotolia.com; Slide xx: element concerned, source: e.g. iStock.com

https://creativecommons.org/licenses/by/4.0/