Uncertainty mapping for air quality Harmonisation modelling ......, Sam Erik Walker , Sef van den...
Transcript of Uncertainty mapping for air quality Harmonisation modelling ......, Sam Erik Walker , Sef van den...
Norwegian Institutefor Air Research
Harmo11 Conference, July 2007, Cambridge
Uncertainty mapping for air quality Uncertainty mapping for air quality modelling and data assimilationmodelling and data assimilation
Bruce Denby1, Ana Margarida Costa2, Alexandra Monteiro2, Agnes Dudek1, Sam Erik Walker1, Sef van den Elshout3 and
Bernard Fisher4
1 Norwegian Institute for Air Research (NILU), Kjeller, Norway2 CESAM & Department of Environment and Planning, University of Aveiro,
Portugal3 DCMR EPA Rijnmond, Schiedam, The Netherlands
4 Environmental Agency, Reading, UK
6th Framework Programme- Policy oriented Research Priority 8.1 Topic 1.5 Task 2
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
IntroductionIntroduction
IntroductionAim, background and applications
Sources of uncertaintyUncertainty parameters
Model errorProbability distribution functionsStandard deviation and bias
Uncertainty mappingSpatial distribution of model errorIndicative uncertaintyUncertainty related to input dataUsing variance from spatial statisticsUncertainty in exceedancesAccounting for temporal covariance, Monte Carlo methods, probability of exceedance, uncertainty in data assimilation methods, etc…
Conclusions
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
IntroductionIntroduction
AimRaise awareness and promote discussion on uncertainty mapping in air quality applications
BackgroundBased on discussions and developments during the FP6 project Air4EU (www.air4eu.nl)Discussions on uncertainty and its mapping can be found in a number of Air4EU case studies, recommendation documents and cross-cutting issue reportsUncertainty maps are presented on the Air4EU mapping tool (www.air4eumaps.info)
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
IntroductionIntroduction
Why do we NOT use uncertainty maps?Decision makers do not want to know about uncertaintyThere is not enough information for the uncertainty assessmentIt may reflect unfavourably on the models
Why use uncertainty maps? It is at the heart of the scientific method to express uncertainty in any resultIt is honest and transparentProvides information on model quality Better basis for decision making
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Sources of model uncertaintySources of model uncertainty
Model descriptionModel formulation
chemistry, dispersion, etc.
Numerical discretisationnumerical schemes, model resolution, etc.
Input dataEmissions MeteorologyBoundary conditions
Surface, horizontal, etc.
Monitoring data Data assimilation
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
RepresentativenessSpatial representativeness
model resolution, subgrid variability
Temporal representativenessstochastic processes
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty parametersUncertainty parameters
Range of parameters for assessing model error
Mean absolute error MAEMean square error MSENormalised mean square error NMSERoot mean square error RMSERelative percentile error RPE (EU directives)Bias BIASAverage normalised absolute bias ANBFractional bias FBCorrelation R2, rStandard deviation SD Normalised standard deviation NSDRegression coefficients slope, interceptIndex of agreement DFraction of predictions … FAC2
e.g. Chang and Hanna (2004), Borrego et al. (2007)
Model error is an example of poor spatial sampling of the PDF
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty parametersUncertainty parameters
Probability distribution functionsAre used in Bayesian approaches to uncertaintyDescribe the probable model result, given a ‘true’ or observed value (or visa versa)Contain all the uncertainty informationCan be used to derive other statistical parameters
EMEP model PDF. Daily mean PM10 concentrations on the regional scale
-1.5 -1 -0.5 0 0.5 1 1.50
0.2
0.4
0.6
0.8
1
1.2
1.4
Log 10(M/O)
Nor
mal
ised
PD
F
Mode l PDF (EMEP), daily me an PM10, all s tations
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
-1.5 -1 -0.5 0 0.5 1 1.50
0.2
0.4
0.6
0.8
1
1.2
1.4
Log 10 (M/O)
Nor
mal
ised
PD
F
Mode l PDF (AirQUIS), Hourly NO2 at Mangle rud, Os lo
AirQUIS model PDF. Hourly mean NO2 concentrations at a traffic station
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty parametersUncertainty parameters
The use of standard deviation (SD) and biasA PDF is not easy to show in map form, so …Use SD and bias to describe the normal PDFNOTE: PDFs in air quality tend to be log-normal rather than normal
Standard deviation is useful because…It can be statistically extracted from any sampleIt is similar to RMSEIt can by calculated from ensemble methodsIt can be derived from spatial statistical methods such as kriging
Bias must be includedSeparately or implicitly through, e.g., RMSEIf bias is known it should be removed
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty mappingUncertainty mappingExample maps of model error, legislative applications Spatial representation of RPE, Berlin O3 , REM-Calgrid model
RPE interpolated between points
using kriging11 11.5 12 12.5 13 13.5 14 14.5 15
51
51.5
52
52.5
53
53.5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Spatial mapping of RPE for O3(Krigging interpolation method)
Spatial mapping of RPE for O3(no interpolation method)
11 11.5 12 12.5 13 13.5 14 14.5 1551
51.5
52
52.5
53
53.5
0.0-0.1
0.5-0.6
RPE (O3)
0.1-0.2
0.2-0.3
0.3-0.4
0.4-0.5RPE represented by dot sizes
P
PP
OMO
RPE−
=
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty mappingUncertainty mapping
Indicative uncertaintyPresenting maps that are indicative of the standard deviationGeneralised description of the standard deviation based on an absolute and a relative standard deviation
222 ),(),( yxMyx RAM σσσ +=
σR and σA can be calculated using:
Expert knowledge, e.g. σR = 25% ( )
M
OMn
n
iii
R
∑=
−= 1
21
σNormalised RMSE
∑=
⎟⎟⎠
⎞⎜⎜⎝
⎛−
−=n
i i
iR M
On 1
2
)1(11σSD of the PDF
( ) ( ) ( )
( )21 1
2
2
2
2,2/ −
′−
⎥⎥⎦
⎤
⎢⎢⎣
⎡ −+
+=
∑=
− n
MOMM
nnt
n
iii
MSDnM σ
σ αPrediction intervals
Fits to scatter plots
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty mappingUncertainty mapping
Example of an indicative uncertainty mapPrague, annual mean NO2 ,ATEM model
Model
Uncertainty
Based on normalised RMSEσR = 27%σA = 0
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty mappingUncertainty mappingApplication for scenario prediction
Predicted concentrations of PM10 for 2010 in Rotterdam, using the Urbis model
Model prediction
Uncertainty due to emissions
Expert assessment of emission uncertainty in the various source sectors
Spatial map of the emission scenario uncertainty
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty mappingUncertainty mappingSpatial statisticsMaps may also be made using kriging, residual kriging or other statistical interpolation methods
Uncertainty map
),(),( yxVaryxM =σ
SD is calculated in these methods using the kriging variance
Residual kriging of annual mean rural background PM10 using the
EMEP model and Airbase stations
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
Uncertainty mappingUncertainty mapping
Uncertainty in exceedancesUse daily mean uncertainty
probability of exceedanceUse the annual mean uncertainty
spatial representativeness biasCalculate the uncertainty in the number of exceedance days
Number of exceedances, and related uncertainty, of the daily mean EU limit
for PM10 rural background only
Large number of exceedance days (> 60 days)
Large uncertainty in the areas removed from
observations (SD > 35 days)
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
ConclusionsConclusions
General commentsThere is currently no established methodology, parameter or presentation method for communicating uncertaintyIt will be necessary to have a common parameter(s) for representing uncertainty if it is to be useful for intercomparison purposes Different mapping methods use different methodologies for uncertainty assessment
RecommendationsSpatial interpolation of model error is not recommendedSome indication of the uncertainty must always be given, preferably as maps but otherwise as a single valuePresent assessment maps with a contour or colour selection that is indicative of the uncertaintyBias should be removed from maps when it is known
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007
Norwegian Institutefor Air Research
ConclusionsConclusions
Future challengesConvincing air quality modellers to include uncertainty in their maps
- How many presentations of maps at this conference will include uncertainty?
Establishing homogenous and accepted methodologies for determining and presenting spatial uncertaintyIncrease cooperation between atmospheric modellers and spatial statistical groups Inclusion of uncertainty in the entire process, from emissions to risk assessment
- e.g. EU projects HEIMTSA and INTERESSEConvince decision makers that they should want to know about uncertaintyReformulation of directives and other legislation to properly include aspects of uncertainty
IntroductionIntroduction
Sources of Sources of uncertaintyuncertainty
Uncertainty Uncertainty parametersparameters
Uncertainty Uncertainty mappingmapping
ConclusionsConclusions
11th Harmonisation Conference
C
ambridge 2007