Product Validation Report - Servizio...
Transcript of Product Validation Report - Servizio...
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 1
Products Validation Programme
Italian Meteorological Service Italian Department of Civil Defence
H-SAF Product Validation Report (PVR)
PR-ASS-1 - Instantaneous and accumulated precipitation at ground computed by a NWP model
Zentralanstalt für Meteorologie und
Geodynamik
Vienna University of Technology Institut für Photogrammetrie
und Fernerkundung
Royal Meteorological Institute of Belgium
European Centre for Medium-Range Weather Forecasts
Finnish Meteorological Institute
Finnish Environment Institute
Helsinki University of Technology
Météo-France CNRS Laboratoire Atmosphères, Milieux, Observations Spatiales
CNRS Centre d'Etudes Spatiales de la BIOsphere
Bundesanstalt für Gewässerkunde
Hungarian Meteorological Service
CNR - Istituto Scienze dell’Atmosfera
e del Clima Università di Ferrara
Institute of Meteorology and Water Management
Romania National Meteorological Administration
Slovak Hydro-Meteorological Institute
Turkish State Meteorological Service
Middle East Technical University
Istanbul Technical University
Anadolu University
30 May 2010
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 2
H-SAF Product Validation Report PVR-06
Product PR-ASS-1
Instantaneous and accumulated precipitation at ground computed by a NWP model
INDEX
Page
Acronyms [not including those in the Appendix] 06
1. The EUMETSAT Satellite Application Facilities and H-SAF 08
2. Introduction to product PR-ASS-1 09
2.1 Principle of the product 08
2.2 Algorithm principle 09
2.3 Main operational characteristics 10
3. Validation strategy, methods and tools 11
3.1 Validation team and work plan 11
3.2 Validation philosophy 11
3.2.1 Objectives and problems 11
3.2.2 Tools to be used for validation 12
3.2.3 Techniques to bring observations comparable 13
3.2.4 Structuring the results of the validation activity 14
3.3 Definition of statistical scores 16
3.4 Inventory of validation facilities 18
3.4.1 Facilities in Belgium (IRM) 18
3.4.2 Facilities in Germany (BfG) 22
3.4.3 Facilities in Hungary (OMSZ) 24
3.4.4 Facilities in Italy (UniFe) 27
3.4.5 Facilities in Poland (IMWM) 29
3.4.6 Facilities in Slovakia (SHMÚ) 31
3.4.7 Facilities in Turkey (ITU) 35
4. Validation of the product release as at the end of the Development Phase 39
4.1 Introduction 39
4.2 Validation in Belgium (IRM) 40
4.3 Validation in Germany (BfG) 42
4.4 Validation in Hungary (OMSZ) 48
4.5 Validation in Italy (UniFe) 55
4.6 Validation in Poland (IMWM) 58
4.7 Validation in Slovakia (SHMÚ) 63
4.8 Validation in Turkey (ITU) 66
5. Overview of findings 69
5.1 Synopsis of validation results 69
5.2 Summary conclusions on comparative elements 72
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 3
Appendix - Collection of validation experiment reports [low-level of editing] [4 pages] 75
Yellow: statistical analyses over several months Blue: case studies in specified few days
§ Validation experiments on PR-ASS-1 Period Institute
2. Contribution not expected (exclusive task of CNMCA) Belgium/IRM
3. Contribution not expected (exclusive task of CNMCA) Germany/BfG
4. Contribution not expected (exclusive task of CNMCA) Hungary/OMSZ
5.1 Half-year statistics September 2008 - February 2009 Italy/CNMCA
5.2 Round-year statistics December 2008 - November 2009 Italy/CNMCA
6. Contribution not expected (exclusive task of CNMCA) Poland/IMWG
7. Contribution not expected (exclusive task of CNMCA) Slovakia/SHMÚ
8. Contribution not expected (exclusive task of CNMCA) Turkey/ITU
List of Tables
Table 01 - List of H-SAF products 08
Table 02 - Validation Team for precipitation products 11
Table 03 - List of ground data used for precipitation products validation in Belgium 18
Table 04 - Precipitation data available at BfG 22
Table 05 - Location of the 16 meteorological radar of the DWD 22
Table 06 - Characteristics of the three meteorological Doppler radars in Hungary 24
Table 07 - Accuracy requirements for product PR-ASS-1 [RMSE (%)] 39
Table 08 - Summary results of PR-ASS-1 validation in Belgium by IMR 40
Table 09 - Summary results of the validation of PR-ASS-1rate in Germany by BfG 42
Table 10 - Summary results of the validation of PR-ASS-1accumulated in Germany by BfG 44
Table 11 - Summary results of the validation of PR-ASS-1rate in Hungary by OMSZ 48
Table 12 - Summary results of the validation of PR-ASS-1accumulated in Hungary by OMSZ 51
Table 13 - Summary results of the validation of PR-ASS-1rate in Italy by UniFe 55
Table 14 - Summary results of the validation of PR-ASS-1accumulated in Italy by UniFe 55
Table 15 - Summary results of the validation of PR-ASS-1rate in Poland by IMWM 58
Table 16 - Summary results of the validation of PR-ASS-1accumulated in Poland by IMWM 58
Table 17 - Summary results of the validation of PR-ASS-1rate in Slovakia by SHMÚ 63
Table 18 - Summary results of the validation of PR-ASS-1accumulated in Slovakia by SHMÚ 64
Table 19 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS
over inner land
66
Table 20 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS
over coastal zones
66
Table 21 - Comparative results of validation in several Countries/Teams split by season. PR-
ASS-1 00 h (rain rate)
70
Table 22 - Comparative results of validation in several Countries/Teams split by season. PR-
ASS-1 accumulated precip.
71
Table 23 - Simplified compliance analysis for product PR-ASS-1 73
Table 24 - Synthesis of all validation results, including yearly average 74
List of Figures
Fig. 01 - Conceptual scheme of the EUMETSAT application ground segment 08
Fig. 02 - Current composition of the EUMETSAT SAF network (in order of establishment) 08
Fig. 03 - COSMO-ME model formulation and current area coverage 09
Fig. 04 - Cloud microphysical processes considered in the two-category ice scheme 10
Fig. 05 - Structure of the Precipitation products validation team 11
Fig. 06 - The network of 4100 rain gauges used for H-SAF precipitation products validation 12
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 4
Fig. 07 - The network of 40 C-band radar used for H-SAF precipitation products validation 13
Fig. 08 - Classes and sub-classes for evaluating Precipitation Rate products. Applicable to PR-
OBS-1, PR-OBS-2, PR-OBS-3, PR-OBS-4 and PR-ASS-1rate
15
Fig. 09 - Classes and sub-classes for evaluating Accumulated Precipitation products. Applicable
to PR-OBS-5 and PR-ASS-1accumulated
16
Fig. 10 - Meteorological radar in Belgium 18
Fig. 11 - RMI raingauges: daily ( ) and AWS ( ) 19
Fig. 12 - SETHY AWS network in Walloon Region 19
Fig. 13 - Left: Gaussian filter; right: sketch of the up-scaling procedure. The circle corresponds to
the range of the weather radar. The square in the middle is a common area such that it is
entirely included in the selected PR-OBS-2 files. The grey rectangle, the tilted dark grey
rectangle and the black ellipse are explained in the text
20
Fig. 14 - Left panel: radar coverage in Germany as of 01/03/2007. Right panel: location of
ombrometers for online calibration in RADOLAN; squares: hourly data provision
(about 500), circles: event-based hourly data provision (about 800 stations): red: AMDA
III, blue: aggregational network federal states (Bartels et al., 2004)
23
Fig. 15 - Flowchart of online calibration RADOLAN (adapted from Bartels et al. 2004) 23
Fig. 16 - The automatic rain gauge network in Hungary 24
Fig. 17 - Location and coverage of the three meteorological Doppler radars in Hungary 24
Fig. 18 - Map of SHMÚ raingauge stations: green – operational (98) , blue – climatological
(586), red - hydrological stations in H-SAF selected test basins (37). White points show
regular grid of experimental NOAA Snow water equivalent data
31
Fig. 19 - Example of Slovak radar network coverage - left circle corresponds to radar site Malý
Javorník, right one corresponds to Kojšovská hoľa 31
Fig. 20 - Example of 5-days cumulative precipitation constructed from raingauge measurements
by means of 3D extrapolation method 32
Fig. 21 - Map of SHMÚ hydrological stations: red - hydrological stations in H-SAF selected test
basins (34) 33
Fig. 22 - General hydrological model of HBV type 33 Fig. 23 - Example of pick up pixels from radar measurement for selected area 34 Fig. 24 - Example of separated ALADIN forecast pixel to smaller pixels 34 Fig. 25 - The Susurluk and Western Black Sea catchments selected for the precipitation product
validation in Turkey 35
Fig. 26 - Network of Meteorological stations in Susurluk (on the left) and Western Black Sea (on
the right) catchments 36
Fig. 27 - Position of 193 AWOS sites used for ground truth for the precipitation product
validation in western Turkey 36
Fig. 28 - H02 product footprint centres with a sample footprint area as well as the AWOS ground
observation sites 37
Fig. 29 - Meshed structure of the sample H02 product footprint 38
Fig. 30 - Mean Error of PR-ASS-1, 24h accumulated precipitation (monthly values over Belgium
in mm day-1) 40
Fig. 31 - Root Mean Square Error of PR-ASS-1, 24 accumulated precipitation (monthly values
over Belgium in mm day-1) 41
Fig. 32 - Time evolution of Mean Error and Standard Deviation for the three precipitation
categories 42
Fig. 33 - Time evolution of the Root Mean Square Error (mm/h and %) for the three precipitation
categories 43
Fig. 34 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success
Index 43
Fig. 35 - Time evolution of the Mean Error, function of the accumulation period 44
Fig. 36 - Time evolution of the Standard Deviation, function of the accumulation period 45
Fig. 37 - Time evolution of the Root Mean Square Error (mm), function of the accumulation 45
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 5
period
Fig. 38 - Time evolution of the Root Mean Square Error (%), function of the accumulation period 45
Fig. 39 - Time evolution of the Correlation Coefficient, function of the accumulation period 46
Fig. 40 - Time evolution of the Probability of Detection, function of the accumulation period 46
Fig. 41 - Time evolution of the False Alarm Rate, function of the accumulation period 46
Fig. 42 - Time evolution of the Critical Success Index, function of the accumulation period 47
Fig. 43 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for
medium and light precipitation 48
Fig. 44 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for
heavy precipitation 49
Fig. 45 - Time evolution of the Correlation Coefficient for the three categories of precipitation 49
Fig. 46 - Time evolution of Probability of Detection, False Alarm Rate and Critical Success
Index for rain rate 50
Fig. 47 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for the
integration intervals 3, 6, 12 and 24 hours 52
Fig. 48 - Time evolution of the Correlation Coefficient for the integration intervals 3, 6, 12 and
24 hours 52
Fig. 49 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success
Index for integration intervals 3, 6, 12 and 24 hours (four panels) 54
Fig. 50 - Probability Density Function for instantaneous rain rate and accumulation periods 3 h
and 24 h for the two months January and July 2009, for satellite-derived and ground
measurements
56
Fig. 51 - Time evolution of continuous statistical scores for the precipitation intensity 57
Fig. 52 - Time evolution of the Correlation Coefficient for instantaneous rain rate and integration
intervals 3 and 24 hours 57
Fig. 53 - Mean error (ME) of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar
2010 for Poland 59
Fig. 54 - RMSE % of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar 2010
for Poland 59
Fig. 55 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-
ASS-1 v.1.0 rain rate product using Polish RG data in the period of Jan 2009 – Mar
2010
60
Fig. 56 - Mean error (ME) of PR-ASS-1 3 h cumulated precipitation v.1.0 for the period of
Jan 2009 – Mar 2010 for Poland 61
Fig. 57 - RMSE (%) of PR-ASS-1 3-hour cumulated precipitation v.1.0 for the period of
Jan 2009 – Mar 2010 for Poland 61
Fig. 58 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-
ASS-1 3-hour cumulated precipitation v.1.0 product using Polish RG data for the period
of Jan 2009 – Mar 2010
62
Fig. 59 - Time evolution of Mean Error, Root Mean Square Error (%) and dichotomous scores for
rain rate 63
Fig. 60 - Time evolution of Mean Error and Root Mean Square Error (%), function of the
integration interval 3, 6, 12 and 24 h 65
Fig. 61 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success
Index, function of the integration interval (3, 6, 12 and 24 h) 65
Fig. 62 - Continuous and multi-categorical statistics for inner land (3 and 24 hourly
accumulation) 67
Fig. 63 - Continuous and multi-categorical statistics for coastal zones (3 and 24 hourly
accumulation) 67
Fig. 64 - Correlation coefficients for inner land and coastal zones (3 and 24 hourly accumulation) 67
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 6
Acronyms [not including those in the Appendix]
ACC Fraction correct Accuracy
AMSU-B Advanced Microwave Sounding Unit - B (on NOAA up to 17)
ATDD Algorithms Theoretical Definition Document
AU Anadolu University (in Turkey)
AWOS Automated Weather Observing Stations
AWS Automatic Weather Station
BfG Bundesanstalt für Gewässerkunde (in Germany)
CAF Central Application Facility (of EUMETSAT)
CC Correlation Coefficient
CESBIO Centre d'Etudes Spatiales de la BIOsphere (of CNRS, in France)
CM-SAF SAF on Climate Monitoring
CNMCA Centro Nazionale di Meteorologia e Climatologia Aeronautica (in Italy)
CNR Consiglio Nazionale delle Ricerche (of Italy)
CNRS Centre Nationale de la Recherche Scientifique (of France)
COSMO Consortium for Small-Scale Modelling
COSMO-ME Consortium for Small-Scale Modelling - version for Mediterranean
CSI Critical Success Index
DPC Dipartimento Protezione Civile (of Italy)
DWD Deutscher Wetterdienst
DWR Dry to Wet Ratio
ECMWF European Centre for Medium-range Weather Forecasts
ETS Equitable Threat Score
EUM Short for EUMETSAT
EUMETSAT European Organisation for the Exploitation of Meteorological Satellites
FAR False Alarm Rate
FBI Frequency BIas
FMI Finnish Meteorological Institute
GEO Geostationary Earth Orbit
GIS Geographic Information System
GPM Global Precipitation Measurement mission
GRAS-SAF SAF on GRAS Meteorology
HAWK Hungarian Advanced Weather Workstation
HBV Hydrologiska Byrans Vattenbalansavdelning (hydrological model)
HRON Hydrological model for the Hron basin
H-SAF SAF on Support to Operational Hydrology and Water Management
HSS Heidke skill score
IFOV Instantaneous Field Of View
IMWM Institute of Meteorology and Water Management (in Poland)
IPF Institut für Photogrammetrie und Fernerkundung (of TU-Wien, in Austria)
IR Infra Red
IRM Institut Royal Météorologique (of Belgium) (alternative of RMI)
ISAC Istituto di Scienze dell‟Atmosfera e del Clima (of CNR, Italy)
ITU İstanbul Technical University (in Turkey)
LATMOS Laboratoire Atmosphères, Milieux, Observations Spatiales (of CNRS, in France)
LEO Low Earth Orbit
LSA-SAF SAF on Land Surface Analysis
ME Mean Error
Météo France National Meteorological Service of France
METU Middle East Technical University (in Turkey)
MSG Meteosat Second Generation
MTF Modulation Transfer Function
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 7
MW Micro-Wave
NMA National Meteorological Administration (of Romania)
NOAA National Oceanic and Atmospheric Administration (Agency and satellite)
NWC Nowcasting
NWC-SAF SAF in support to Nowcasting & Very Short Range Forecasting
NWP Numerical Weather Prediction
NWP-SAF SAF on Numerical Weather Prediction
O3M-SAF SAF on Ozone and Atmospheric Chemistry Monitoring
OMSZ Hungarian Meteorological Service
ORR Operations Readiness Review
OSI-SAF SAF on Ocean and Sea Ice
Pixel Picture element
POD Probability of Detection
POFD Probability Of False Detection
PUM Product User Manual
PVR Product Validation Report
REP-3 H-SAF Products Valiadation Report
RMI Royal Meteorological Institute (of Belgium) (alternative of IRM)
RMS Root Mean Square
RMSE Root Mean Square Error
SAF Satellite Application Facility
SD Standard Deviation
SETHY Service d'Ètudes Hydrologiques, Walloon Ministry of Public Works - Belgium SEVIRI Spinning Enhanced Visible and Infra-Red Imager (on Meteosat from 8 onwards)
SHMÚ Slovak Hydro-Meteorological Institute
SRE Scale Recursive Estimation
SYKE Suomen ympäristökeskus (Finnish Environment Institute)
TB or TB Brightness Temperature (used for MW and, inappropriately, for IR)
TKK Teknillinen korkeakoulu (Helsinki University of Technology)
TSMS Turkish State Meteorological Service
TU-Wien Technische Universität Wien (in Austria)
UniFe University of Ferrara (in Italy)
UTC Universal Coordinated Time
ZAMG Zentralanstalt für Meteorologie und Geodynamik (of Austria)
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 8
1. The EUMETSAT Satellite Application Facilities and H-SAF
The “EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water
Management (H-SAF)” is part of the distributed application ground segment of the “European
Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)”. The application ground
segment consists of a “Central Application Facility (CAF)” and a network of eight “Satellite
Application Facilities (SAFs)” dedicated to development and operational activities to provide satellite-
derived data to support specific user communities. See Fig. 01.
Fig. 01 - Conceptual scheme of the EUMETSAT application ground segment.
Fig. 02 reminds the current composition of the EUMETSAT SAF network (in order of establishment).
Nowcasting & Very
Short Range Forecasting Ocean and Sea Ice Ozone & Atmospheric Chemistry Monitoring Climate Monitoring Numerical Weather
Prediction GRAS Meteorology Land Surface Analysis Operational Hydrology
& Water Management
Fig. 02 - Current composition of the EUMETSAT SAF network (in order of establishment).
The H-SAF was established by the EUMETSAT Council on 3 July 2005; its Development Phase started
on 1st September 2005 and ends on 31 August 2010. The list of H-SAF products is shown in Table 01.
Table 01 - List of H-SAF products
Code Acronym Product name
H01 PR-OBS-1 Precipitation rate at ground by MW conical scanners (with indication of phase)
H02 PR-OBS-2 Precipitation rate at ground by MW cross-track scanners (with indication of phase)
H03 PR-OBS-3 Precipitation rate at ground by GEO/IR supported by LEO/MW
H04 PR-OBS-4 Precipitation rate at ground by LEO/MW supported by GEO/IR (with flag for phase)
H05 PR-OBS-5 Accumulated precipitation at ground by blended MW and IR
H06 PR-ASS-1 Instantaneous and accumulated precipitation at ground computed by a NWP model
H07 SM-OBS-1 Large-scale surface soil moisture by radar scatterometer
H08 SM-OBS-2 Small-scale surface soil moisture by radar scatterometer
H09 SM-ASS-1 Volumetric soil moisture (roots region) by scatterometer assimilation in NWP model
H10 SN-OBS-1 Snow detection (snow mask) by VIS/IR radiometry
H11 SN-OBS-2 Snow status (dry/wet) by MW radiometry
H12 SN-OBS-3 Effective snow cover by VIS/IR radiometry
H13 SN-OBS-4 Snow water equivalent by MW radiometry
Decentralised processing
and generation of products
EUM Geostationary Systems
Systems of the EUM/NOAA
Cooperation
Centralised processing
and generation of products
Data Acquisition and Control
Data Processing EUMETSAT HQ
Meteorological Products Extraction
EUMETSAT HQ
Archive & Retrieval Facility (Data Centre)
EUMETSAT HQ
USERS
Application Ground Segment
other data sources
Satellite Application
Facilities (SAFs)
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 9
2. Introduction to product PR-ASS-1
2.1 Principle of the product
Product PR-ASS-1 (Instantaneous and accumulated precipitation at ground computed by a NWP
model) is the output of the operational COSMO-ME NWP model in use at CNMCA. Its main
characteristics are shown in Fig. 03.
Model Equations
Basic hydro-thermodynamical equations in advection form,
no scale approximations (i.e. fully compressible and non-hydrostatic),
substraction of horizontally homogeneous basic state at rest.
Prognostic Variables
Horizontal and vertical wind components (u,v,w)
Temperature (T)
Pressure perturbation (p', deviation from the reference state)
Specific humidity (qv),
Specific cloud water content (qc)
Specific cloud ice content (qi)
Specific rain content (qr)
Specific snow content (qs)
Turbulent kinetic energy (tke)
Optionally, specific graupel content (qg)
Coordinate System
Rotated geographical (lat/lon) coordinate system horizontally,
Generalized terrain-following height-coordinate vertically.
Grid Structure
Arakawa C-grid, Lorenz vertical grid staggering
Spatial Discretization
Second-order horizontal and vertical differencing (centred)
Time Integration
3 time-level (Leapfrog) split explicit using extensions proposed by Skamarock and Klemp (1992).
Additional Options: - 2 time-level Runge-Kutta 3rd-order scheme (regular or TVD) with
various options for high-order spatial discretization (Forstner and Doms, 2004),
- 3 time-level semi-implicit scheme (Thomas et al., 2000), - 2 time-level Runge-Kutta 2nd order split-explicit scheme (Wicker and
Skamarock, 1998).
Numerical Smoothing
Rayleigh damping layer at upper boundary
4th order linear horizontal diffusion with option for a monotonic version including an orographic limiter,
3-D divergence damping and off-centering in split steps
Lateral Boundaries
1-way nesting using the lateral boundary formulation to Davies and Turner (1977).
Initialization
Diabatic digital filtering initialization scheme (Lynch et al. 1997)
Grid-Scale Clouds and Precipitation
Cloud water formation dissipation by saturation adjustment.
Precipitation formation by a bulk-parameterization including water vapour, cloud water, cloud ice, rain and snow as hydrometeor categories (Doms 2002; Baldauf and Schultz 2004; Reinhardt and Seifert 2006).
Subgrid-Scale Clouds
Subgrid-scale cloudiness (fractional cloud cover) is interpreted by an empirical function depending on relative humidity. A corresponding cloud water content is also diagnosed.
Moist Convection
Mass-flux convection scheme after Tiedtke (1989) with closure based on moisture convergence.
Option for a modified closure based on CAPE.
Radiation
δ-two stream radiation scheme based on Ritter and Geylen (1989) for short- and longwave fluxes; full cloud-radiation feedback.
Turbulent Diffusion
Level 2.5-scheme with a prognostic treatment of turbulent kinetic energy; effects of subgrid-scale condenstation and evaporation are included.
Optionally, diagnostic K-closure (at hierarchy level 2) for vertical diffusion.
Subgrid-scale orography
Blocked flow and gravity wave drag (Lott and Miller, 1997).
Surface Layer
Scheme based on turbulent kinetic energy; includes effects from subgrid-scale thermal circulations.
Optionally, constant flux layer parameterization.
Soil Processes
Multi-layer soil model including freezing of soil water (Schrodin and Heise, 2001).
Optionally, two-layer soil model after Jacobsen and Heise (1984) with Penman-Monteith type transpiration. Snow and interception storage are included. Climate values changing monthly (but fixed during forecast) in third layer.
Current area coverage
Fig. 03 - COSMO-ME model formulation and current area coverage.
The role of PR-ASS-1 is to provide a background precipitation field regular in space and time, unlike
satellite-derived observations that are available at changing times and locations, depending on the
specific orbit. The product has been developed, is operationally running and is being progressively
improved at CNMCA. It is a “best effort” product: e.g., the covered area (as shown in Fig. 03), and the
number of runs/day will not meet H-SAF requirement by the end of the Development Phase; but it is
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 10
fully operational product. The output consists of five figures at 3-hour intervals: the rate and the
accumulated precipitation over 3. 6. 12 and 24 hours.
For more information, please refer to the Products User Manual (specifically, PUM-06).
2.2 Algorithm principle
The baseline algorithm for PR-ASS-1 processing is described in ATDD-06. Only essential elements are
highlighted here. Fig. 04 illustrates the module that most concerns the precipitation products.
2.3 Main operational characteristics
The operational characteristics of PR-ASS-1 are discussed in PUM-06. Here are the main highlights.
The horizontal resolution ( x), for a product generated by a NWP model, depends on the scale of
motion correctly represented in the model. For COSMO-ME, as concerns precipitation products, it is
estimated as ~ 30 km. The sampling interval consist of the model grid mesh ~ 7 km. Thus:
resolution x ~ 30 km - sampling distance: 7 km.
The observing cycle ( t). All products are outputted at 3-hour intervals. This is the sampling interval
for the product). However, they derive from forecasting cycles that currently are run at 00 and 12 UTC,
therefore the output products are correlated. It is correct to quote an observing cycle ~ 12 h (to be
reduced to t ~ 6 h in the near future, when runs at 06 and 18 UTC will be added). Thus:
observing cycle t ~ 12 h - sampling time: 3 h.
The timeliness ( ), for a product from a NWP process, is intended as the difference between the nominal
time of the run start and the availability of forecast products, inclusive of the window cut-off for
observation collection, analysis, initialisation, processing and output stabilisation. For COSMO-ME we
currently have:
timeliness ~ 4 h.
The accuracy (RMS) is the convolution of model features and observational data availability and
quality. Its evaluation is the task of the validation activity.
Fig. 04 - Cloud microphysical processes considered in the two-category ice scheme.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 11
3. Validation strategy, methods and tools
3.1 Validation team and work plan
Whereas the previous operational characteristics have been evaluated on the base of system
considerations (number of satellites, their orbits, access to the satellite) and instrument features (IFOV,
swath, MTF and others), the evaluation of accuracy requires validation, i.e. comparison with the ground
truth or with something assumed as “true”. PR-ASS-1, as any other H-SAF product, has been submitted
to validation entrusted to a number of institutes (see Fig. 05).
Precipitation products validation group
Leader: Italy (DPC)
Belgium IRM
Germany
BfG
Hungary OMSZ
Italy
UniFe
Poland IMWM
Slovakia
SHMÚ
Turkey ITU
Fig. 05 - Structure of the Precipitation products validation team.
Table 02 lists the persons involved in the validation of H-SAF precipitation products
Table 02 - Validation Team for precipitation products
Silvia Puca (Leader) Dipartimento Protezione Civile (DPC) Italy [email protected]
Emmanuel Roulin Institut Royal Météorologique (IRM) Belgium [email protected]
Angelo Rinollo Institut Royal Météorologique (IRM) Belgium [email protected]
Thomas Maurer Bundesanstalt für Gewässerkunde (BfG) Germany [email protected]
Peer Helmke Bundesanstalt für Gewässerkunde (BfG) Germany [email protected]
Eszter Lábó Hungarian Meteorological Service (OMSZ) Hungary [email protected]
Federico Porcu' Ferrara University, Department of Physics (UniFe) Italy [email protected]
Bozena Lapeta Institute of Meteorology and Water Management (IMWM) Poland [email protected]
Ján Kaňák Slovenský Hydrometeorologický Ústav (SHMÚ) Slovakia [email protected]
Ľuboslav Okon Slovenský Hydrometeorologický Ústav (SHMÚ) Slovakia [email protected]
Ahmet Öztopal Istanbul Technical University (ITU) Turkey [email protected]
Ibrahim Sonmez Turkish State Meteorological Service (TSMS) Turkey [email protected]
The Precipitation products validation programme started with a first workshop in Rome, 20-21 June
2006, soon after the H-SAF Requirements Review (26-27 April 2006). The first activity was to lay
down the Validation plan, that was finalised as early as 30 September 2006, i.e. about one year after the
start of the H-SAF Development Phase. After the first Workshop, other ones followed, at roughly
yearly intervals, often joined with the Hydrological validation group.
At the 1st H-SAF Workshop (Rome,16-18 October 2007), a first set of significant validation exercises
was presented. An internal document, called REP-3 (H-SAF Products Validation Report) started being
compiled since then. Now, moving to the end of the H-SAF Development Phase, REP-3 has been
restructured into this Product Validation Report (PVR) split into 13 volumes, one for each H-SAF
product. The validation experiments recorded in REP-3 constitute “Appendixes” to the various
volumes. Because of the initial aim of REP-3 (internal document at working level) the editorial level of
the Appendixes is of rather low standard.
3.2 Validation philosophy
3.2.1 Objective and problems
The products validation activity has to serve multiple purposes:
most urgent, to provide input to the product developers for improving calibration for better quality
of baseline products, and for guidance in the development of more advanced products;
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 12
also urgent, to characterise the products error structure in order to enable the Hydrological
validation programme to appropriately use the data; the Education & Training programme, part of
the Hydrological validation programme, was particularly instrumental for this;
to build up the background information necessary for online quality control of the products before
distribution;
in general, to enable attaching the necessary information on error structure to accompany H-SAF
products distribution in an open environment, after the initial phase of distribution limited to the so-
called “beta users”.
Validation is obviously a hard work in the case of precipitation, both because the sensing principle from
space is very much indirect, and because of the natural space-time variability of the precipitation field
(sharing certain aspects with fractal fields), that places severe sampling problems. It is known that an
absolute „ground truth‟ does not exist. For the performance evaluation of the H-SAF precipitation
products the radar and rain gauge measurements have been assumed as „ground truth‟. This is due to the
large use and experience of these data by the hydrologists, the main users of the products. Comparison
with results of numerical models obviously suffer of the incompatible scales between the natural
phenomenon and the model (for hydrostatic NWP models) or the limits of atmospheric predictability
when entering the scale of convection (for Cloud Resolving Models). A mixture of all this techniques is
generally used, and the results change with the climatic situation and the type of precipitation. It is
therefore necessary a European cooperation for this programme.
3.2.2 Tools to be used for validation
The areas chosen for the validation task include the basins where the hydrological validation is
performed. The data used for the validation of the satellite precipitations products are:
Ground data:
- automatic rain gauges with different time resolution: 5 min, 10 min, 15 min, 30 min;
- meteorological radars with different time resolution: 5 min, 10 min, 30 min.
Data for cloud types classification, containing information about water content in vertical column
and for the discrimination of the synoptic situation are also foreseen. The main products used to
derived these information are: products from the Nowcasting SAF; output from NWP models;
SEVIRI composite images.
Fig. 06 provides a view of the raingauge network used for precipitation products validation in H-SAF.
Fig. 06 - The network of 4100 rain gauges used for H-SAF precipitation products validation.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 13
Fig. 07 provides a view of the radar network utilised for precipitation products validation in H-SAF.
Fig. 07 - The network of 40 C-band radar used for H-SAF precipitation products validation.
3.2.3 Techniques to bring observations comparable
Due to the time and space structure of precipitation and to the sampling characteristics of both the
precipitation products and ground data used for validation, care has to be taken to bring data
comparable. At a given place, precipitation occurs intermittently and at highly fluctuating rates. Over
space, precipitation is distributed with a high variability, in cells of high intensity nested in larger area
with lower rain rate. Aimed at observing this complex phenomenon, the satellite-based products are
defined with a spatial resolution of several kilometres and with different sampling rate. On the other
hand, reference ground data used to validate precipitation data from satellite are also characterized by
their own spatial resolution ranging from point information measured on rain-gauge networks to grids
with cells of several hundreds of meters to several kilometres for weather radar. Furthermore, none of
these reference observations are without error. For this reason it was decided to compare the satellite
data with ground data on the satellite product native grid. All the institutes applied the same up-scaling
method to compare the satellite precipitation estimations with ground data.
There are several approach to bring the observation comparable. The simplest consists in comparing
untransformed data, e.g. comparing areal data to observations at a nearest gauge station, or
instantaneous images with information available within a time window. Doing so, part of the error has
to be attributed to the differences between sample volumes: this “representativeness error” may be
estimated by using high spatial and temporal resolution gauge data (e.g. Kitchen and Blackall 1992)1 or
may be simulated in numerical experiments (e.g. Tustison et al. 2001)2.
An alternative approach consists in upscaling reference observations to areal averages corresponding to
the resolution of the precipitation products but in an equal-area map projection. For rain-gauge data,
this step requires the use appropriate interpolation scheme (e.g. Thiessen polygons, kriging, etc.). For
radar images, it requires to average the values measured at radar pixels included in each of the product
1 Kitchen M. and R.M. Blackall, 1992: “Representativeness errors in comparisons between radar and gauge
measurements of rainfall”. J. Hydrol., 134, 13-33. 2 Tustison B., D. Harris and E. Foufoula-Georgiou, 2001: “Scale issues in verification of precipitation forecasts”. J.
Geophys. Res., 106, 11,775-11,784.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 14
pixel. For cumulate products, data from radar images have to be further integrated over the time
intervals and using advection procedures to correct the effect of time sampling.
The Scale Recursive Estimation (SRE) (Primus et al. 20013, Tustison et al. 20034, Gupta et al. 20065)
can be used for the validation since observations are available at one ore more scales different than the
scale of the H-SAF products. This methodology consists in filtering noisy observations taking into
account the scale-dependant variability and the nested spatial structure of precipitation. It provides
optimal estimates of precipitation at the desired scale i.e. unbiased and with the minimum variance as
well as it gives information about uncertainty at that scale. Nevertheless, it may require some
resampling of the data to make it compatible with a cascade structure.
There is a trade-off to be found between pooling the data in space and time in order to have a validation
sample large enough and stratifying into sub-samples with comparable situations so as to avoid that the
performance results be biased towards the dominant regime. The validation data may be separated in
seasons, night and day, sea and land, geographical regions, rainfall intensity and cloud or precipitation
type.
As mentioned before for the validation exercises inside this project the radar and rain gauge data were
up-scaled taking into account the satellite scanning geometry and IFOV resolution of AMSU-B scan,
SSMI and SEVIRI. Radar and rain gauge instruments provide many measurements within a single
satellite IFOV, those measurements were averaged following the satellite antenna pattern of AMSU-B,
SSMI and SEVIRI. This activity was developed in collaboration with the precipitation product
developers.
Two codes were developed by the validation group for upscaling ground data data vs AMSU-B and
SSMI IFOV. All institutes involved in precipitation product validation activity uses these two codes
developed by University of Ferrara and RMI6.
About the SEVIRI data a common code was not developed, but all institutes involved in precipitation
product validation activity uses the same up-scaling technique which was indicated by CNR-ISAC. A
common code will be developed in CDOP.
3.2.4 Structuring the results of the validation activity
During the development phase a twofold validation strategy was applied: one based on large statistics
(multi-categorical and continuous), and one on selected case studies. Both components were, and still
are, considered complementary in assessing the accuracy of the implemented algorithms. Large
statistics help in identifying existence of pathological behaviour, selected case studies are useful in
identifying the roots of such behaviour where present.
Common validation
To produce a large statistical analysis of the H-SAF Precipitation Products it was necessary to define a
„common validation methodology’ in order to make comparable the results obtained by several
institutes and to better understand their meanings.
To achieve these goal it was necessary:
standardization of the up-scaling techniques of radar and rain gauge data vs AMSU, SSMI and
SEVIRI data,
introduction of quality filter,
3 Primus I., D. McLaughlin and D. Enthekabi, 2001: “Scale-recursive assimilation of precipitation data”. Adv. Water
Resour., 24, 941-953. 4 Tustison B., E. Foufoula-Georgiou and D. Harris, 2003: “Scale-recursive estimation for multisensor Quantitative
precipitation Forecast verification: a preliminary assessment”. J. Geophys. Res., 108, 8377-8390. 5 Gupta R., V. Venugopal and E. Foufoula-Georgiou, 2006: “A methodology for merging multisensor precipitation
estimates based on expectation-maximization and scale-recursive estimation”. J. Geophys. Res., 111, D02102,
doi:10.1029/2004JD005568. 6 Van de Vyver H. and E. Roulin, 2009: “Scale recursive estimation for merging precipitation data from radar and
microwave cross-track scanners”. J. Geophys. Res., 114, D08104, doi: 10.1029/2008JD010709.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 15
development and sharing of software packages.
The Common Validation Methodology is based on comparisons with rain gauges and radar data to
produce monthly Continuous verification and Multi-Categorical statistic scores for sea, land and coast
area.
The main steps are:
all the institutes compare the national radar and rain gauge data with the precipitation values
estimated by satellite on the satellite native grid using the same up-scaling techniques;
all the institutes evaluate the monthly continuous scores (below reported) and contingency tables for
the precipitation classes (below reported) producing numerical files called „CS‟ and „MC‟ files;
all the institutes evaluate PDF producing numerical files called „DIST‟ files and plots;
the precipitation product validation leader collect all the validation files (MC, CS and DIST files),
verify the consistency of the results and evaluate the monthly common statistical results;
The results obtained were:
discussed inside the validation group and with product developers by email and two annual
meetings,
reported in the project document,
published in the H-SAF web page.
Case studies
Each Institute, in addition to the common validation methodology, developed a more specific
Validation Methodology based on the knowledge and experience of the Institute itself. This activity is
focused on case studies analysis. Each institute decides whether to use ancillary data such as lightning
data, SEVIRI images, the output of numerical weather prediction and nowcasting products.
The main steps are:
description of the meteorological event;
comparison of ground data and satellite products;
visualization of ancillary data deduced by nowcasting products or lightning network;
discussion of the satellite product performances;
indications to Developers;
making the ground data (if requested) available to satellite product developers.
The results obtained were:
discussed inside the validation group and with product developers by email and two annual
meetings,
reported in the project document,
published in the H-SAF web page.
Subdivision in classes
Since the accuracy of precipitation measurements depends on the type of precipitation or, to simplify
matters, the intensity, the verification is carried out split in more classes. For intensity, user
requirements have been expressed for three classes; however, for working purposes, finer subdivision in
11 sub-classes is used (see Fig. 08).
Class 1 2 3
< 1 mm/h (light precipitation) 1 - 10 mm/h (medium precipitation) > 10 mm/h (intense precipitation)
Subclass 1 2 3 4 5 6 7 8 9 10 11
(mm/h) < 0.25 0.25-0.5 0.5 - 1.0 1.0 - 2.0 2.0 - 4.0 4.0 - 8.0 8.0 - 10 10 - 16 16 - 32 32 - 64 > 64
Fig. 08 - Classes and sub-classes for evaluating Precipitation Rate products. Applicable to PR-OBS-1, PR-OBS-2, PR-OBS-3, PR-OBS-4 and PR-ASS-1rate
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 16
For accumulated precipitation, user requirements are unclear in terms of dependence on amount. We
have adopted a 5-class splitting for results presentation and a 10-subclass subdivision for working
purpose (see Fig. 09).
Class 1 2 3 4 5
< 8 mm 8 - 32 mm 32-64 mm 64-128 mm > 128 mm
Subclass 1 2 3 4 5 6 7 8 9 10
(mm) < 1 1 - 2 2 - 4 4 - 8 8 - 16 16 - 32 32 - 64 64 - 128 128 - 256 > 128
Fig. 09 - Classes and sub-classes for evaluating Accumulated Precipitation products. Applicable to PR-OBS-5 and PR-ASS-1accumulated
The evaluation of the statistical scores split by precipitation classes allows to analyse the product
performances not only for precipitation mean values (light precipitation being the more frequent) but
also for higher value, the most interesting for Hydrology.
Each Institute, in addition to the common validation methodology, developed a more specific
Validation Methodology based on the knowledge and experience of the Institute itself. This activity is
focused on case studies analysis. Each institute decides whether to use ancillary data such as lightning
data, SEVIRI images, the output of numerical weather prediction and nowcasting products. Specific
validation activities are indicated by each partner in the appropriate section of Chapter 3.4.
3.3 Definition of statistical scores
It is appropriate to deploy the definitions of the statistical scores utilised in H-SAF product validation
activities. Some apply to “continuous statistics”, some to “dichotomous statistics”. Although neither
rain gauges nor radar constitute a very accurate ground truth, we assume as “true” these observations,
thus the departures of satellite observations will be designated as “errors”
Scores for continuous statistics:
- Mean Error (ME) or Bias
- Standard Deviation (SD)
- Correlation Coefficient (CC)
- Root Mean Square Error (RMSE)
- Root Mean Square Error percent (RMSE %), used for precipitation since error grows with rate.
N
1k
kk )true(satN
1biasorME
N
1k
2
kk MEtruesatN
1SD
N
1k
N
1
2
k
2
k
N
1k
kk
truetruesatsat
truetruesatsat
CC with N
1k
ksatN
1sat and
N
1k
ktrueN
1true ;
N
1k
2
kk truesatN
1RMSE
N
1k
2
k2
kk
true
truesat
N
1%RMSE
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 17
Scores for dichotomous statistics
Stemming from the contingency Table: Contingency Table
Observed (ground)
yes no total
yes hits false alarms forecast yes
Forecast (satellite) no misses correct negatives forecast no
total observed yes observed no total
where:
- hit: event observed from the satellite, and also observed from the ground
- miss: event not observed from the satellite, but observed from the ground
- false alarm: event observed from the satellite, but not observed from the ground
- correct negative: event not observed from the satellite, and also not observed from the ground.
A large variety of scores have been defined. The following are used in H-SAF
- Frequency BIas (FBI)
- Probability Of Detection (POD)
- False Alarm Rate (FAR)
- Probability Of False Detection (POFD)
- Fraction correct Accuracy (ACC)
- Critical Success Index (CSI)
- Equitable Threat Score (ETS)
- Heidke skill score (HSS)
- Dry-to-Wet Ratio (DWR).
yesobserved
yesforecast
misseshits
alarmsfalsehitsFBI Range: 0 to ∞. Perfect score: 1
yesobserved
hits
misseshits
hitsPOD Range: 0 to 1. Perfect score: 1
yesforecast
alarmsfalse
alarmsfalsehits
alarmsfalseFAR Range: 0 to 1. Perfect score: 0
noobserved
alarmsfalse
alarmsfalsenegativescorrect
alarmsfalsePOFD Range: 0 to 1. Perfect score: 0
total
negativescorrecthitsACC Range: 0 to 1. Perfect score: 1
alarmfalsemisseshits
hitsCSI Range: 0 to 1. Perfect score: 1
random
random
hitsalarmfalsemisseshits
hitshitsETS with
total
yesforecastyesobservedhitsrandom
ETS ranges from -1/3 to 1. 0 indicates no skill. Perfect score: 1.
random
random
correct)pected(exN
correct)pected(exnegatives)correct(hitsHSS with
no)edno)(observ(forecastyes)astyes)(forec(observedN
1correct)pected(ex random
HSS ranges from -1 to 1. 0 indicates no skill. Perfect score: 1.
yesobserved
no observed
misseshits
negative correctalarm falseDWR Range: 0 to ∞. Perfect score: n/a.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 18
3.4 Inventory of validation facilities
In the following sections the facilities utilised in the various Institutes to perform validation of
precipitation products are described. It is apologised that editing is not well homogenised since the
various sections are recorded as they were contributed by the individual institutes, with minimum
harmonisation effort in respect of length and level of detail.
3.4.1 Facilities in Belgium (IRM)
Ground data
The validation results for Belgium presented in this report were obtained by comparison of the rain rates
products with weather radar data and of the cumulated precipitation products with either cumulated
weather radar data or rain gauge data. Table 03 summarizes the ground data used as well as the domain
over which the validation extends. The last row has been included but refers to results to be presented in
the report on hydrological validation.
Table 03 - List of ground data used for precipitation products validation in Belgium
Product Ground data Validation domain
PR-OBS-1 MW Conical Wideumont Radar 230 km 230 km
PR-OBS-2 MW Cross-Track Wideumont Radar 230 km 230 km
PR-OBS-3 IR+MW Rapid Update Wideumont Radar 230 km 230 km
PR-OBS-5 Cumulated 24h Cum. Wideumont Radar 230 km 230 km
PR-OBS-5 Cumulated 24h SETHY Raingauges Walloon Region
PR-OBS-5 Cumulated 24h RMI Daily Raingauges Test Catchments
Weather Radar
Belgium is well covered with three radars (see Fig. 10). A further radar is currently under construction
in the coastal region. These are Doppler, C-band, single polarization radars with beam width of 1° and a
radial resolution of 250 m. Data are available at 0.6, 0.66 and 1 km horizontal resolution for the
Wideumont, Zaventem and Avesnois radars respectively.
In this report, only the Wideumont radar has been used. The data of this radar are controlled in three
steps. First, a long-term verification is performed as the mean ratio between 1-month radar and gauge
accumulation for all gauge stations at less than 120 km from the radar. The second method consists in
fitting a second order polynomial to the mean 24 h (8 to 8 h local time) radar / gauge ratio in dB and the
Fig. 10 - Meteorological radar in Belgium.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 19
range; only the stations within 120 km and where both radar and gauge values exceed 1 mm are taken
into account. The third method is the same as the second but is performed on-line using the 90
telemetric stations of the SETHY (Ministry of the Walloon Region). Corrected 24 h images are then
calculated. New methods for the merging of radar and raingauge data have been recently evaluated
(Goudenhoofdt and Delobbe 2009)7.
Raingauge
Several raingauge networks are managed in Belgium (see Fig. 11). RMI has a dense network of daily
raingauge and an increasing network of automatic weather stations equipped with tipping bucket
gauges. Other networks are operated by the Regional Authorities in charge of rivers. For the validation
of the PR-OBS-5, we have used hourly data from the SETHY raingauge which are quality controlled
daily at RMI. The daily data are gathered and checked with 1.5 to 2 month delay. These later data are
mainly used in the hydrological validation programme.
For the validation of the PR-OBS-5 cumulated rainfall product, a validation with raingauge data has
been performed, in parallel to the radar validation. The reference data used are hourly rain gauge records
from the SETHY (Walloon Region) network (Fig. 12). The network includes 89 automatic non-heated
stations and 3 heated stations (in coincidence with non-heated ones). Only the non-heated stations have
been considered, for the sake of uniformity. The data have been interpolated in onto a 5 km 5 km grid,
following the Barnes method. The sensitivity parameter in the Barnes procedure has been set to 108,
considering the fact that the mean distance between every station and its closest neighbor is roughly 104
m. The interpolation procedure is iterative. If the mean squared difference between the source field and
the interpolated field falls below 0.01 mm h-1, or if the improvement is below 1% between two steps, the
procedure is stopped, otherwise it goes on for a maximum 20 iterations. The result is a series of files
with interpolated data, one per hour.
The quality of the interpolated data has been checked for several months in the following way: the
interpolation is calculated taking into account all the stations except one, and the value corresponding to
the missing station is estimated. The procedure is repeated for all the stations. A set of 89 reconstructed
values is obtained, and compared with the measured data. The verification refers to the period from
August to November 2008. The interpolation is first assessed in its capacity to reconstruct the rain / no
rain field. Taking 0.01 mm h-1 as a threshold, the probability of correct rain (POD) is 0.79, the false
reconstruction (FAR) is 0.07 and the equitable threat score (ETS) is 0.71. Then, statistical scores are
calculated on a monthly basis. The bias ranges from 0.06 to 0.14 mm h-1, the root mean square from
7 Goudenhoofdt E. and L. Delobbe, 2009: “Evaluation of radar-gauge merging methods for quantitative precipitation
estimates”. Hydrol. Earth Syst. Sci., 13, 195-203.
Fig. 11 - RMI raingauges: daily ( ) and AWS ( ). Fig. 12 - SETHY AWS network in Walloon Region.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 20
0.37 to 1.00 mm h-1 and the mean relative error from 0.09 to 0.19 for mean observed values of 0.50 to
1.00 mm h-1.
As preliminary test for the hydrological validation of PR-OBS-5, the data of the daily raingauge stations
have been interpolated using the Thiessen polygons method and spatially averaged over the two test
catchments. The values obtained have been compared with the corresponding cumulated values from
satellite.
Miscellaneous information
For the analysis of test cases, additional information has been used like the cloud types identified using
the SAF-NWC tools, the expertise of weather forecasters to select and analyze the synoptic conditions,
the SAFIR maps of lightning impacts.
Methodology
From a local point of view, rain rates products based on microwave sensors onboard of Low Earth Orbit
satellites are characterized by a varying coverage and projection. To make the statistics comparables
from one file to the other, a validation domain has been defined which is a square of 230 km 230 km
centered on the Wideumont radar location and only the products covering entirely this common area
have been considered. To be more precise, for every product file, a sub-set of lines and columns
including the common square has been extracted. Then, the radar data have been up-scaled to the
projection of the sub-set of pixels and compared with the product estimates.
The up-scaling of the radar data is performed taking the footprints of the microwave sensors into
account. In Fig. 13, the Gaussian filter corresponding to the first scan position of the AMSU-B antenna
(PR-OBS-2) is represented on the left. The filtering procedure is organized as follows (see the figure, on
the right). First, a part of the radar image is selected (grey). Then, the radar data (0.6 km resolution are
re-sampled onto a tilted grid (2 km resolution) where the Gaussian filter is 1% of maximum (dark
grey). Tilting depends on the scan position and on the satellite overpass mode. Finally the Gaussian
filter is applied. The black ellipse corresponds to half power. Additional information about the up-
scaling equations and about the tilting of the PR-OBS-2 pixels can be found in Van de Vyver and
Roulin (2008).
For PR-OBS-3 and PR-OBS-5, a sub-set of lines and columns has been also extracted which comprises
the common validation area. The up-scaling has been simply performed by averaging the radar values
included in each pixel in the SEVIRI projection. For the validation of PR-OBS-5 using raingauge data,
the ground data have been interpolated as explained above and the comparison has been performed
between the product estimate and the nearest interpolated grid point over a domain corresponding to the
Walloon region in Belgium. Finally, the scores of the continuous statistics, the contingency tables and
the probability distribution functions have been prepared on a monthly basis according to the rules
common to all the teams involved in the precipitation products validation.
Fig. 13 - Left: Gaussian filter; right: sketch of the up-scaling procedure. The circle corresponds to the range of the weather radar. The square in the middle is a common area such that it is entirely included in the selected PR-
OBS-2 files. The grey rectangle, the tilted dark grey rectangle and the black ellipse are explained in the text.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 21
Scale Recursive Estimation
As specific development, we have investigated an application of scale recursive estimation (SRE) to
assimilate rainfall rates during a storm estimated from the data of two remote sensing devices. These are
ground based weather radar and space-born microwave cross-track scanner (PR-OBS-2). Our approach
operates directly on the data and does not require a pre-specified multi-scale model structure. We
introduce a simple and computational efficient procedure to model the variability of the rain rate process
in scales. The measurement noise of the radar is estimated by comparing a large number of datasets with
rain gauge data. The noise in the microwave measurements is roughly estimated by using up-scaled
radar data as reference. Special emphasis is placed on the specification of the multi-scale structure of
precipitation under sparse or noisy data. The new methodology is compared with the latest SRE method
for data fusion of multi-sensor precipitation estimates. Applications to the Belgian region show the
relevance of the new methodology (Van de Vyver and Roulin 2009)8.
8 Van de Vyver H. and E. Roulin, 2009. “Scale recursive estimation for merging precipitation data from radar and
microwave cross-track scanners”. J. Geophys. Res., 114, D08104, doi: 10.1029/2008JD010709.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 22
3.4.2 Facilities in Germany (BfG)
Precipitation data
One of the responsibilities of the Federal Institute of Hydrology (BfG) is the shipping related water level
forecast for the river Rhine at low and middle flows. For tasks like hydrological modeling there is
mainly a need for hourly and daily meteorological data which are provided to BfG by Germany‟s
National Meteorological Service (Deutscher Wetterdienst DWD).
It is intended to conduct precipitation validation activities for the territory of Germany.
Germany has a rather dense network of raingauges and it is covered by 16 radars plus one research radar
at Hohenpeissenberg (see Table 04, Table 05 and Fig. 14, from Bartels et al. 20049).
Table 04 - Precipitation data available at BfG
Data Number Resolution Delay Annotation
Synoptical stations About 200 6h / 12h Near-real-time
TTRR stations About 1000 hourly Near-real-time
PI (Picture Composite International)
European radar sites
15 min, 4 km x 4 km
Provided in hourly intervals
International composite image with ground-proximate radar reflectivity distribution
RW (High Resolution Calibrated Quantitative Composite (national)
16 German radar sites
1 hour, 1 km x 1 km
Near-real-time Quantitative radar composite product from RADOLAN software
Table 05 - Location of the 16 meteorological radar of the DWD
Radar site Launch Model WMO No. Radar site Launch Model WMO No.
München 1987 DWSR-88 C 10871 Rostock 1995 METEOR 360 AC 10169
Frankfurt 1988 DWSR-88 C 10637 Ummendorf 1996 METEOR 360 AC 10356
Hamburg 1990 DWSR-88 C 10147 Feldberg 1997 METEOR 360 AC 10908
Berlin-Tempelhof 1991 DWSR-88 C 10384 Eisberg 1997 METEOR 360 AC 10780
Essen 1991 DWSR-88 C 10410 Flechtdorf 1997 METEOR 360 AC 10440
Hannover 1994 METEOR 360 AC 10338 Neuheilen-bach 1998 METEOR 360 AC 10605
Emden 1994 METEOR 360 AC 10204 Türkheim 1998 METEOR 360 AC 10832
Neuhaus 1994 METEOR 360 AC 10557 Dresden 2000 METEOR 360 AC 10488
9 Bartels H. et al. / Deutscher Wetterdienst, Abteilung Hydrometeorologie, 2004: „Projekt RADOLAN -
Routineverfahren zur Online-Aneichung der Radarniederschlagsdaten mit Hilfe von automatischen
Bodenniederschlagsstationen (Ombrometer)“. Summary report for the project period 1997-2004.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 23
RADOLAN
RADOLAN (Routine procedure for an online calibration of radar precipitation data by means of
automatic surface precipitation stations ‘ombrometers’) is a quantitative radar composite product
provided in near-real time (via ftp) by DWD to BfG. Radar data are calibrated with hourly precipitation
data from automatic surface precipitation stations. For a description of the radar network see
http://www.dwd.de/de/Technik/Datengewinnung/Radarverbund/Standorte.htm .
The process chain from the five-minute-interval radar signals to the final hourly precipitation product is
presented in the Fig. 15. RADOLAN data of hourly precipitation (sampling period hh:51 min to
(hh+1):50 min) have a precision of 0.1 mm/h and cover the whole territory of Germany with a spatial
resolution of 1 km.
Fig. 15 - Flowchart of online calibration
RADOLAN (adapted from Bartels et al. 2004).
Fig. 14 - Left panel: radar coverage in Germany as of 01/03/2007. Right panel: location of ombrometers for online calibration in RADOLAN; squares: hourly data provision (about 500), circles: event-based hourly data provision (about 800 stations): red: AMDA III, blue: aggregational network federal states (Bartels et al., 2004).
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 24
3.4.3 Facilities in Hungary (OMSZ)
Ground data description (instrument characteristic and map)
In Hungary, about 90 automatic stations work (Fig. 16), where 10-min precipitation is measured by
tipping bucket rain gauges. This data is used to correct the accumulated precipitation radar data
Fig. 16 - The automatic rain gauge network in Hungary.
The main data used for validation in Hungary would be the data of meteorological radars. There are
three C-band dual polarized Doppler weather radars operated routinely by the OMSZ-Hungarian
Meteorological Service (see Fig. 17 and Table 06).
Table 06 - Characteristics of the three meteorological Doppler radars in Hungary
Year of installation Location Radar type Parameters measured
1999 Budapest Dual-polarimetric, Doppler radar Z, ZDR
2003 Napkor Dual-polarimetric, Doppler radar Z,ZDR,KDP,ΦDP
2004 Poganyvar Dual-polarimetric, Doppler radar Z,ZDR,KDP,ΦDP
Ground data quality and accessibility
Access to Hungarian radar data can be set up through contact with the responsible of the institute within
the HSAF project. It will be provided for developers if required for case studies.
Pogányvár Napkor
Budapest
Fig. 17 - Location and coverage of the three meteorological Doppler radars in Hungary.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 25
The quality of the radar measurements is influenced by several factors:
- the accuracy of the Marshall-Palmer equation in deriving rain rates
- the beam blockage which causes the lack of precipitation in areas behind mountains.
The Hungarian radar data bears the consequences of both problems. The Marshall-Palmer calculations
can result in a factor of multiplying or dividing by 2; whereas the beam blockage can result in serious
underestimation of precipitation amounts (e.g. behind the Börzsöny mountains at the north of Budapest).
Besides, the Hungarian radar data is filtered from WLAN signal, which is also a source of false signals
in the radar.
A filter to disregard signals below 7 dBz is also applied because in general, these data is not coming
from real rain drops, but false targets.
Ground data products
Precipitation intensity is derived from radar reflectivity with the help of an empirical formula, the
Marshall-Palmer equation. From the three radar images a composite image over the territory of Hungary
is derived every 15 minutes applying the maximum method in order to make adjustments in overlapping
regions.
The non-corrected precipitation field can be corrected by rain gauge measurements. As recent
researches have shown it is only possible to produce adequate precipitation fields by the correction of
raw radar data at time scales of the order of a few hours or more, thus we do not make corrections to 15
minutes radar data. In our institute, we only use a correction for the total precipitation over a 12 hour
period.
For the 3h and 6h accumulated products, we use a special method as well: we interpolate the 15-minutes
measurements for 1-minute grid by the help of displacement vectors also measured by the radar, and
then sum up the images which we got after the interpolation. It is more precise especially when we have
storm cells on the radar picture, because a storm cell moves a lot during 15 minutes and thus we do not
get continuous precipitation fields when we sum up only with 15.minutes periods. This provides
satisfying results. However, there is still a need for rain-gauge adjustment because there are obviously
places (behind mountains) that the radar does not see.
The radars are corrected with rain gauge data every 12 hours. The correction method using raingauge
data for 12 hour total precipitation consists of two kinds of corrections: the spatial correction which
becomes dominant in the case of precipitation extended over a large area, whereas the other factor, the
distance correction factor prevails in the case of sparse precipitation. These two factors are weighted
according to the actual situation. The weighting factor depends on the actual effective local station
density, and also on the variance of the differences of the bias between radar and rain gauge
measurements. On the whole, we can say that our correction method is efficient within a radius of 100
km from the radar. In this region, it gives a final underestimation of about 10%, while at bigger
distances, the underestimation of precipitation fields slightly increases. Besides, we also produce 12
hour total composite images: first the three radar data are corrected separately, and then the composite is
made from them. The compositing technique consists of weighting the intensity of each radar at a given
point according to the distance of the given point from the radars.
Ground data interpolation
No interpolation is used for the radar data.
Institute validation methodology
The data used for the discrimination of case studies:
OMSZ-Hungarian Meteorological Service receives operationally the MSG data and several products
based on satellite data are derived and sent to the weather forecasters. The SAFNWC/MSG program
package derives cloud type product and precipitation products (convective rain rate and probability
of precipitation) also. MSG composite images are created operationally. For the description of
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 26
meteorological environment, the Cloud Type product is the most useful as it distinguishes high-level
and optically thick clouds from medium- and low-level clouds; and optically thin clouds. It is also
useful for analyzing the cloud systems that have occurred at the time of satellite measurements.
SAFIR lightning system works operationally at Hungarian Meteorological Service. We have five
stations. The accuracy of the lightning network has been significantly increased the last two due to
the large amount of case studies investigated during summer periods. Therefore we intend to use the
lightning data, first of all in visual comparison as descriptor of the synoptic situation.
The main steps of the precipitation products validation procedure are:
Time and space alignment of the data from different sources, represented on different scales – pixel
size of radar and satellite data is not the same. Especially in the case of microwave measurements,
we have large footprints that need to be treated carefully. The time alignment can be solved by
simply matching the closest data available in time to the H-SAF products. There are advanced
methods to solve the up-scaling of ground and/or the down-scaling of satellite precipitation data.
The OMSZ-Hungarian Meteorological Service uses the techniques discussed and proposed by the
members of the validation group. We will contribute to the selection of the appropriate method and
to the investigation of the accuracy of the matching method.
The common task of the Precipitation Validation Group is the statistical validation. The up-scaling
method has a key role in this process. We can only compare the radar, satellite, raingauge, and
lightning if we have previously determined the coherent values from different sources. The data will
be collected into one file containing the following information to calculate statistic values either for
15 minutes samples, and for accumulated total precipitation information (over 3, 6, 12, 24 hours):
- radar-derived precipitation intensity,
- the satellite product values derived by H-SAF.
Statistical characteristics (RMS, BIAS, etc) will be then calculated between the satellite-based
precipitation and radar data for the different cases. These statistics will be performed for different
periods (months, summer-winter, day-night) as well as for some satellite pass in extreme cases .
We will also calculate multi-categorical statistics according to the classes determined by common
agreement by the validation group and the hydrologists. We will prepare contingency tables, scatter
plots and bias distribution functions to illustrate the performance of the precipitation products. These
statistics will be performed for different periods (months, summer-winter, day-night) and maybe for
some specific satellite passes. H-SAF data is going to be verified against radar data which preferably
will be adjusted to rain gauge data. Our radar data is already gauge-adjusted in the case of 12, and 24h
total precipitation.
The first task to the preparation of case studies is the selection of the periods with significant rain
amounts. This is done by the plotting of the radar and satellite rainfall estimate sums on a daily basis
of each month.
The second task to the preparation of case studies is the collection of all relevant data to the period
in question. This means the collect of radar, lightning data, and of MSG images and SAFNWC
products which correspond to the time period investigated.
The visual comparison including both rain gauge and radar data is an inevitable tool for the
understanding of the validation results of H-SAF products, because it:
- provides an overall picture of the H-SAF database (resolution, territories seen, general values)
- highlights similarities and differences in the structures and in the values
- helps to monitor the meteorological situation as well
To visualize the different products and to make subjective comparison we use the Hungarian Advanced
Weather Workstation (HAWK) developed by OMSZ-Hungarian Meteorological Service which can
interpret all kind of meteorological products together.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 27
3.4.4 Facilities in Italy (UniFe)
Introduction
This report aims to set up a precipitation validation plan for Italian region. The satellite precipitation
estimation products to be validated come from the development and operation activity. A quick
overview of the products characteristics as request from the approved H-SAF development proposal is
provided. Measurements from raingauge and retrievals from weather radar will be considered as
precipitation “truth” and used to validate those products. A description of the availability of raingauge
and radar precipitation data over Italy is provided. A validation plan at different levels is suggested.
Satellite precipitation estimation products
CNR-ISAC and CNMCA are in charge to develop algorithms to estimate precipitation from microwave
(MW) and infrared (IR) sensors on board respectively of polar and geostationary satellite. The spatial
resolution of MW precipitation product is around 15 km whereas the time resolution is about 6 hours.
The precipitation estimation from merging/morphing MW and IR will be generated at around 5 km of
spatial resolution and 15 minutes of time resolution. Both the products have to be validated. Both
instantaneous and cumulated precipitation (3, 6, 12 and 24 hours) have to be validated, however, in case
of MW precipitation estimation due to its very low time resolution only instantaneous precipitation
estimation can be considered and therefore validated.
Data availability and facilities
The Dipartimento Protezione Civile (DPC) and the Centro Nazionale di Meteorologia e Climatologia
Aeronautica (CNMCA) will provide data set to be used as precipitation “truth”. Measurements from
1200 raingauge distributed along the Italian region are considered the main source of precipitation
“truth”. Those data are available at 30 minutes of cumulated time. There is also the possibility to have
some data set of raingauges with a cumulated time of 5 minutes. These data can be used to validate
instantaneous precipitation estimation. For cumulated precipitation validation a longer cumulated time
have to be considered. The radar network will provide precipitation estimation every 30 minutes and the
sum of data inside longer interval is considered as cumulated radar rain. A digital elevation model of the
Italian region is necessary to select land and sea region and also to evaluate the impact of orography on
the precipitation data.
For the following validation procedure the use of IDL, Fortran 90 and C++ software is foreseen. The
first one is useful to visualise the images and therefore for a visual inspection. The second one is useful
in case of validation of very large data set.
Validation recommendation
Precipitation estimation validation means to compare satellite precipitation estimation against other
source of precipitation data assumed as “truth”, as provided by raingauge and radar, and the comparison
procedures have to be specified. Let us to consider two rainfall maps: the first one from raingauge/radar
data and the second one from satellite estimation, three kind of comparison are here suggested.
VISUAL COMPARISON: the two maps are compared by visual inspection. The human eye has a natural
and powerful ability at judging the similarity of two images. IDL software is used to produce and
compare satellite and raingauge/radar rainfall maps,
CONTINUOS STATISTICS COMPARISON: the rainfall rate from satellite estimation and radar/raingauge
are provided in a continuous range and to quantify the similarity of the two images it is convenient
to calculate some parameters considering pixel by pixel. The suggested parameters are: Mean Error
(ME, or bias), Root Mean Square Error (RMSE) and correlation coefficient (CC).
CATEGORICAL STATISTICS COMPARISON: the rainfall rate from satellite estimation and radar/raingauge
data have to be organized in classes of precipitation. At the first step by fixing a rainfall threshold of
0.1 mm/h only two classes of precipitation (rain and no-rain) can be considered. Then the
precipitation classes can be those indicated in Fig. 08 and Fig. 09. Once the continuous data are
organized in classes the contingency table have to be built. Then some statistical parameters have to
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 28
be calculated to quantify the similarity of two images. In case of two classes of precipitation the
suggested parameters are: FAR, POD, BIAS, ETS and HSS. In case of more than two classes the
useful parameters are: HSS and correlation coefficient.
For a reliable validation plan an ensemble of cases (each of them consisting in estimated and “truth‟
maps) have to be considered. The larger the ensemble is the more reliable the validation is. The visual
comparison for several pairs of images become not easy and therefore only the 2 and 3 procedures are
suggested. However it does worth to mention that there are two ways to carry out those procedures: a)
all the pixels from the ensemble of cases are considered together. We obtain an ensemble of pixels of
satellite precipitation estimation and an ensemble of pixels of precipitation “truth”. The two ensemble
are compared by calculating the previously indicated parameters. b) for each case the comparison is
carried out by calculating the above statistics parameters. The average values of those statistics
parameters are calculated over the ensemble of cases.
It is also suggested to carry out the validation for two different areas: sea and land. This because the
precipitation “truth” over the land is mainly based on raingauges whereas for the sea only radar
estimation are available. A further distinction will be made considering orography, by using a digital
elevation model (e.g. GTOPO30 by the U.S. Geological Survey).
Validation: complete plan
The precipitation “truth” data set is built for a complete year at 15 minutes of time resolution.
Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over Italy during the
considered year can be collected and used to generate the corresponding MW-based precipitation
estimation. All of them can be validated. MW+IR algorithm: for each time of the day and for each
day of the year it is possible to validate the MW+IR-based precipitation estimation. In this case the
variability of performance along the day and along the year can be assessed. Moreover the
degradation of MW+IR performance far from the MW overpass can be evaluated.
Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3, 6, 12 and 24 hours
intervals can be validated by using the corresponding cumulated “truth” rain. The variability of
performance along the year can be assessed as well.
Validation: minimal plan
The precipitation “truth” data set is built for summer and winter at around 12 and 24 UTC.
Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over Italy in that
period can be collected and used to generate the corresponding MW-based precipitation estimation.
All of them can be validated.
Instantaneous rain validation (MW+IR algorithm): for 12 and 24 UTC times and for each day of
summer and winter it is possible to validate the MW+IR-based precipitation estimation. In this case
it is expected to get information about the maximum variability of performance.
Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3,6,12 and 24 hours
intervals for summer and winter can be validated by using the corresponding cumulated “truth” rain.
Validation: specific plan
The precipitation “truth” data set is built for particular region of Italy and/or for particular month and/or
particular time of the day.
Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over that region and
in that period can be collected and used to generate the corresponding MW-based precipitation
estimation. All of them can be validated. MW+IR algorithm: for that region and that period it is
possible to validate the MW+IR-based precipitation estimation.
Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3, 6, 12 and 24 hours
intervals for that region and that period can be validated by using the corresponding cumulated
“truth” rain.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 29
3.4.5 Facilities in Poland (IMWM)
Precipitation data measurements and observations in Poland
The validation of the H-SAF precipitation products will be carried out in the IMWM on the base of the
ground measurements networks that include:
Telemetry network of automatic rain gauges.
Network of standard rain gauges.
Pluviograph network.
Meteorological radars.
The telemetry network encompasses 430 posts with automatic rain-gauges (2 instruments at each post).
The posts are located all over the whole territory of Poland, however the network is more dense in the
Southern Poland where the flood danger is very high. The measurements are available in the near-real
time and its frequency may be configured at the telemetric network. In standard operational mode the
rainfall and water level data are provided every 10 minutes from the post in the Southern Poland and
every 1 hour from the other ones. The quality of the data is checked on the level of processing in
operational mode.
The network of standard rain gauges consists of 61 synoptic stations and 210 climatological stations, all
equipped with rain-gauges operated by observers as well as 1027 rain-gauge posts. The network is
operational and the data are available in real time from the stations situated in the Southern Poland and
off-line from other parts of country. The synoptic stations provide the standard, 6 hour cumulative
values. The data from other posts are available once per day at normal mode and every 3 hours during a
flood. The quality of the data is checked on the level of processing in the annual mode.
The pluviograph network includes 102 stations located all over the whole country. The network status is
historical. The data are collected only during the period between the 1st of May and the 31st of October
with 10 minute time resolution. The quality of the data is checked on the level of processing in the
annual mode.
The meteorological radar network consists of 8 Doppler radars located all over the country. The data are
available in the real time with 10 minute resolution for analysis and 15 minute resolution for the
forecast. Generally, radar measurements are processed calibrated through/in NIMROD system.
Quality and Accessibility
The data from telemetric posts, rain gauges and synoptic stations located in the Southern Poland are
operationally collected in the database of the System of Hydrology SH located in the Krakow Branch of
IMWM. The data from all standard measuring stations and post are collected in the central data base of
the IMWM in Warsaw. The quality checked data are available from this data base with at least half of a
year delay. The radar measurements are collected in the Radar Centre of IMWM in Warsaw.
The databases available in the IMWM have been check in order to estimate their usefulness for H-SAF
purposes. For data collection, the database existing in the frame of the System of Hydrology will be
used. However, the present database capacity does not allow introducing new users. Therefore it has
been decided that, before the further database development, the data required for H-SAF will be
archived separately in order to simplify their use in the future.
Validation
The validation of the precipitation products will be carried out in the Satellite Research Department of
IMWM in Krakow. Due to its characteristics and availability of quality checked measurements, the 10
minute data from the telemetric network will be used for the validation. It is assumed The standard
observations will play supporting role especially in long term analysis.
The validation process/chain will be performed for satellite projection and include two parts. Firstly, the
analysis, common for all valiadation partners, will be carried out. In the frame of this part of validation
the continuous statistical analysis for all data set and separately for rainfall classes (convective,
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 30
stratiform) and different seasons will be performed. The following characteristics are to be obtained:
correlation coefficient, STD, ME, root mean square error. For merging between satellite and ground
measurements, the technique proposed by the validation group will be applied.
Next, the categorical statistical analysis will be performed both for the classes proposed by the
validation group and the classes agreed with Polish hydrologist.
In the second part of the validation it is assumed that the satellite derived cumulative precipitation field
will be compared with the one obtained on the base of the ground measurements. In the analysis the GIS
techniques will be used. The works on the spatialisation methods for precipitation are being developed
in the Krakow Branch of IMWM;
The use of the radar data in validation heavily depends on the scheme of the Radar Operational Centre
team. They may provide the unique methodology of satellite-radar comparison but there are still issues o
be decided.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 31
3.4.6 Facilities in Slovakia (SHMÚ)
Instruments
Slovak Hydrometeorological Institute (SHMÚ) has a large network of the ground stations able to
measure rainfall (see Fig. 18). 684 of these raingauges will be used for validation purpose during this
project covering the complete territory of Slovak Republic. 98 gauges are used operationally and 586
are for research purpose (climatology). It is expected that the number of operational gauges will increase
during the project. Data are available in 1 min, 10 min, 1, 6, 12, 24 hours interval depending on the
gauge type. SHMÚ is performing the offline automatic and manual quality check.
Fig. 18 - Map of SHMÚ raingauge stations: green – operational (98) , blue – climatological (586), red - hydrological stations
in H-SAF selected test basins (37). White points show regular grid of experimental NOAA Snow water equivalent data.
Radar data are planed to be used in this project too. The Slovak meteorological radar network consists
of 2 meteorological radars (see Fig. 19). One is situated at Maly Javornik near city Bratislava and
second is on top of Kojsovska hola close to the city Kosice. Both are Doppler, C-band radars with the
beam width 1 degree and the radial resolution from 256 m (operational). The newer radar at Kojsovska
hola is able to measure the dual polarization variables (non operational). The radar measured
precipitation intensity is available for the whole radar network every 15 minutes, cumulative
precipitation is done for a period of 1, 3, 6 and 24 hours. There is no quality check applied to radar data
operationally. The radar rainfall intensity and the radar cumulative precipitation are not yet corrected or
adjusted to the gauges observations, it will be a part of this project.
Fig. 19 - Example of Slovak radar network coverage - left circle corresponds to radar site Malý Javorník, right one corresponds to Kojšovská hoľa.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 32
Methods
Satellite data are usually the raster data with defined projection and pixel size/resolution. Radar data are
also raster data, but usually in a different projection and resolution from satellite data. Therefore the first
step is to transform both data to the common projection and pixel resolution map optimal for data
comparison. SHMÚ has good experiences how to perform necessary re-projections operationally.
Raingauge data are point data, which can be mapped by means of 2D or 3D interpolation which use the
method of regularised splain with tension. When 3D interpolation is used, orography is reflected in the
outputs. This method is generally called kriging. In this way, the original point data are transformed to
the raster file comparable with satellite and radar data (see Fig. 20).
Fig. 20 - Example of 5-days cumulative precipitation constructed from raingauge measurements by means of 3D
extrapolation method.
Radar data should be validated and adjusted before being used for the validation of other products.
SHMÚ has made calibration study on radar site Maly Javornik based on regression between set of
raingauges and radar 24-hour precipitation products in 150 km radar range few years ago. The
corrections on distance from radar and partial beam blockage by terrain were subject of this study. We
plan to make similar study for the whole territory of Slovakia. Output should be used for the operational
calibration and quality control of radar precipitation measurements. We are also planning to test method
for precipitation estimation based on polarisation techniques for new radar in eastern part of Slovakia.
Quality control of radar data will be based on NWC SAF products like cloud types classification and
products containing information about water content in vertical column.
SHMÚ is planning to use three methods to perform final validation of satellite precipitation products:
Direct comparison of satellite data with point raingauge measurements, calculation of standard
statistical parameters RMSE, ME and other.
Comparison of satellite raster data with precipitation raster based data (interpolated raingauges,
corrected radar precipitation fields,…) and with calculation of standard statistical parameters.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 33
Dividing of precipitation amounts to the defined intervals (classes) according to precipitation
intensity and to evaluate the following parameters: probability of detection (POD), equitable threat
score (ETS), false alarm ratio (FAR) and critical success index (CSI).
Hydrological validation
Slovak Hydrometeorological Institute (SHMÚ) has a large hydrological network of the ground stations
able to measure water level (see Fig. 21). There are 212 stations measuring in online mode. Some of
them (on the Myjava, Kysuca, Hron and Topl‟a river) will be used as input data for H-SAF application.
The data from these stations are available in 15 min interval. SHMÚ is performing the offline automatic
and manual quality check.
Fig. 21 - Map of SHMÚ hydrological stations: red - hydrological stations in H-SAF selected test basins (34).
Hydrological model HRON are conceptual (HBV type) model, semi-distributed with altitude zones
(Fig. 22).
Fig. 22 - General hydrological model of HBV type.
(pictures are from http://www.smhi.se/en/index.htm)
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 34
Fig. 23 - Example of pick up pixels from radar measurement for selected area.
Model input data are discharge and average data of precipitation
and temperature. We calculate average value from ground stations
and from radar measures. Meteorological forecasted data are taken
from weather prediction model ALADIN.
Average catchment precipitation value from ground station is
calculated using Thiessen polygons method and arithmetic mean
with different station weights assigned a priori. Average
temperature value is calculated as weighted arithmetic mean.
Catchment average precipitation value of 1-hour precipitation
forecast from radar product is calculated as average value of pixels
from the selected area (mask of subcatchment) (see Fig. 24).
Model ALADIN supplies precipitation and temperature forecasts to
hydrological model HRON. Since the resolution of forecast is too
low for selected areas, the large pixel is separated into smaller
pixels for better average value results (see Fig. 24). SHMÚ is
planning to use satellite precipitation products in forecasting hydrological models in similar way.
Mask of selected area. Radar measurement sector with selected area border.
Radar measurement in the selected area only.
Fig. 24 - Example of separated ALADIN
forecast pixel to smaller pixels.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 35
3.4.7 Facilities in Turkey (ITU)
Locations and instruments
For the validation of precipitation products, a first set of locations were selected as Susurluk and
Western Black Sea catchments which represent different physiographic and climatic conditions of
Turkey (see Fig. 25). The Susurluk catchment is influenced by Mediterranean type of climate with mild-
wet winters and hot-dry summers. The basin is covered by forest lands together with fertile alluvial
plains. On the other hand, the Western Black-Sea catchment is characterized by high rainfall dominating
almost two thirds of the year and humid summers. The catchment is covered by highly dense forests and
agricultural lands. The catchment has mountains with low elevations rarely exceeding 1,500 m.
Fig. 25 - The Susurluk and Western Black Sea catchments selected for the precipitation product validation in Turkey.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 36
Validation of precipitation products is based mainly on the comparison with the AWOS and weather
radar data measurements. Turkey has an extensive coverage of meteorological observations, including
automated weather observing stations (AWOS), synoptic and climatological stations, and 4 radars. At
present nearly 450 meteorological, 60 airport, 180 climate, and 7 upper air (radiosonde) stations are in
operation. The meteorological observation network has been strengthened by installation of AWOS in
recent years, and in the near future almost entire country will be covered by an AWOS network to
replace the conventional network.
As shown in Fig. 26, the Western Black Sea catchment has 23 AWOS and 7 synoptic stations in
addition to radar coverage. Similarly, the Susurluk basin has 17 AWOS and 3 synoptic stations and
radar coverage.
Fig. 26 - Network of Meteorological stations in Susurluk (on the left) and Western Black Sea (on the right) catchments.
Starting from January 2009, the validation area has been extended to the middle and western Turkey.
193 Automated Weather Observation Station (AWOS) located in the western part of Turkey are used for
the validation of the precipitation products. The locations of the AWOS sites are shown in Fig. 27.
Fig. 27 - Position of 193 AWOS sites used for ground truth for the precipitation product validation in western Turkey.
Validation Methodology
Due to the time and space structure of precipitation and sampling characteristics of both the
precipitation products and observations used for validation, care has to be taken to bring data to a
comparable level. At a given place, precipitation occurs intermittently and at highly fluctuating rates.
Over space, precipitation is distributed with a high variability, in cells of high intensity nested in larger
area with lower rain rate. Aimed at observing this complex phenomenon, the satellite-based products are
defined with a spatial resolution of several kilometres and with different sampling rate and accumulation
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 37
time. On the other hand, reference observations against which the products are going to be validated are
also characterized by their own spatial resolution ranging from point information measured on rain-
gauge networks to grids with cells of several hundreds of meters to several kilometres for weather radar.
Furthermore, none of these reference observations are without error.
Each precipitation product within the H-SAF project represents a foot print geometry. Among these,
H01 and H02 products represent an elliptical geometry while H03 and H05 have a rectangular
geometry. On the other hand, the ground observation (rain-gauge) network consists of point
observations. The main problem in the precipitation product cal/val activities occurs in the dimension
disagreement between the product space (area) and the ground observation space (point). To be able to
compare both cases, either area to point (product to site) or point to area (site to product) procedure has
to be defined. However, the first alternative seems easier. The basic assumption in such an approach is
that the product value is homogenous within the product footprint. Fig. 28 presents satellite foot print
(IFOV) centres of the H02 product, an elliptical footprint for the corresponding centre (area within the
yellow dots) and AWOS ground observation sites. The comparison statistic can be performed by
considering just the sites in the footprint area. Although this approach is reasonable on the average but it
is less useful in spatial precipitation variability representation. The comparison is not possible when no
site is available within the footprint area.
Fig. 28 - H02 product footprint centres with a sample footprint area as well as the AWOS ground observation sites.
Alternatively, the point to area approach is more appealing for the realistic comparison of the
precipitation product and the ground observation. This approach is simply based on the determination of
the true precipitation field underneath the product footprint area. To do so, the footprint area is meshed
and precipitation amounts are estimated at each grid point by using the precipitation observations at the
neighbouring AWOS sites as shown in Fig. 29. A 3x3 km grid spacing is considered for the products
with elliptical geometry while 2x2 km spacing is considered for the products with rectangular geometry.
At each grid point, the precipitation amount is estimated by,
n
i
mi
n
i
imi
m
rW
ZrW
Z
1
,
1
,
)(
)(
(1)
where Zm is the estimated value and W(ri,m) is the spatially varying weighting function between the i-th
site and the grid point m.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 38
Fig. 29 - Meshed structure of the sample H02 product footprint.
Determination of the W(ri,m) weighting function in Equation 1 is crucial. In open literature, various
approaches are proposed for determining this function. For instance, Thiebaux and Pedder 198710
suggested weightings in general as,
Rrfor
RrforrR
rR
rW
mi
mi
mi
mi
mi
,
,2
,
2
2
,
2
,
0
)( (2)
where R is the radius of influence, r is the distance from centre to the point and is a power parameter
that reflects the curvature of the weighting function. Another form of geometrical weighting function
was proposed by Barnes 196411 as,
R
rrW
mi
mi
,
, 4exp)( (3)
Unfortunately none of these functions are observation dependent but suggested on the basis of the
logical and geometrical conceptualizations only. They are based only on the configuration, i.e. geometry
of the measurement stations and do not take into consideration the natural variability of the
meteorological phenomenon concerned. In addition, the weighting functions are always the same from
site to site and time to time. However, in reality, it is expected that the weights should reflect to a certain
extent the regional and temporal dependence behaviour of the phenomenon concerned.
For the validation activities, the point cumulative semi-variogram technique proposed by Şen and Habib
199812 is used to determine the spatially varying weighting functions. In this approach, the weightings
not only vary from site to site, but also from time to time since the observed data is used. In this way,
the spatial and temporal variability of the parameter is introduced more realistically to the validation
activity.
10
Thiebaux H.J. and M.A. Pedder, 1987: “Spatial objective analysis”. Academic Press, 299 pp. 11
Barnes S.L., 1964: “A technique for maximizing details in numerical weather map analysis”. J. App. Meteor., 3,
pp.396-409. 12
Şen Z. and Z. Habib, 1998: “Point cumulative semivariogram of areal precipitation in mountainous regions”. Journal
of Hydrology 205 (1–2), 81–91.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 39
4. Validation of the product release as at the end of the Development Phase
4.1 Introduction
This Chapter collects the results of the validation experiments as at the end of the H-SAF Development
Phase. The validation is performed on the product release in force at the time of the Operations
Readiness Review (ORR). The organisation was as follows:
The validation period is 1st January 2009 to 31 March 2010, in order to cover all seasons with
margins. Granularity: one month.
The results of the previous validation cycles, including case studies, are recorded in the Appendix to
this PVR-06, reproduced from the so-called “REP-3 (H-SAF Products Validation Report)”, last
issue dated 28 February 2010. The Appendix is a simple transcription of the experiments, thus it is
characterised by a low level of editing, as it was appropriate to a project-internal working document.
REP-3 was split in volumes, each H-SAF product being addressed by one volume. PR-ASS-1 was
addressed by REP-3/07 (Results of validation activities for product PR-ASS-1). However, only
internal tests made in the framework of the COSMO Consortium were reported there, whereas this
PVR-06 is based on results achieved by the Units of the H-SAF validation team for precipitation.
This Chapter 4 is structured by Country / Team, one section each. Each section records the main
statistical scores across the January 2009 - March 2010 period, and provides comments, possibly
supported by further analysis material. The next Chapter 5 (Overview of findings) provides comparative
features among the results from the various Countries / Teams, so that the User of the product is
informed of the variability of the performances with climatological and morphological conditions, as
well as with seasonal effects.
Unlike the previous validation campaigns, that placed priority to supporting the product development,
thus made large use of case studies, this campaign focused on the objective of the ORR of assessing the
degree of compliance of the product quality with the User requirements. For product PR-ASS-1 the
User requirements are recorded in Table 07.13 As a matter of fact, the original requirements do not
specify the range of interest for precipitation rate, nor the integration time for the accumulated
precipitation. We have complemented the requirements having regard to other precipitation rate and
accumulated precipitation products.
Table 07 - Accuracy requirements for product PR-ASS-1 [RMSE (%)]
Product threshold target optimal Additional specifications
Precipitation rate 100 50 25 To be validated for RR < 1.0, 1-10 and > 10 mm/h.
To be verified for RR in the range 1-10 mm/h
Accumulated precipitation 200 100 50 To be validated for integration over 3, 6, 12 and 24 h.
To be verified for integration over 24 h
This formulation of the requirement implies that the main score to be evaluated is the Root Mean Square
Error that, since grows with intensity or amount, is better evaluated as percent, i.e. RMSE (%).
Supportive scores are: the ordinary RMSE (mm), the Mean Error (or bias, ME), and the Standard
Deviation (SD). In addition, the Correlation Coefficient (CC), the Probability Of Detection (POD), the
False Alarm Rate (FAR) and the Critical Success Index (CSI, necessary to compare POD - FAR
coupled scores), also are reported, although rather unstable quantities for this type of geophysical
parameter that does not at all comply with Gaussian characteristics.
Each Country / Team should conclude its Section by listing the main features of the product, function of
whatever the Team considers as a significant change of conditions associated to change of performance.
The purpose is to characterise the applicability of the product for a correct use, especially in hydrology.
13
There is evidence that the user requirements for precipitation observation from space adopted by authoritative bodies
(WMO, EUMETSAT, the GPM planning board) are overstated. However, currently another reference is not available.
The situation will be re-assessed during CDOP-1.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 40
4.2 Validation in Belgium (IRM)
The facilities available to IRM, and the methodology adopted for validation, are described in section
3.4.1. The ground truth is provided by meteorological radar.
In respect of product PR-ASS-1, IRM has validated only the 24-hour accumulated precipitation. The
results are reported in Table 08. It is noted that in February and March 2010 the validation could not be
performed because of unavailability of the radar.
Table 08 - Summary results of PR-ASS-1 validation in Belgium by IMR
H06 v1.0 Belgium Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Radar Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 9952 15604 16595 12504 15288 13576 22295 9602 6468 13383 10204 21066 11321 16256 12595
ME (mm) -0.37 -0.61 -3.28 -2.56 -1.22 -1.76 -2.84 -5.19 -1.95 -1.68 0.27 -0.50 -0.17 -0.04 -0.38
SD (mm) 6.03 5.25 11.7 9.87 5.19 7.32 6.71 10.9 3.89 6.07 5.88 4.62 3.75 6.04 5.46
RMSE (mm) 6.61 4.78 12.8 11.3 6.78 11.4 8.29 12.8 5.74 7.02 4.36 5.03 3.80 5.40 6.19
24-h cumulate RMSE (%) 142 134 105 202 155 332 137 182 134 170 137 127 116 133 192
CC 0.46 0.61 0.03 0.16 0.28 0.15 0.21 0.04 0.16 0.24 0.79 0.34 0.59 0.54 0.31
POD 0.75 0.73 0.84 0.63 0.61 0.58 0.53 0.35 0.68 0.68 0.78 0.83 0.75 0.90 0.83
FAR 0.37 0.41 0.21 0.31 0.35 0.28 0.28 0.49 0.18 0.18 0.31 0.26 0.54 0.35 0.27
CSI 0.52 0.48 0.69 0.49 0.46 0.47 0.44 0.26 0.59 0.60 0.58 0.64 0.40 0.61 0.64
The monthly values of the Mean Error (ME) and of the Root Mean Square Error (RMSE) are shown in
Fig. 30 and Fig. 31, respectively. The ME shows an underestimation for all months except one. The
results are better during winter. The RMSE is also lower during winter months. The ME and RMSE for
winter are of the same order as found by Roulin and Vannitsem (2005) when analyzing the ECMWF
EPS forecasts for a catchment situated in the validation area of the present study. For the summer
season, the ME and RMSE were lower (|ME| < 1 mm day-1 and RMSE < 7 mm day-1). However, the
validation methodology was different: precipitation was first areally averaged and scores were
computed over 6-months season).
Mean Error
-6,00
-5,00
-4,00
-3,00
-2,00
-1,00
0,00
1,00
200901 200903 200905 200907 200909 200911 201001 201003
Fig. 30 - Mean Error of PR-ASS-1, 24h accumulated precipitation (monthly values over Belgium in mm day-1).
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 41
RMSE
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
200901 200903 200905 200907 200909 200911 201001 201003
Fig. 31 - Root Mean Square Error of PR-ASS-1, 24 accumulated precipitation (monthly values over Belgium in mm day-1).
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 42
4.3 Validation in Germany (BfG)
The facilities available to BfG, and the methodology adopted for validation, are described in section
3.4.2. The ground truth is provided by meteorological radar.
BfG has performed full validation of PR-ASS-1, i.e. the precipitation rate and the accumulated
precipitation over 3, 6, 12 and 24 hours. The results are reported in Table 09 for the rate and Table 10
for the accumulated.
Table 09 - Summary results of the validation of PR-ASS-1rate in Germany by BfG
H06 v1.0 Germany Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h
Radar Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
Number of > 10 mm/h 0 0 43 18 1532 1458 3550 1068 879 708 122 70 0 32 167
comparisons 1-10 mm/h 40846 56237 88986 30613 63647 91683 113267 41003 40853 66682 118869 110608 33988 58379 58068
(ground obs.) < 1 mm/h 143498 305101 275032 64891 88870 100240 95569 36959 48806 147546 239426 294369 258754 235152 176178
> 10 mm/h - - -10.5 -11.6 -17.1 -13.2 -15.7 -14.3 -14.5 -12.3 -11.5 -12.1 - -13.7 -11.4
ME (mm/h) 1-10 mm/h -1.10 -0.92 -0.96 -1.07 -1.67 -1.71 -1.84 -2.20 -1.82 -1.37 -1.12 -1.02 -0.90 -0.99 -1.16
< 1 mm/h -0.30 -0.30 -0.26 -0.18 -0.11 -0.12 -0.17 -0.19 -0.17 -0.15 -0.20 -0.25 -0.34 -0.33 -0.27
> 10 mm/h - - 0.69 2.70 15.7 5.66 8.99 7.76 5.12 3.33 2.31 1.30 - 2.13 5.30
SD (mm/h) 1-10 mm/h 1.04 0.86 1.99 1.50 2.03 1.98 2.53 2.11 1.86 1.80 1.33 1.10 0.77 0.95 1.34
< 1 mm/h 0.49 0.44 0.77 0.86 1.01 1.07 1.30 1.23 0.90 0.84 0.69 0.58 0.38 0.52 0.61
> 10 mm/h - - 10.6 11.9 23.2 14.4 18.1 16.2 15.4 12.8 11.7 12.1 - 13.85 12.60
RMSE (mm/h) 1-10 mm/h 1.51 1.26 2.21 1.84 2.62 2.62 3.13 3.05 2.60 2.26 1.74 1.50 1.18 1.37 1.77
< 1 mm/h 0.58 0.53 0.82 0.88 1.01 1.07 1.31 1.25 0.91 0.85 0.71 0.63 0.51 0.62 0.67
> 10 mm/h - - 98 95 92 93 93 97 96 94 96 100 - 100 86
RMSE (%) 1-10 mm/h 87 82 167 101 101 101 130 108 96 95 84 83 83 90 89
< 1 mm/h 118 109 134 181 218 228 282 262 197 186 148 128 101 110 139
> 10 mm/h - - -0.04 -0.30 -0.04 0.16 0.07 -0.06 -0.02 0.00 -0.19 -0.03 - - -0.03
CC 1-10 mm/h 0.14 0.12 0.05 0.09 0.14 0.07 0.08 0.02 0.05 0.11 0.14 0.11 0.19 0.13 0.25
< 1 mm/h 0.14 0.18 0.11 0.09 0.06 0.04 0.02 0.02 0.04 0.09 0.13 0.16 0.16 0.13 0.13
POD ≥ 0.25 mm/h 0.19 0.20 0.21 0.21 0.21 0.21 0.18 0.12 0.17 0.25 0.29 0.25 0.17 0.18 0.21
FAR ≥ 0.25 mm/h 0.67 0.68 0.67 0.79 0.83 0.77 0.81 0.89 0.83 0.75 0.60 0.63 0.70 0.64 0.74
CSI ≥ 0.25 mm/h 0.14 0.14 0.15 0.12 0.11 0.13 0.10 0.06 0.09 0.14 0.20 0.18 0.12 0.13 0.13
The time evolution of the statistical scores are depicted in Fig. 32, Fig. 33 and Fig. 34.
Fig. 32 - Time evolution of Mean Error and Standard Deviation for the three precipitation categories.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 43
Fig. 33 - Time evolution of the Root Mean Square Error (mm/h and %) for the three precipitation categories.
Fig. 34 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index.
Comments on PR-ASS-1 rain rate
Strong seasonality is observable in quantitative measures of the high precipitation class (> 10 mm/h)
during 2009-2010.
High standard deviation and peak mean error (17.1 mm/h) in the high precipitation class in May
2009 were caused by at least two frontal systems with extended precipitation area that crossed
Germany and were not sufficiently described by the model data.
Mean error is negative all over the period with an average of -1.4 mm/h (-0.2 mm/h) in the mid
(low) precipitation class for the year 2009.
Normalized root mean squared error peaked in July 2009 with the low precipitation class to 282 %.
The overall average for all three classes for 2009 was 129 %.
Performance of PR-ASS-1 in terms of categorical score CSI was highest in November 2009 with
0.20 and lowest in August 2009 with 0.06 with a yearly average of 0.13.
POD and CSI showed little variability and indistinct seasonality, while FAR had a clearer summer
maximum (0.89) and fall minimum (0.60).
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 44
Table 10 - Summary results of the validation of PR-ASS-1accumulated in Germany by BfG
H06 v1.0 Germany Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Radar Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 160542 283005 316996 85757 175815 223199 283683 105681 109352 215188 338513 356182 233475 238020 205093
ME (mm) -1.35 -1.19 -1.16 -1.04 -1.55 -1.84 -1.96 -2.34 -2.01 -1.25 -1.31 -1.26 -1.28 -1.39 -1.27
SD (mm) 1.69 1.48 2.43 2.78 3.76 4.28 4.87 4.41 3.76 3.14 2.41 2.05 1.19 1.61 2.13
RMSE (mm) 2.16 1.90 2.69 2.96 4.07 4.66 5.25 4.99 4.26 3.38 2.75 2.41 1.75 2.13 2.48
03-h cumulate RMSE (%) 90 88 130 128 141 137 157 151 119 126 97 95 87 89 99
CC 0.31 0.30 0.18 0.24 0.27 0.16 0.18 0.04 0.08 0.22 0.31 0.27 0.31 0.26 0.38
POD 0.30 0.30 0.37 0.38 0.37 0.36 0.30 0.25 0.34 0.40 0.41 0.38 0.22 0.28 0.32
FAR 0.40 0.37 0.40 0.61 0.68 0.62 0.67 0.77 0.66 0.54 0.40 0.37 0.44 0.39 0.54
CSI 0.25 0.26 0.30 0.24 0.21 0.23 0.19 0.14 0.20 0.27 0.32 0.31 0.19 0.24 0.23
No. samples 272860 480812 512731 130097 292328 367745 477316 178420 185833 367259 528307 576171 434224 413197 367969
ME (mm) -1.67 -1.43 -1.30 -0.92 -1.32 -1.80 -1.83 -2.19 -1.94 -1.11 -1.40 -1.40 -1.61 -1.66 -1.32
SD (mm) 2.38 2.13 3.73 4.03 4.88 5.90 6.42 6.07 5.10 4.15 3.42 2.85 1.74 2.26 2.81
RMSE (mm) 2.91 2.57 3.95 4.14 5.06 6.17 6.67 6.46 5.45 4.29 3.69 3.18 2.37 2.81 3.11
06-h cumulate RMSE (%) 91 90 167 151 167 167 184 187 148 137 104 99 88 89 106
CC 0.37 0.38 0.21 0.33 0.34 0.23 0.25 0.08 0.10 0.31 0.40 0.38 0.36 0.33 0.42
POD 0.39 0.38 0.48 0.48 0.55 0.46 0.43 0.37 0.46 0.51 0.49 0.47 0.30 0.35 0.40
FAR 0.28 0.28 0.32 0.55 0.63 0.52 0.55 0.68 0.54 0.43 0.33 0.27 0.31 0.31 0.43
CSI 0.34 0.33 0.40 0.30 0.28 0.31 0.28 0.21 0.30 0.37 0.39 0.40 0.26 0.30 0.31
No. samples 459404 830407 861388 218171 507704 594748 817796 306216 306828 626782 891787 975725 751545 731859 662517
ME (mm) -2.18 -1.95 -1.72 -1.07 -1.08 -1.82 -1.62 -1.92 -1.70 -1.04 -1.74 -1.78 -2.10 -2.08 -1.46
SD (mm) 3.23 3.02 5.72 5.45 6.07 7.82 7.96 8.03 6.28 5.11 4.43 3.80 2.41 2.85 3.62
RMSE (mm) 3.90 3.59 5.97 5.56 6.17 8.03 8.13 8.25 6.51 5.21 4.76 4.19 3.20 3.52 3.90
12-h cumulate RMSE (%) 91 91 217 159 194 188 207 216 170 134 103 99 87 90 115
CC 0.46 0.45 0.22 0.43 0.37 0.32 0.29 0.15 0.17 0.39 0.49 0.46 0.42 0.45 0.44
POD 0.46 0.45 0.56 0.58 0.64 0.56 0.54 0.48 0.58 0.59 0.54 0.53 0.37 0.39 0.49
FAR 0.19 0.18 0.23 0.47 0.53 0.44 0.43 0.59 0.44 0.33 0.23 0.16 0.21 0.23 0.29
CSI 0.42 0.41 0.48 0.38 0.38 0.39 0.39 0.29 0.40 0.46 0.47 0.48 0.34 0.35 0.41
No. samples 734563 1278683 1377631 352453 908862 977058 1362757 539123 501401 1052271 1438542 1515607 1195796 1219091 1099085
ME (mm) -2.88 -2.78 -2.21 -1.58 -0.87 -1.80 -1.30 -1.88 -1.35 -1.11 -2.16 -2.50 -2.91 -2.67 -1.85
SD (mm) 4.44 4.34 8.57 7.15 7.48 10.10 9.77 10.52 7.71 6.22 5.59 5.23 3.59 3.81 4.73
RMSE (mm) 5.29 5.16 8.85 7.32 7.53 10.26 9.85 10.69 7.83 6.32 5.99 5.80 4.62 4.65 5.07
24-h cumulate RMSE (%) 99 92 235 181 218 202 232 252 201 140 102 98 89 90 114
CC 0.49 0.50 0.20 0.52 0.36 0.38 0.32 0.15 0.18 0.44 0.51 0.50 0.42 0.46 0.42
POD 0.53 0.53 0.64 0.65 0.66 0.64 0.64 0.58 0.64 0.65 0.60 0.57 0.47 0.44 0.59
FAR 0.14 0.12 0.13 0.42 0.41 0.34 0.29 0.50 0.38 0.25 0.15 0.10 0.13 0.14 0.17
CSI 0.48 0.50 0.59 0.44 0.45 0.48 0.51 0.37 0.46 0.53 0.55 0.54 0.44 0.41 0.52
Fig. 35 to Fig. 42 display the time evolution of the various statistical scores for the different
accumulation periods.
Fig. 35 - Time evolution of the Mean Error, function of the accumulation period.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 45
Fig. 36 - Time evolution of the Standard Deviation, function of the accumulation period.
Fig. 37 - Time evolution of the Root Mean Square Error (mm), function of the accumulation period.
Fig. 38 - Time evolution of the Root Mean Square Error (%), function of the accumulation period.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 46
Fig. 39 - Time evolution of the Correlation Coefficient, function of the accumulation period.
Fig. 40 - Time evolution of the Probability of Detection, function of the accumulation period.
Fig. 41 - Time evolution of the False Alarm Rate, function of the accumulation period.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 47
Fig. 42 - Time evolution of the Critical Success Index, function of the accumulation period.
Comments on PR-ASS-1 accumulated precipitation
Mean error in all accumulated precipitation over 3, 6, 12 and 24 hours was generally negative with a
diffuse bimodal pattern with two maxima in April/May and October and a pronounced minimum in
January 2010 in 24 h precipitation (ME -2.91 mm).
Standard deviation, root mean squared error and normalized root mean squared error all showed
highest values in summer and minima during January/February with absolute values increasing with
the length of accumulation.
Performance of POD, FAR and CSI all showed the strong dependence on accumulation length.
Usually categorical scores of 24 h accumulation were 20-35 % better than those for 3 h
accumulation. This was due to the fact that accumulation masks to a certain degree temporal and
also spatial mismatch of observed and forecast precipitation.
The critical success index varied between 0.14 in August, 3 h accumulation and 0.59 March 2009,
24 h accumulation.
Average CSI for 3 h (6 h, 12 h, 24 h) accumulated precipitation was 0.24 (0.33, 0.41, 0.49) during
the year 2009.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 48
4.4 Validation in Hungary (OMSZ)
The facilities available to OMSZ, and the methodology adopted for validation, are described in section
3.4.3. The ground truth is provided by meteorological radar.
OMSZ has performed full validation of PR-ASS-1, i.e. the precipitation rate and the accumulated
precipitation over 3, 6, 12 and 24 hours. The results are reported in Table 11 for the rate and Table 12
for the accumulated.
Table 11 - Summary results of the validation of PR-ASS-1rate in Hungary by OMSZ
H06 v1.0 Hungary Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h
Radar Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
Number of > 10 mm/h 11 3 10 73 458 1184 925 1548 486 206 86 60 17 13 31
comparisons 1-10 mm/h 6036 4550 11019 9200 20968 45070 24181 31095 13431 23450 19020 15129 9036 8964 5046
(ground obs.) < 1 mm/h 42849 39956 55566 29223 48182 73481 35385 35686 23973 57947 53587 55552 42756 45521 27527
> 10 mm/h -9.85 -10.4 -11.3 -15.8 -16.9 -17.5 -18.6 -18.0 -17.0 -13.3 -45.9 -14.5 -12.0 -9.18 -13.2
ME (mm/h) 1-10 mm/h -1.00 -1.21 -1.09 -1.82 -1.87 -2.04 -2.14 -2.08 -1.98 -1.04 -0.93 -0.84 -1.18 -0.87 -1.39
< 1 mm/h 0.08 -0.07 -0.03 -0.32 -0.21 -0.12 -0.15 -0.03 -0.24 0.30 0.18 0.17 0.06 0.16 0.00
> 10 mm/h 1.24 1.27 1.66 6.30 8.72 11.6 14.8 11.5 10.9 6.37 104 9.23 2.08 1.19 8.44
SD (mm/h) 1-10 mm/h 1.09 0.96 1.37 1.55 1.79 2.07 2.65 3.05 2.66 2.25 1.50 1.70 1.42 1.25 1.56
< 1 mm/h 0.62 0.49 0.72 0.61 0.92 1.24 1.53 2.08 1.30 1.33 0.90 0.94 0.64 0.67 0.69
> 10 mm/h 9.93 10.5 11.5 17.0 19.0 21.0 23.8 21.4 20.2 14.7 114 17.2 12.2 9.25 15.7
RMSE (mm/h) 1-10 mm/h 1.48 1.55 1.75 2.39 2.59 2.91 3.40 3.69 3.31 2.48 1.77 1.90 1.85 1.52 2.09
< 1 mm/h 0.63 0.50 0.72 0.69 0.95 1.25 1.54 2.08 1.32 1.36 0.92 0.95 0.65 0.69 0.69
> 10 mm/h 88 89 97 99 100 99 99 98 98 94 92 90 96 82 93
RMSE (%) 1-10 mm/h 74 80 85 100 104 109 136 157 158 117 85 93 78 73 86
< 1 mm/h 161 125 172 149 213 283 327 473 277 311 217 221 160 173 172
> 10 mm/h 0.16 0.40 -0.41 -0.16 -0.02 -0.06 -0.04 -0.01 0.09 -0.02 0.08 0.01 -0.09 0.47 -0.18
CC 1-10 mm/h 0.16 0.10 0.04 0.04 -0.05 -0.03 -0.02 0.01 0.00 -0.03 0.05 0.07 -0.05 0.14 0.18
< 1 mm/h 0.06 0.03 0.10 0.03 0.02 0.01 0.03 0.02 0.04 0.08 0.10 0.14 0.10 0.13 0.03
POD ≥ 0.25 mm/h 0.59 0.49 0.44 0.16 0.21 0.23 0.18 0.21 0.17 0.50 0.60 0.56 0.56 0.63 0.43
FAR ≥ 0.25 mm/h 0.84 0.91 0.87 0.82 0.80 0.80 0.85 0.76 0.81 0.77 0.77 0.83 0.86 0.83 0.90
CSI ≥ 0.25 mm/h 0.15 0.09 0.11 0.09 0.12 0.12 0.09 0.13 0.10 0.19 0.20 0.15 0.12 0.16 0.09
Fig. 43 to Fig. 46 display the time evolution of the various statistical scores.
Fig. 43 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for medium and light precipitation.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 49
Fig. 44 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for heavy precipitation.
Fig. 45 - Time evolution of the Correlation Coefficient for the three categories of precipitation.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 50
Fig. 46 - Time evolution of Probability of Detection, False Alarm Rate and Critical Success Index for rain rate.
We can conclude from the graphs about the H06 instantaneous scores that:
There is seasonal dependence in the Mean Error. For the Category <1 mm/h, there is slight
overestimation, except for the months April-May-June-July-August-September, where there is slight
underestimation.
For the classes 1-10 mm/h and >10 mm/h, there is always an underestimation, just as in case of H03,
which gets very large in the case of higher rain rates. This means that the H06 can neither capture
well the high intensities.
ME, SD and RMSE values are very high for the case of precipitation >10 mm/h in November 2009.
This can be due to non-expected deficiencies of the algorithms which were corrected afterwards,
because for other months, the scores are stable.
The Standard Deviation and the Root Mean Square Error follow the tendencies of the Mean Error.
However, they increase in the period April-May-June-July-August-September, where the ME for
Category <1mm decreases under zero (for Category 1-10 mm/h, the ME also decreases towards
larger negative values in this period). The reason for the increase in SD and RMSE are larger
deviations from the radar-measured rain rates. Underestimations are general in this period when
convection is quite usual.
We can depict the seasonal dependence only in the POD values, but not in the Correlation
Coefficients and the FAR, CSI values. Probability of Detection is the lowest in summer months,
contrary to the H03 product.
The Correlation Coefficient is very low in case of precipitation of intensity of <1 mm/h. However,
there is a slight improvement in the CC during the period, becoming higher than 0.1 in January and
February 2010. The Correlation Coefficient is the lowest for the summer months for the Rain Rate
Category 1-10 mm/h. There is big variability for the Category >10mm/h.
The False Alarm Rate is very high, always around 0.8.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 51
Table 12 - Summary results of the validation of PR-ASS-1accumulated in Hungary by OMSZ
H06 v1.0 Hungary Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Radar Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 155936 283005 294636 30297 175068 223199 276881 105681 106133 213099 250895 201748 33400 34164 18941
ME (mm) -1.35 -1.19 -1.11 -0.75 -1.45 -1.57 -1.69 -2.07 -1.77 -1.11 -1.11 -1.20 0.06 0.32 -0.30
SD (mm) 1.71 1.48 2.46 3.06 3.81 4.25 4.90 4.52 3.84 3.13 2.40 1.95 2.25 2.18 2.31
RMSE (mm) 2.18 1.90 2.70 3.15 4.08 4.53 5.19 4.97 4.23 3.32 2.64 2.29 2.25 2.21 2.33
03-h cumulate RMSE (%) 90 88 132 125 143 140 163 157 123 126 99 96 140 152 152
CC 0.31 0.30 0.19 0.27 0.27 0.21 0.20 0.07 0.07 0.23 0.38 0.33 0.16 0.27 0.10
POD 0.61 0.47 0.53 0.18 0.24 0.30 0.26 0.30 0.29 0.56 0.66 0.64 0.60 0.69 0.47
FAR 0.87 0.95 0.89 0.86 0.83 0.80 0.83 0.74 0.75 0.78 0.79 0.83 0.88 0.85 0.92
CSI 0.12 0.05 0.10 0.09 0.11 0.13 0.12 0.16 0.16 0.19 0.19 0.15 0.11 0.14 0.07
No. samples 30101 24534 41144 25224 51203 99212 55775 60864 30136 57339 49345 48531 35002 35364 20441
ME (mm) 1.05 0.13 0.31 -1.64 -2.11 -1.31 -1.72 -2.11 -1.50 0.74 0.64 1.04 0.69 0.72 0.15
SD (mm) 3.67 2.75 3.23 3.36 4.08 5.89 6.26 8.52 5.25 5.77 4.03 3.92 3.09 3.03 3.37
RMSE (mm) 3.81 2.75 3.24 3.74 4.59 6.03 6.49 8.77 5.46 5.82 4.08 4.06 3.17 3.11 3.37
06-h cumulate RMSE (%) 282 177 199 158 148 229 218 251 189 314 215 218 190 201 212
CC 0.16 0.09 0.28 0.05 0.02 0.09 0.10 0.12 0.30 0.18 0.29 0.38 0.29 0.35 0.07
POD 0.70 0.56 0.56 0.25 0.28 0.45 0.37 0.41 0.38 0.62 0.73 0.72 0.72 0.74 0.52
FAR 0.81 0.90 0.83 0.77 0.75 0.68 0.74 0.64 0.70 0.71 0.71 0.78 0.82 0.79 0.88
CSI 0.18 0.09 0.15 0.14 0.15 0.23 0.18 0.24 0.20 0.25 0.26 0.20 0.17 0.20 0.11
No. samples 26695 25112 40097 23789 47780 86184 50322 51250 26115 48872 44151 43084 30292 31150 20063
ME (mm) 2.50 1.59 1.36 -1.56 -1.59 -1.15 -1.73 -2.09 -2.22 2.42 2.21 2.27 2.01 2.96 1.16
SD (mm) 4.55 3.95 4.33 3.93 5.48 7.70 6.71 10.1 6.34 9.00 5.42 5.39 4.08 4.60 4.78
RMSE (mm) 5.19 4.26 4.53 4.23 5.71 7.79 6.92 10.4 6.72 9.32 5.85 5.85 4.55 5.47 4.92
12-h cumulate RMSE (%) 310 288 263 167 191 279 216 257 192 398 303 317 287 334 278
CC 0.28 0.21 0.37 0.07 0.19 0.07 0.12 0.18 0.26 0.34 0.34 0.41 0.27 0.34 0.14
POD 0.85 0.77 0.71 0.37 0.35 0.53 0.45 0.51 0.37 0.68 0.81 0.82 0.85 0.90 0.67
FAR 0.76 0.85 0.78 0.65 0.68 0.57 0.63 0.52 0.64 0.67 0.66 0.75 0.78 0.74 0.84
CSI 0.23 0.14 0.20 0.22 0.20 0.31 0.26 0.33 0.22 0.29 0.31 0.24 0.21 0.25 0.14
No. samples 23791 23595 33834 19286 44022 71123 47566 44613 23350 42872 36768 37770 24846 27197 16936
ME (mm) 4.94 3.25 2.64 -1.23 -1.79 -0.51 -1.46 -1.76 -2.12 4.47 4.42 4.28 2.86 5.13 1.91
SD (mm) 7.37 5.54 6.09 5.41 5.86 9.49 7.97 12.3 7.62 12.3 7.08 8.01 5.20 5.87 7.20
RMSE (mm) 8.87 6.42 6.64 5.55 6.12 9.50 8.10 12.4 7.91 13.1 8.35 9.08 5.94 7.79 7.45
24-h cumulate RMSE (%) 430 404 349 204 229 305 242 274 187 494 408 445 329 438 358
CC 0.41 0.18 0.49 0.16 0.14 0.19 0.10 0.17 0.33 0.37 0.39 0.44 0.35 0.48 0.23
POD 0.89 0.89 0.79 0.50 0.46 0.63 0.54 0.59 0.44 0.77 0.90 0.86 0.92 0.95 0.66
FAR 0.72 0.79 0.75 0.57 0.58 0.47 0.54 0.46 0.58 0.62 0.60 0.70 0.75 0.72 0.84
CSI 0.27 0.20 0.23 0.30 0.28 0.41 0.33 0.39 0.27 0.35 0.38 0.29 0.24 0.28 0.14
Fig. 47 (four panels) display the time evolution of Mean Error, Standard Deviation and Root Mean
Square Error for integration intervals 3, 6, 12 and 24 hours. Fig. 48 provides the same information for
the Correlation Coefficient.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 52
Fig. 47 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for the integration intervals 3, 6, 12 and 24 hours.
Fig. 48 - Time evolution of the Correlation Coefficient for the integration intervals 3, 6, 12 and 24 hours.
We can conclude from the graphs about the H06 accumulated continuous scores that:
There is seasonal dependence in the Mean Error. Except for the 3-hourly accumulations, for all the
accumulations there is general overestimation in the winter-spring-autumn seasons, while for the
months April-May-June-July-August-September, there is slight underestimation. This can be due to
convective rain rates, which are very high, and which are underestimated by the H06 product.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 53
The 3-, 6-, 12-, and 24-hourly accumulations have similar tendencies in the Mean Error, SD and
RMSE values through the periods. As the accumulation period gets longer, these values also
increase. the biggest variability is for the 24-hourly accumulated product.
The Standard Deviation and the Root Mean Square Error are the highest in July, August for all the
accumulation periods.
The Standard Deviation and the Root Mean Square Error follow the tendencies of the Mean Error.
The SD and RMSE increase in the period where the ME decreases. The reasons for this increase are
larger deviations from the radar-measured rain rates. Underestimations are general in this period
when convection is quite usual.
The four frames of Fig. 49 display the time evolution of Probability Of Detection, False Alarm Rate and
Critical Success Index for integration intervals 3, 6, 12 and 24 hours, respectively.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 54
Fig. 49 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index for integration intervals 3, 6, 12 and 24 hours (four panels).
We can conclude from the graphs about the H06 accumulated multi-categorical scores that:
We can depict the seasonal dependence clearly in the POD values, a little bit in the FAR values
(especially in the 24 hourly-accumulations), whereas the CSI values are stable.
The 3-, 6-, 12-, and 24-hourly accumulations have similar tendencies in the POD, FAR and CSI
values through the periods. As the accumulation period gets longer, these values ameliorate. The
best scores are found for the 24-hourly accumulated product.
The False Alarm Rate is quite high, always above 0.5. The Probability of Detection decreases for
the summer where there are convective cells. Thus, it is probable that convection is not very well
depicted by H06.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 55
4.5 Validation in Italy (UniFe)
The facilities available to the University of Ferrara, and the methodology adopted for validation, are
described in section 3.4.4. The ground truth is provided by rain gauge networks.
UniFe has performed validation of PR-ASS-1 in respect of rain rate and of accumulation over 3 and 24
hours. The results are reported in Table 13 for the rate and Table 14 for the accumulated.
Table 13 - Summary results of the validation of PR-ASS-1rate in Italy by UniFe
H06 v1.0 Italy Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h
Gauge Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
Number of > 10 mm/h 3145 821 1504 1927 933 2366 1868 1126 4082 3178 2396 1728 1615 1274 890
comparisons 1-10 mm/h 94600 62019 77398 85880 16435 39687 16268 15112 46733 55093 63842 89773 81596 85927 55355
(ground obs.) < 1 mm/h 102312 68213 71788 82350 18740 38255 13903 15909 39579 50865 59332 98533 100090 104304 71646
> 10 mm/h -12.5 -13.3 -21.4 -12.3 -18.1 -15.4 -15.8 -15.6 -16.2 -15.4 -12.0 -11.2 -13.6 -12.4 -12.1
ME (mm/h) 1-10 mm/h -1.52 -1.65 -1.61 -1.46 -2.24 -2.04 -2.57 -2.60 -2.31 -1.82 -1.58 -1.37 -1.51 -1.43 -1.01
< 1 mm/h -0.02 -0.15 -0.02 0.04 -0.25 -0.04 -0.21 -0.14 -0.07 -0.04 0.01 -0.02 -0.02 -0.02 0.09
> 10 mm/h 10.9 7.18 32.4 6.90 10.3 8.56 8.07 7.44 8.21 11.21 7.94 12.47 12.74 9.57 13.9
SD (mm/h) 1-10 mm/h 2.19 1.91 2.12 2.26 2.33 2.80 2.90 2.47 2.88 2.93 2.47 2.05 2.08 1.90 2.16
< 1 mm/h 1.06 0.91 1.08 1.23 0.94 1.65 1.26 1.60 1.66 1.47 1.26 1.02 1.03 0.94 1.10
> 10 mm/h 16.6 15.1 38.8 14.1 20.8 17.6 17.8 17.3 18.2 19.1 14.4 16.7 18.7 15.7 18.4
RMSE (mm/h) 1-10 mm/h 2.67 2.53 2.66 2.69 3.24 3.46 3.88 3.58 3.69 3.45 2.93 2.47 2.57 2.37 2.38
< 1 mm/h 1.06 0.92 1.08 1.23 0.98 1.65 1.28 1.61 1.67 1.47 1.26 1.02 1.03 0.94 1.10
> 10 mm/h 80 89 90 88 98 95 97 97 95 93 85 77 86 85 76
RMSE (%) 1-10 mm/h 98 92 93 100 104 116 119 111 127 122 100 91 95 87 97
< 1 mm/h 231 195 240 268 218 369 292 373 373 350 275 219 223 209 242
> 10 mm/h -0.03 0.01 -0.17 -0.06 0.02 -0.02 0.06 0.02 0.06 -0.09 -0.04 -0.09 -0.06 -0.10 -0.14
CC 1-10 mm/h 0.24 0.17 0.16 0.21 0.07 0.07 0.05 0.01 0.07 0.13 0.20 0.29 0.18 0.25 0.29
< 1 mm/h 0.09 0.07 0.11 0.07 0.04 0.03 0.01 0.01 0.03 0.06 0.11 0.11 0.10 0.11 0.10
POD ≥ 0.25 mm/h 0.50 0.39 0.48 0.47 0.22 0.28 0.18 0.17 0.25 0.35 0.47 0.50 0.47 0.53 0.56
FAR ≥ 0.25 mm/h 0.38 0.42 0.34 0.40 0.63 0.59 0.72 0.79 0.59 0.47 0.33 0.37 0.41 0.37 0.41
CSI ≥ 0.25 mm/h 0.38 0.31 0.38 0.36 0.16 0.20 0.12 0.10 0.18 0.27 0.38 0.39 0.35 0.40 0.40
Table 14 - Summary results of the validation of PR-ASS-1accumulated in Italy by UniFe
H06 v1.0 Italy Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Gauge Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 205174 136831 165062 180447 38282 92678 41646 40206 105765 121412 127005 193015 183115 189603 127370
ME (mm) -2.16 -2.24 -2.51 -1.89 -3.84 -3.35 -4.83 -4.04 -4.40 -3.02 -2.37 -1.81 -1.88 -1.69 -1.09
SD (mm) 6.15 4.46 8.05 5.01 6.74 7.03 8.08 6.40 9.02 7.42 5.85 5.22 5.06 2.91 4.70
RMSE (mm) 6.52 4.99 8.43 5.35 7.76 7.79 9.42 7.57 10.03 8.01 6.31 5.53 5.39 4.76 4.83
03-h cumulate RMSE (%) 130 111 113 128 109 157 134 146 151 140 120 118 123 109 1.28
CC 0.40 0.31 0.16 0.38 0.15 0.17 0.14 0.09 0.15 0.24 0.40 0.43 0.40 0.31 0.16
POD 0.53 0.43 0.53 0.53 0.27 0.38 0.29 0.24 0.33 0.44 0.55 0.55 0.52 0.58 0.59
FAR 0.38 0.40 0.32 0.40 0.61 0.56 0.63 0.76 0.55 0.42 0.31 0.35 0.39 0.36 0.41
CSI 0.40 0.34 0.43 0.39 0.19 0.26 0.19 0.14 0.24 0.34 0.44 0.42 0.39 0.44 0.42
No. samples 676668 474214 512135 608511 181294 386551 216405 232681 415630 439921 415068 652191 627867 664055 445230
ME (mm) -1.67 -2.66 -2.67 -0.23 -3.93 -3.76 -4.60 -3.94 -5.52 -3.69 -2.21 -1.56 -1.37 -1.07 0.78
SD (mm) 14.8 10.7 15.1 12.4 8.8 13.4 12.5 10.2 18.3 14.6 12.8 11.8 12.1 10.1 11.3
RMSE (mm) 14.9 11.0 15.3 12.4 9.67 13.9 13.3 10.9 19.1 15.0 13.0 11.9 12.1 10.1 11.3
24-h cumulate RMSE (%) 263 186 181 232 154 238 226 236 250 218 194 224 214 192 269
CC 0.56 0.58 0.51 0.69 0.31 0.45 0.39 0.26 0.43 0.50 0.64 0.64 0.54 0.59 0.64
POD 0.77 0.67 0.72 0.75 0.45 0.61 0.53 0.53 0.39 0.59 0.68 0.76 0.74 0.81 0.78
FAR 0.20 0.19 0.16 0.20 0.43 0.29 0.82 0.33 0.48 0.28 0.19 0.16 0.22 0.19 0.22
CSI 0.65 0.58 0.63 0.63 0.34 0.49 0.16 0.42 0.29 0.48 0.59 0.66 0.61 0.68 0.64
A preliminary analysis of the h06 performance is carried on by comparing the Probability Density
Functions (PDFs) of simulated and measured rainrates and rain amount cumulated over different
sampling times. In Fig. 50 the PDFs for simulated and measured rain patterns for January (top panels)
and July (bottom panels) 2009 are plotted in order to also highlight seasonal features for instantaneous
(left panels), 3 hourly (middle panels) and 24 hourly (right panels) cumulated rain fields.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 56
Fig. 50 - Probability Density Function for instantaneous rain rate and accumulation periods 3 h and 24 h for the two months January and July 2009, for satellite-derived and ground measurements.
The instantaneous PDFs shows that the model is able to describe the rain rate dynamics, with a slight
underestimation, for January, of mid-high rates, while the underestimation for July is effective
especially for highest rainrates. The difference between winter and summer is also clear looking at POD
FAR and CSI values, relatively better in winter than in summer. This can be due to the well known
difficulties of regional models in predict the correct position and timing of small and medium-scale
convective systems that characterizes the summertime precipitation over Italy.
The 3 hourly cumulated PDFs show model underestimation for moderate and high rain amounts (above
10 mm) for both the considered months. The 24 hourly cumulated curves show very good matching
below 60 mm in January and underestimation for higher rain amounts, while in July there is
underestimation between 10 and 75 mm and overestimation above 75 mm.
The cumulation over 3 and 24 hours progressively increases the POD and CSI values, keeping a marked
difference between winter (generally better) and summertime values of these indices.
A deeper analysis of the instantaneous rain field is carried on with the help of Fig. 51, where the values
of the relevant statistical parameters are plotted as function of the month for the year 2009.
The Mean Error (black solid line) is always negative, indicating overall underestimation of the rainrate,
with worst performances during summer months. The Multiplicative Bias is, therefore, well below 1,
and reaching the lowest value in July.
Also RMSE and NRMSE show a weak seasonal signal, with higher values in warm months.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 57
Fig. 51 - Time evolution of continuous statistical scores for the precipitation intensity.
The correlation coefficient is plotted in Fig. 52 as function of the months, for the instantaneous (black
line), 03 (red line) and 24 (green line) hours cumulated rain fields. The correlation coefficient, as most
of the other parameters, reaches higher values in cold months: cumulation over different time ranges
markedly increases the values.
Other parameters such as ME, RMSE and NRMSE cannot be compared for different cumulation ranges,
given the different ranges of rain amount values. Increasing the length of the cumulation window,
however, it can be seen that the seasonal sensitivity of the parameters decreases: the cumulation
mitigates the small scale convection timing/placing errors of the model.
Fig. 52 - Time evolution of the Correlation Coefficient for instantaneous rain rate and integration intervals 3 and 24 hours.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 58
4.6 Validation in Poland (IMWM)
The facilities available to IMWM, and the methodology adopted for validation, are described in section
3.4.5. The ground truth is provided by rain gauge networks.
IMWM has performed validation of PR-ASS-1 in respect of rain rate and of accumulation over 3 hours.
The results are reported in Table 15 for the rate and Table 16 for the accumulated.
Table 15 - Summary results of the validation of PR-ASS-1rate in Poland by IMWM
H06 v1.0 Poland Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h
Gauge Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
Number of > 10 mm/h 0 0 10 9 133 231 227 119 65 56 1 6 0 1 7
comparisons 1-10 mm/h 404 714 2323 172 2664 3653 1974 1862 907 4165 2026 1161 663 453 871
(ground obs.) < 1 mm/h 1661 3225 4192 232 2788 4247 2220 1692 1206 6290 2997 2559 2609 1840 1959
> 10 mm/h - - -13.4 -18.6 -18.4 -21.8 -24.5 -19.7 -18.2 -37.3 -10.2 -20.2 - -24.6 -14.2
ME (mm/h) 1-10 mm/h -1.05 -0.99 -1.12 -2.47 -1.78 -1.94 -2.16 -1.78 -2.01 -1.16 -0.82 -0.84 -1.11 -0.81 -1.37
< 1 mm/h -0.28 -0.09 -0.12 -0.37 -0.11 -0.21 -0.18 0.10 -0.27 -0.16 -0.02 -0.06 -0.13 -0.12 -0.20
> 10 mm/h - - 4.28 9.16 14.3 18.7 20.2 26.7 10.0 112 0.00 18.4 - 0.00 4.14
SD (mm/h) 1-10 mm/h 0.80 0.87 1.14 2.13 2.13 2.33 2.90 2.94 2.33 1.49 1.10 1.08 0.99 0.89 1.43
< 1 mm/h 0.35 0.45 0.49 0.82 1.47 1.02 1.31 2.63 0.99 0.66 0.63 0.57 0.4 0.46 0.52
> 10 mm/h - - 14.1 20.8 23.3 28.7 31.7 33.2 20.7 118 10.2 27.3 - 24.6 14.8
RMSE (mm/h) 1-10 mm/h 1.32 1.32 1.60 3.26 2.77 3.03 3.62 3.44 3.07 1.89 1.37 1.37 1.5 1.2 1.98
< 1 mm/h 0.45 0.46 0.50 0.90 1.47 1.04 1.32 2.63 1.03 0.68 0.63 0.57 0.42 0.47 0.55
> 10 mm/h - - 93 100 100 98 98 95 99 95 100 95 - 100 97
RMSE (%) 1-10 mm/h 75 68 70 91 100 103 138 162 110 80 65 71 72 64 79
< 1 mm/h 78 85 91 152 253 188 253 449 194 125 119 107 78 87 98
> 10 mm/h - - -0.61 0.69 -0.02 -0.04 -0.04 -0.09 -0.05 -0.09 - -0.15 - - -0.01
CC 1-10 mm/h -0.02 0.02 -0.01 -0.16 0.03 0.01 -0.03 -0.01 -0.01 0.06 0.12 0.17 -0.04 -0.09 0.01
< 1 mm/h 0.03 0.02 0.06 0.06 0.03 0.02 0.01 0.03 0.00 0.04 0.06 0.09 0.08 0.07 0.06
POD ≥ 0.25 mm/h 0.56 0.74 0.68 0.22 0.43 0.34 0.26 0.38 0.27 0.54 0.69 0.68 0.7 0.69 0.58
FAR ≥ 0.25 mm/h 0.85 0.79 0.72 0.89 0.60 0.68 0.79 0.59 0.73 0.54 0.63 0.77 0.81 0.80 0.78
CSI ≥ 0.25 mm/h 0.14 0.19 0.25 0.08 0.26 0.20 0.13 0.25 0.16 0.33 0.32 0.21 0.18 0.18 0.19
Table 16 - Summary results of the validation of PR-ASS-1accumulated in Poland by IMWM
H06 v1.0 Poland Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Gauge Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 1637 3215 7361 654 8358 12569 7657 5606 3427 12397 5821 3856 2559 1903 3059
ME (mm) -0.73 0.02 -0.60 -2.93 -2.02 -2.72 -3.31 -2.62 -2.40 -1.18 -0.33 -0.22 -0.26 -0.02 -0.78
SD (mm) 7.93 1.37 1.97 9.75 6.54 5.22 7.75 5.97 4.54 6.84 2.36 1.99 1.68 1.81 1.77
RMSE (mm) 7.96 1.37 2.08 10.18 6.85 5.89 8.43 6.55 5.13 6.94 2.39 2.00 1.70 1.81 1.94
03-h cumulate RMSE (%) 73 86 77 90 111 114 138 181 117 91 92 99 82 88 80
CC -0.01 0.18 0.18 0.03 0.11 0.09 0.02 0.16 0.22 0.09 0.27 0.3 0.07 0.11 0.26
POD 0.52 0.73 0.65 0.25 0.44 0.35 0.30 0.39 0.37 0.54 0.69 0.71 0.67 0.74 0.52
FAR 0.80 0.76 0.63 0.81 0.47 0.60 0.70 0.48 0.53 0.44 0.54 0.66 0.79 0.76 0.71
CSI 0.17 0.22 0.31 0.12 0.32 0.23 0.18 0.29 0.26 0.38 0.38 0.30 0.19 0.22 0.23
Analysis of rain rate
The PR-ASS-1 v.1.0 rain rate product has been validated against automatic rain gauges data. Polish
network of automatic rain gauges consists of 430 posts located all over the country, however, the
network density increases in the Southern Poland, where the flood danger is very high. Each post is
equipped with two gauges: heated and non-heated, what enables some quality control of data. For
validation purposes, readings from both gauges were compared in order to eliminate the cases of
clogged instruments. If both gauges worked properly, higher values was taken (automatic RG are known
to underestimate the real precipitation).
The measurements time resolution is 10 minutes, what allows estimating the rain rate with reasonable
quality, especially for stratiform rainfalls. The ground rain rate (in mm/h) was calculated from 10
minute cumulative values from the timeslot closest to the satellite overpass assuming that the real
precipitation rate was constant within that time span.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 59
In order to combine satellite products with rain gauges data, the following simple method was applied.
For each satellite pixels, the automatic posts situated within that pixel were found. If more than one rain
gauge were found within one satellite pixel, the ground rain rate value was calculated as a mean of all
rain gauges measurements within that pixel.
Results
Following the methodology agreed in the H-SAF precipitation validation group, both continuous and
categorical statistics were calculated on the monthly mean basis for three precipitation categories:
- category 1: >10 mm/h,
- category 2: 1-10 mm/h,
- category 3: <1 mm/h
In Fig. 53 and Fig. 54, values of the Mean Error, and RMSE % calculated for the three categories and
for each month of the analysed period are presented. It should be pointed out here that the analysis was
performed only for situations when the ground rain rate was 0.25 mm/h
-40
-35
-30
-25
-20
-15
-10
-5
0
5
Jan
'09
Feb
'09
Mar
'09
Apr
'09
Ma
y'0
9
Jun
'09
Jul'
09
Au
g'0
9
Sep'
09
Oct
'09
Nov
'09
Dec
'09
Jan
'10
Feb
'10
Mar
'10
ME
> 10 mm/h
1-10 mm/h
< 1 mm/h
Fig. 53 - Mean error (ME) of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar 2010 for Poland.
The PR-ASS-1 v.1.0 underestimates the measured rain rate values for the whole analysed period. The
underestimation is stronger in summer and early autumn, especially for high precipitation category (>10
mm/h). For this category, the seasonal variability of ME is the strongest.
0
50
100
150
200
250
300
350
400
450
500
Jan'
09
Feb'
09
Mar
'09
Apr
'09
May
'09
Jun'
09
Jul'0
9
Aug
'09
Sep'
09
Oct
'09
Nov
'09
Dec
'09
Jan'
10
Feb'
10
Mar
'10
RMSE %
> 10 mm/h
1-10 mm/h
< 1 mm/h
Fig. 54 - RMSE % of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar 2010 for Poland.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 60
Similarly to ME, the quality of the product in rain rate estimation also reveals seasonal variability: it
decreases in spring and summer and increases in late autumn and winter. However, amplitude of these
changes is different for each category i.e. the smallest for high precipitation category (> 10 mm/h) and
the biggest for light precipitation (<1 mm/h). The last one includes light rain cases that are were not
detected by PR-ASS-1 v.1.0 rain rate product.
The quality of PR-ASS-1 v.1.0 in precipitaion detection was validated using the dichotomous (yes/no)
statistics. The 0.25 mm/h threshold was used for rain/no rain differentiation. In Fig. 55 the variability of
Probability of Detection and False Alarm Ratio are presented.
0
0,2
0,4
0,6
0,8
1
1,2
1,4
1,6
1,8
Jan
'09
Feb'
09
Mar
'09
Ap
r'0
9
May
'09
Jun
'09
Jul'
09
Aug
'09
Sep
'09
Oct
'09
No
v'0
9
Dec
'09
Jan
'10
Feb'
10
Mar
'10
FAR
POD
Fig. 55 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-ASS-1 v.1.0 rain rate product using Polish RG data in the period of Jan 2009 – Mar 2010.
The quality of PR-ASS-1 rain rate product v.1.0 in precipitation detection undergoes seasonal
variability: it increases in winter and autumn and decreases in spring and summer, what would indicate
the problem with convective precipitation detection. For the most of the analysed period, the POD is
lower than FAR, however in winter and autumn this difference is very small. Only in November the
POD is bigger than FAR. So, it can be concluded that the quality of PR-ASS-1 v.1.0 rain rate product in
precipitation recognition is the best in autumn and winter.
Analysis of accumulated precipitation
The PR-OBS-6 3 hour cumulated precipitation v.1.0 product has been validated against automatic rain
gauges data. Polish network of automatic rain gauges consists of 430 posts located all over the country,
however, the network density increases in the Southern Poland, where the flood danger is very high.
Each post is equipped with two gauges: heated and non-heated, what enables some quality control of
data. For validation purposes, readings from both gauges were compared in order to eliminate the cases
of clogged instruments. If both gauges worked properly, higher values was taken (automatic RG are
known to underestimate the real precipitation).
The measurements time resolution is 10 minutes. The ground 3 hour cumulated precipitation was
calculated by aggregating 10 minute cumulative values from 3 hours before the satellite timeslot.
In order to combine satellite products with rain gauges data, the following simple method was applied.
For each satellite pixels, the automatic posts situated within that pixel were found. If more than one rain
gauge were found within one satellite pixel, the ground rain rate value was calculated as a mean of all
rain gauges measurements within that pixel.
Results
Following the methodology agreed in the H-SAF precipitation validation group, both continuous and
categorical statistics were calculated on the monthly mean basis.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 61
In Fig. 56 and Fig. 57, values of the mean error and RMSE calculated on the base of all PR-ASS-1 3h
cumulated precipitation data are presented for each month of the analysed period. It should be pointed
out here that the analysis was performed only for situations when the accumulated rain on the ground
was 1.0 mm.
-3,5
-3
-2,5
-2
-1,5
-1
-0,5
0
0,5
Jan
'09
Feb
'09
Mar
'09
Ap
r'09
May
'09
Jun
'09
Jul'0
9
Au
g'09
Sep
'09
Oct
'09
No
v'09
De
c'09
Jan
'10
Feb
'10
Mar
'10
ME (mm)
Fig. 56 - Mean error (ME) of PR-ASS-1 3 h cumulated precipitation v.1.0 for the period of Jan 2009 – Mar 2010 for Poland.
For the whole analysed period, the PR-ASS-1 underestimates the measured 3-hour cumulated
precipitation. The underestimation is stronger in the spring and summer months. The seasonal
variability of the quality of PR-ASS-1 in estimation of 3 hour cumulated precipitation was found – the
worst results were obtained for summer and the best in winter.
0
20
40
60
80
100
120
140
160
180
200
Jan
'09
Feb
'09
Mar
'09
Ap
r'09
May
'09
Jun
'09
Jul'0
9
Au
g'09
Sep
'09
Oct
'09
No
v'09
De
c'09
Jan
'10
Feb
'10
Mar
'10
RMSE (%)
Fig. 57 - RMSE (%) of PR-ASS-1 3-hour cumulated precipitation v.1.0 for the period of Jan 2009 – Mar 2010 for Poland.
The quality of PR-ASS-1 3-hour cumulated precipitation v.1.0 product in precipitaion detection was
validated using the dichotomous (yes/no) statistics. The 1.0 mm threshold was used for rain/no rain
differentiation. In Fig. 58 the variability of Probability of Detection and False Alarm Ratio is presented.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 62
0
0,2
0,4
0,6
0,8
1
1,2
1,4
1,6
Jan
'09
Feb
'09
Mar
'09
Ap
r'09
May
'09
Jun
'09
Jul'0
9
Au
g'09
Sep
'09
Oct
'09
No
v'09
De
c'09
Jan
'10
Feb
'10
Mar
'10
FAR
POD
Fig. 58 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-ASS-1 3-hour cumulated precipitation v.1.0 product using Polish RG data for the period of Jan 2009 – Mar 2010.
Both, POD and FAR undergo seasonal variability but they changes in different way. The POD is lower
in spring and summer, while FAR in autumn. Therefore it should be concluded that the ability of PR-
ASS-1 3-hour cumulated product of precipitation detection is the best in autumn and winter and worse
in summer.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 63
4.7 Validation in Slovakia (SHMÚ)
The facilities available to SHMÚ, and the methodology adopted for validation, are described in section
3.4.6. The ground truth is provided by meteorological radar.
SHMÚ has performed full validation of PR-ASS-1, i.e. the precipitation rate and the accumulated
precipitation over 3, 6, 12 and 24 hours. The results are reported in Table 17 for the rate and Table 18
for the accumulated.
Table 17 - Summary results of the validation of PR-ASS-1rate in Slovakia by SHMÚ
H06 v1.0 Slovakia Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h
Radar Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
Number of > 10 mm/h 0 0 1 24 109 238 216 124 23 0 0 0 0 0 0
comparisons 1-10 mm/h 496 544 2144 2039 5883 12846 7691 7888 3831 3504 1555 1745 120 322 114
(ground obs.) < 1 mm/h 12595 15947 21460 5462 16880 21574 10861 17018 11285 19679 18355 14061 9932 7679 5621
> 10 mm/h - - -10.7 -15.6 -14.6 -15.2 -15.3 -14.7 -13.6 - - - - - -
ME (mm/h) 1-10 mm/h -0.77 -0.93 -0.66 -1.81 -1.69 -1.73 -1.98 -1.38 -0.99 -1.09 -0.61 -0.76 -0.75 -0.65 -0.99
< 1 mm/h 0.09 0.02 0.14 -0.37 0.01 -0.06 -0.12 0.06 0.05 0.17 0.29 0.24 0.22 -0.02 -0.11
> 10 mm/h - - 0.00 6.07 5.20 5.71 5.85 5.16 4.42 - - - - - -
SD (mm/h) 1-10 mm/h 1.19 0.58 1.20 1.39 2.00 1.97 2.11 2.60 2.70 1.37 1.34 1.18 0.54 0.69 0.63
< 1 mm/h 0.62 0.47 0.72 0.43 0.97 1.25 1.37 1.90 1.88 0.91 0.90 0.97 0.56 0.58 0.43
> 10 mm/h - - 10.7 16.7 15.5 16.2 16.4 15.6 14.3 - - - - - -
RMSE (mm/h) 1-10 mm/h 1.42 1.10 1.37 2.28 2.62 2.62 2.89 2.94 2.88 1.75 1.47 1.40 0.93 0.94 1.18
< 1 mm/h 0.63 0.47 0.73 0.56 0.97 1.25 1.38 1.90 1.88 0.92 0.94 1.00 0.60 0.58 0.45
> 10 mm/h - - 89 98 100 98 99 99 96 - - - - - -
RMSE (%) 1-10 mm/h 79 78 76 92 114 108 115 148 163 91 104 87 70 69 87
< 1 mm/h 186 131 195 114 225 295 331 430 373 242 247 266 185 158 118
> 10 mm/h - - - -0.17 -0.07 -0.10 -0.11 0.08 -0.07 - - - - - -
CC 1-10 mm/h -0.18 -0.10 0.13 0.04 -0.09 -0.02 -0.04 -0.04 0.00 -0.01 -0.06 -0.05 -0.13 0.22 0.23
< 1 mm/h 0.04 0.03 0.08 0.08 0.06 0.02 0.00 0.03 0.10 0.02 0.09 0.08 0.00 0.08 0.09
POD ≥ 0.25 mm/h 0.52 0.53 0.55 0.15 0.39 0.31 0.22 0.30 0.33 0.48 0.60 0.54 0.68 0.41 0.35
FAR ≥ 0.25 mm/h 0.86 0.89 0.84 0.81 0.69 0.79 0.84 0.71 0.68 0.84 0.83 0.87 0.90 0.94 0.94
CSI ≥ 0.25 mm/h 0.12 0.10 0.14 0.09 0.21 0.14 0.10 0.17 0.20 0.13 0.15 0.12 0.09 0.06 0.05
Graphical deployment of selected validation parameters for PR-ASS-1rate in Slovakia is in Fig. 59.
Mean Error for H06-00 v1.0
-17
-15
-13
-11
-9
-7
-5
-3
-1
1
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
Me
an
Err
or
[mm
/h]
1-10mm/h
<1mm/h
>10mm/h
Mean Error for H06-00 v1.0
-2,5
-2
-1,5
-1
-0,5
0
0,5
1
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
Me
an
Err
or
[mm
/h]
1-10mm/h
<1mm/h
RMSE [%] for H06-00 v1.0
0%
50%
100%
150%
200%
250%
300%
350%
400%
450%
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
>10mm/h
1-10mm/h
<1mm/h
Scores of dichotomous statistics for H06-00 v1.0
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
Sc
ore
s
POD
FAR
CSI
Fig. 59 - Time evolution of Mean Error, Root Mean Square Error (%) and dichotomous scores for rain rate.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 64
Scores of dichotomous characteristics show that in general POD is too low and FAR is too high. The
highest POD=0.68 was reached in January 2010 and the lowest FAR=0.68 in September 2009. The
highest CSI=0.21 was reached in May 2009. Seasonal variability can be observed on POD values. POD
is lower in summer months probably due to convective season.
Seasonal variability is well observed on RMSE values in case of precipitation rate RR<1 mm/h. The
best RMSE = 69 % occurred in February 2010 for rain rate 1<RR<10mm/h.
ME shows acceptable performance for rain rate RR<10mm/h but very high underestimation was found
in cases when RR>10mm/h.
Table 18 - Summary results of the validation of PR-ASS-1accumulated in Slovakia by SHMÚ
H06 v1.0 Slovakia Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Radar Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 5169 6754 9612 6437 18597 37131 22604 24604 11884 16700 9748 8973 2447 2439 1337
ME (mm) 0.40 -0.11 0.21 -1.77 -1.08 -1.33 -1.79 -0.72 -0.14 0.19 0.92 0.84 0.49 0.01 -0.43
SD (mm) 1.87 1.40 1.98 2.00 3.27 4.01 4.59 4.54 4.10 2.96 2.88 3.12 1.31 1.64 1.23
RMSE (mm) 1.91 1.40 1.99 2.67 3.44 4.23 4.92 4.60 4.10 2.96 3.02 3.23 1.40 1.64 1.31
03-h cumulate RMSE (%) 146 105 139 89 144 170 191 204 181 178 215 223 119 128 97
CC 0.31 0.02 0.30 0.04 -0.13 0.03 -0.06 0.03 0.17 0.04 0.10 0.11 -0.05 0.08 0.00
POD 0.55 0.50 0.56 0.17 0.37 0.34 0.27 0.32 0.39 0.52 0.62 0.58 0.62 0.49 0.31
FAR 0.92 0.94 0.91 0.74 0.70 0.75 0.77 0.69 0.67 0.85 0.90 0.89 0.97 0.97 0.98
CSI 0.07 0.06 0.09 0.11 0.20 0.17 0.14 0.19 0.22 0.13 0.09 0.10 0.03 0.03 0.02
No. samples 18123 24504 24548 12685 32519 63954 39812 41279 21485 38625 29195 22437 14274 8746 5431
ME (mm) 2.25 0.53 0.77 -1.81 -0.83 -0.75 -1.09 -0.10 0.27 1.21 2.03 1.78 1.98 0.81 0.04
SD (mm) 4.45 2.06 2.79 2.26 4.08 5.39 6.17 6.58 4.81 4.27 3.98 4.58 2.14 2.53 1.72
RMSE (mm) 4.98 2.13 2.90 2.90 4.16 5.44 6.27 6.58 4.82 4.44 4.47 4.91 2.92 2.65 1.72
06-h cumulate RMSE (%) 373 146 198 93 165 205 239 246 190 251 282 288 235 199 128
CC 0.13 0.11 0.29 0.05 -0.02 0.15 0.04 0.11 0.25 0.11 0.20 0.14 0.01 -0.02 0.14
POD 0.72 0.63 0.59 0.22 0.42 0.47 0.39 0.40 0.50 0.61 0.75 0.70 0.87 0.63 0.49
FAR 0.80 0.84 0.84 0.66 0.54 0.62 0.65 0.56 0.58 0.71 0.75 0.82 0.86 0.91 0.93
CSI 0.18 0.15 0.15 0.15 0.28 0.27 0.23 0.27 0.30 0.24 0.23 0.17 0.13 0.08 0.06
No. samples 36873 63475 49760 21301 48283 96983 65595 61832 33830 75634 60677 47860 43893 18838 11979
ME (mm) 3.91 1.38 1.27 -1.73 -0.24 -0.22 -0.34 0.16 0.51 2.71 3.92 3.01 4.10 2.97 0.78
SD (mm) 5.55 2.76 3.36 2.57 5.48 6.64 7.36 7.31 5.36 6.30 5.17 5.91 3.73 3.70 2.22
RMSE (mm) 6.79 3.09 3.59 3.10 5.48 6.64 7.36 7.31 5.38 6.85 6.49 6.63 5.54 4.75 2.36
12-h cumulate RMSE (%) 395 191 246 102 192 235 284 259 185 376 352 347 372 346 171
CC 0.30 0.26 0.14 0.06 0.13 0.20 0.07 0.15 0.29 0.16 0.38 0.23 0.29 -0.02 0.02
POD 0.77 0.77 0.65 0.25 0.45 0.56 0.49 0.46 0.55 0.73 0.83 0.83 0.89 0.87 0.73
FAR 0.72 0.71 0.75 0.59 0.43 0.50 0.55 0.44 0.52 0.54 0.60 0.73 0.72 0.84 0.86
CSI 0.26 0.27 0.22 0.19 0.33 0.36 0.31 0.34 0.35 0.40 0.37 0.26 0.27 0.15 0.13
No. samples 61518 125536 90126 30119 59595 135549 102604 83210 47525 117568 100896 101377 91779 38314 14516
ME (mm) 5.50 2.70 2.30 -1.92 -0.80 -0.19 0.74 0.33 0.63 4.44 5.75 4.35 6.00 6.34 2.72
SD (mm) 6.78 3.99 4.63 3.00 5.47 6.47 8.12 7.94 5.76 9.75 6.85 7.33 5.99 5.42 3.21
RMSE (mm) 8.73 4.82 5.17 3.56 5.52 6.47 8.15 7.95 5.79 10.72 8.94 8.52 8.48 8.34 4.20
24-h cumulate RMSE (%) 412 267 322 108 157 226 295 242 188 496 410 430 453 563 318
CC 0.48 0.29 0.14 0.13 0.22 0.23 0.13 0.22 0.30 0.19 0.59 0.26 0.49 0.17 -0.05
POD 0.79 0.85 0.73 0.26 0.43 0.58 0.62 0.52 0.64 0.77 0.82 0.84 0.89 0.93 0.84
FAR 0.67 0.58 0.62 0.54 0.33 0.37 0.41 0.32 0.47 0.41 0.50 0.62 0.60 0.78 0.84
CSI 0.30 0.39 0.33 0.20 0.36 0.43 0.43 0.42 0.41 0.50 0.45 0.35 0.38 0.22 0.16
Graphical representations of selected validation parameters for PR-ASS-1accumulated in Slovakia are in
Fig. 60 (for Mean Error and Root Mean Square Error) and Fig. 61 (for dichotomous scores).
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 65
Mean Error for H06-03,06,12,24 v1.0
-2,50
-1,50
-0,50
0,50
1,50
2,50
3,50
4,50
5,50
6,50
Jan
2009
Mar May Jul Sep Nov Jan
2010
Mar
Me
an
Err
or
[mm
/h]
Period 3h
Period 6h
Period 12h
Period 24h
RMSE [%] for H06-03,06,12,24 v1.0
0%
100%
200%
300%
400%
500%
600%
Jan
2009
Mar May Jul Sep Nov Jan
2010
Mar
Period 3h
Period 6h
Period 12h
Period 24h
Fig. 60 - Time evolution of Mean Error and Root Mean Square Error (%), function of the integration interval 3, 6, 12 and 24 h.
Mean error of PR-ASS-1 accumulated demonstrates very strong seasonal variability. This variability is very
significant for long integration periods (24 hours) and reduced in case of short integration period (3
hours). While during summer we observed slight underestimation, during winter it was overestimation
of precipitation. RMSE is too high in winter period, especially for longer time integration periods (12,
24 hours).
Dichotomous statistics - POD for H06-03,06,12,24 v1.0
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0,90
1,00
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
Sc
ore
s
POD 3h
POD 6h
POD 12h
POD 24h
Fig. 61 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index, function of the integration interval (3, 6, 12 and 24 h).
Dichotomous statistics parameters
POD, FAR and CSI also demonstrate
seasonal variability and confirm better
performance of PR-OBS-1accumulated for
winter period which is typical by
absence of convection in central
Europe. POD is typically higher and
FAR lower for longer cumulated time
periods. CSI reached the maximum
value of 0.5 in October 2009 for 24
hour cumulated period, POD=0.77 and
FAR=0.41 for this month. Absolute
maximum POD=0.93 was reached in
February 2010 and absolute minimum
FAR=0.32 in August 2009, both values
for the longest integration time period
24 hours.
Dichotomous statistics - FAR for H06-03,06,12,24 v1.0
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0,90
1,00
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
Sc
ore
s
FAR 3h
FAR 6h
FAR 12h
FAR 24h
Dichotomous statistics - CSI for H06-03,06,12,24 v1.0
0,00
0,10
0,20
0,30
0,40
0,50
0,60
0,70
0,80
0,90
1,00
Jan
2009
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2010
Feb Mar
Sc
ore
s
CSI 3h
CSI 6h
CSI 12h
CSI 24h
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 66
4.8 Validation in Turkey (ITU)
The facilities available to ITU (in collaboration with TSMS), and the methodology adopted for
validation, are described in section 3.4.7. The ground truth is provided by rain gauge networks.
ITU and TSMS have performed validation of PR-ASS-1 accumulated precipitation over 3 and 24 hours.
Separate statistics are reported for inner land and coastal zones in Table 19 and Table 20, respectively.
Table 19 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS over inner land
H06 v1.0 Turkey Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Gauge Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 67614 245327 142723 76476 53254 28699 29716 5803 51179 79742 114836 195630 200368 193036 94046
ME (mm) -0.36 -0.06 -0.11 -1.63 -1.68 -3.19 -3.52 -2.80 -3.34 -0.67 -0.44 -0.73 -0.40 -0.28 -0.28
SD (mm) 2.63 2.92 2.27 2.92 3.16 4.71 4.37 2.71 5.77 3.39 3.58 3.63 2.24 3.14 2.04
RMSE (mm) 2.66 2.92 2.27 3.34 3.57 5.69 5.61 3.90 6.67 3.45 3.60 3.70 2.28 3.16 2.06
03-h cumulate RMSE (%) 139 135 120 104 114 117 117 100 140 159 127 132 110 128 106
CC 0.16 0.32 0.36 0.31 0.16 0.06 0.08 -0.01 0.08 0.24 0.43 0.34 0.47 0.46 0.27
POD 0.51 0.53 0.54 0.39 0.40 0.14 0.15 0.018 0.21 0.32 0.52 0.52 0.49 0.53 0.50
FAR 0.45 0.45 0.54 0.62 0.63 0.69 0.78 0.96 0.70 0.50 0.42 0.39 0.49 0.44 0.50
CSI 0.36 0.37 0.33 0.24 0.24 0.11 0.10 0.01 0.14 0.24 0.38 0.39 0.33 0.37 0.33
No. samples 721196 845555 714936 347073 262853 183373 149410 40727 224855 384553 454379 766484 873949 840744 523407
ME (mm) 1.10 1.10 0.71 -1.47 -1.40 -3.65 -4.82 -3.16 -3.91 -0.87 0.09 -0.80 0.02 0.61 0.11
SD (mm) 8.62 8.13 5.25 6.59 7.62 6.76 9.45 4.01 14.09 6.46 9.22 8.32 5.69 7.69 4.13
RMSE (mm) 8.69 8.20 5.30 6.75 7.75 7.68 10.61 5.11 14.62 6.52 9.22 8.35 5.69 7.71 4.13
24-h cumulate RMSE (%) 144 151 136 144 209 125 159 120 237 167 134 124 128 163 107
CC 0.59 0.54 0.55 0.41 0.46 0.27 0.36 0.03 0.23 0.44 0.56 0.58 0.60 0.70 0.59
POD 0.81 0.79 0.79 0.67 0.51 0.24 0.31 0.16 0.47 0.48 0.80 0.77 0.78 0.75 0.74
FAR 0.10 0.12 0.17 0.26 0.39 0.34 0.48 0.61 0.30 0.15 0.08 0.14 0.13 0.13 0.17
CSI 0.75 0.71 0.68 0.54 0.38 0.21 0.24 0.13 0.39 0.44 0.74 0.68 0.70 0.67 0.65
Table 20 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS over coastal zones
H06 v1.0 Turkey Coast Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm
Gauge Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar
No. samples 38577 12482 8618 2774 1343 1077 990 200 3121 5998 6501 10654 11972 11577 5382
ME (mm) -0.28 -0.44 -0.24 -1.77 -2.67 -4.06 -4.22 -3.07 -4.78 -0.57 -0.83 -0.32 -0.4 -0.3 -0.3
SD (mm) 3.08 3.71 3.10 3.30 3.31 7.41 7.07 2.96 9.19 4.35 3.90 5.27 2.8 4.0 2.3
RMSE (mm) 3.09 3.74 3.11 3.75 4.25 8.45 8.23 4.26 10.36 4.38 3.99 5.28 2.9 4.0 2.4
03-h cumulate RMSE (%) 136 146 143 129 99 318 147 97 198 214 122 164 129 141 114
CC 0.35 0.35 0.37 0.10 0.08 0.00 0.04 0.03 0.05 0.26 0.49 0.34 0.50 0.50 0.40
POD 0.54 0.48 0.50 0.39 0.32 0.18 0.23 0.05 0.25 0.35 0.41 0.57 0.49 0.56 0.47
FAR 0.26 0.34 0.40 0.60 0.63 0.58 0.75 0.96 0.65 0.42 0.29 0.28 0.38 0.29 0.34
CSI 0.45 0.38 0.37 0.24 0.21 0.14 0.14 0.02 0.17 0.28 0.35 0.47 0.37 0.46 0.38
No. samples 34551 38816 34587 11236 7012 6020 4587 1656 11040 20460 20531 32187 39091 40123 20966
ME (mm) -0.20 -0.84 -0.04 -1.06 -2.79 -5.55 -6.37 -3.34 -7.96 -0.34 -2.20 0.67 0.30 -0.09 -0.30
SD (mm) 9.05 9.47 7.41 8.15 6.74 10.5 13.9 3.98 23.0 9.94 10.4 12.9 8.41 10.5 4.40
RMSE (mm) 9.05 9.51 7.41 8.21 7.29 11.9 15.2 5.20 24.4 9.94 10.6 12.9 8.42 10.5 4.41
24-h cumulate RMSE (%) 128 126 156 145 159 129 285 97 250 249 122 144 139 197 88
CC 0.61 0.53 0.57 0.46 0.45 0.28 0.39 0.21 0.23 0.41 0.59 0.60 0.59 0.66 0.72
POD 0.74 0.71 0.72 0.67 0.46 0.26 0.37 0.22 0.53 0.57 0.74 0.76 0.75 0.73 0.75
FAR 0.08 0.10 0.11 0.25 0.29 0.27 0.53 0.76 0.27 0.15 0.06 0.10 0.12 0.07 0.12
CSI 0.70 0.66 0.66 0.55 0.39 0.24 0.26 0.13 0.44 0.52 0.71 0.70 0.68 0.69 0.68
Fig. 62 shows the time evolution for inner land of Mean Error, Standard Deviation, Root Mean Square
Error, Probability Of Detection, False Alarm Rate and Critical Success Index for accumulation periods 3
and 24 hours. Fig. 63 shows the same for coastal zones. Fig. 64 shows the Correlation Coefficient for
inner land and coastal zone, for accumulation periods 3 and 24 hours.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 67
Mean Error, Standart Deviation and RMS for H06_03 (Land)
-6
-4
-2
0
2
4
6
8
Jan
2009
Feb Mar
Apr
May Ju
nJu
lAgu
Sep
.O
ctNov
Dec
Jan
2010
Feb MarS
co
res (
mm
)
ME (mm)
SD (mm)
RMSE (mm)
Mean Error, Standart Deviation and RMS for H06_24 (Land)
-10
-5
0
5
10
15
20
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s (
mm
)
RMSE (mm)
SD (mm)
ME (mm)
Multi-categorical Scores for H06_03 (Land)
0
0.2
0.4
0.6
0.8
1
1.2
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s
POD
FAR
CSI
Multi-categorical Scores for H06_24 (Land)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s
POD
FAR
CSI
Fig. 62 - Continuous and multi-categorical statistics for inner land (3 and 24 hourly accumulation).
Mean Error, Standart Deviation and RMS for H06_03 (Coast)
-6
-4
-2
0
2
4
6
8
10
12
Jan
2009
Feb Mar
AprM
ay Jun
Jul
AguSep
.O
ctNov
Dec
Jan
2010
Feb Mar
Sco
res (
mm
)
ME (mm)
SD (mm)
RMSE (mm)
Mean Error, Standart Deviation and RMS for H06_24 (Coast)
-10
-5
0
5
10
15
20
25
30
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s (
mm
)
RMSE (mm)
SD (mm)
ME (mm)
Multi-categorical Scores for H06_03 (Coast)
0
0.2
0.4
0.6
0.8
1
1.2
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s
POD
FAR
CSI
Multi-categorical Scores for H06_24 (Coast)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s
POD
FAR
CSI
Fig. 63 - Continuous and multi-categorical statistics for coastal zones (3 and 24 hourly accumulation).
Correlation Coefficients for Accumulated (Land)
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s
CC - 3 Hourly
CC - 24 Hourlya
Correlation Coefficients for Accumulated (Coast)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Jan
2009
Feb Mar
Apr
May Ju
nJu
l
Agu
Sep
.Oct
Nov
Dec
Jan
2010
Feb Mar
Sc
ore
s
CC - 3 Hourly
CC - 24 Hourlya
Fig. 64 - Correlation coefficients for inner land and coastal zones (3 and 24 hourly accumulation).
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 68
The following interpretations can be drawn from graphs concerning inner land and coastal precipitation.
When the patterns in Figures 62 and 63 are compared each other, very similar patterns between
inner land and coastal zones for multi-categorical and continuous statistics can be seen.
Maximum False Alarm Rate (FAR), minimum Probability of Detection (POD) and Critical Success
Index (CSI) can be seen in August for all hourly accumulations. FAR increases till August and then
it decreases. When POD and CSI are compared with FAR, reverse tendency can be seen for POD
and CSI as expected. Moreover, the 24-hourly accumulated product has best multi-categorical
statistics.
The best Correlation Coefficients (CC) are found for the 24-hourly accumulated product in Fig. 64
and CC are very low in summer months.
There is underestimation between April and October because Mean Error (ME) has negative value.
For other months, it is almost zero.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 69
5. Overview of findings
5.1 Synopsis of validation results
In the various sections of Chapter 4 the validation results have been quoted separately by each Team
operating on a different geographic area associated to a proper climatic condition. This is correct, since
the precipitation field is affected by orography and local climatology. In this section a synoptic
overview is provided, of the results achieved in the different countries, and in different seasons.
In order to reduce the volume of data to be commented, and not to overlap with the detailed reports in
Chapter 4, the results are summarised by seasons of three months each. The phase is:
Spring: Summer: Autumn: Winter:
March, April and May 2009 June, July and August 2009 Sept., Oct. and Nov. 2009 Dec. 2009, Jan. and Feb. 2010
Table 21 and Table 22, split in four sections, one for each season, report the Country/Team results side-
by-side. There are three sets of columns:
one set for four Countries/Teams that compared satellite data with meteorological radar in inner land
areas: Belgium/IMR, Germany/BfG, Hungary/OMSZ and Slovakia/SHMÚ; and their average
weighed by the number of comparisons;
one set for three Countries/Teams that compared satellite data with rain gauges in inner land areas:
Italy/UniFe, Poland/IMWM and Turkey/ITU; and their average weighed by the number of
comparisons;
one column for Turkey/ITU that compared satellite data with rain gauges in coastal zones.
It is noted that not all Countries/Teams performed validation of all products, as shown in the following
map (shadowed cells: no validation).
Belgium Germany Hungary Italy Poland Slovakia Turkeyland Turkeycoast
PR-ASS-1 00 h
PR-ASS-1 03 h
PR-ASS-1 06 h
PR-ASS-1 12 h
PR-ASS-1 24 h
It is reminded, from Chapter 4 Table 07, that the User requirements are:
Table 07 - Accuracy requirements for product PR-ASS-1 [RMSE (%)]
Product threshold target optimal Additional specifications
Precipitation rate 100 50 25 To be validated for RR < 1.0, 1-10 and > 10 mm/h.
To be verified for RR in the range 1-10 mm/h
Accumulated precipitation 200 100 50 To be validated for integration over 3, 6, 12 and 24 h.
To be verified for integration over 24 h
The unit for accuracy specification is Root Mean Square Error percent (RMSE %), used since error
grows with rate. In Chapter 4 further scores were recorded: Mean Error (or bias, ME), Standard
Deviation (SD) and Correlation Coefficient (CC), Probability Of Detection (POD), False Alarm Rate
(FAR) and Critical Success Index (CSI). In this summary Chapter 5, in order to streamline the
discussion and minimize repetition with Chapter 4, we only focus on RMSE (%), ME (mm/h) and POD
/ FAR.
Tables 21 and 22 highlight the RMSE (%) rows by yellow colour, the weighed average column by blue
and the averaged RMSE (%) values (first thing to attract the attention) by green. The column for
“coast”, being single, is not averaged.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 70
Table 21 - Comparative results of validation in several Countries/Teams split by season. PR-ASS-1 00 h (rain rate)
PR-ASS-1rate Spring IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge
> 10 mm/h 1,593 541 134 2,268 4,364 152 4,516
N. of samples 1-10 mm/h 183,246 41,187 10,066 234,499 179,713 5,159 184,872
< 1 mm/h 428,793 132,971 43,802 605,566 172,878 7,212 180,090
> 10 mm/h -16.9 -16.6 -14.7 -16.7 -16.7 -18.1 -16.7
ME (mm/h) 1-10 mm/h -1.22 -1.65 -1.49 -1.31 -1.60 -1.51 -1.60
< 1 mm/h -0.22 -0.16 0.03 -0.19 -0.02 -0.12 -0.02
> 10 mm/h 92 100 99 94 91 99 91
RMSE (%) 1-10 mm/h 133 98 101 125 97 86 97
< 1 mm/h 158 182 196 166 251 155 247
POD ≥ 0.25 mm/h 0.21 0.29 0.43 0.24 0.45 0.55 0.45
FAR ≥ 0.25 mm/h 0.73 0.83 0.77 0.75 0.40 0.67 0.41
PR-ASS-1rate Summer IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge
> 10 mm/h 6,076 3,657 578 10,311 5,360 577 5,937
N. of samples 1-10 mm/h 245,953 100,346 28,425 374,724 71,067 7,489 78,556
< 1 mm/h 232,768 144,552 49,453 426,773 68,067 8,159 76,226
> 10 mm/h -14.8 -18.0 -15.1 -15.9 -15.6 -22.4 -16.3
ME (mm/h) 1-10 mm/h -1.85 -2.08 -1.70 -1.90 -2.28 -1.96 -2.25
< 1 mm/h -0.15 -0.11 -0.03 -0.12 -0.10 -0.14 -0.10
> 10 mm/h 94 98 98 96 96 97 96
RMSE (%) 1-10 mm/h 115 130 121 119 116 127 117
< 1 mm/h 255 341 349 295 354 260 344
POD ≥ 0.25 mm/h 0.18 0.21 0.29 0.20 0.23 0.33 0.24
FAR ≥ 0.25 mm/h 0.81 0.80 0.78 0.80 0.66 0.69 0.66
PR-ASS-1rate Autumn IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge
> 10 mm/h 1,709 778 23 2510 9,656 122 9,778
N. of samples 1-10 mm/h 226,404 55,901 8,890 291,195 165,668 7,098 172,766
< 1 mm/h 435,778 135,507 49,319 620,604 149,776 10,493 160,269
> 10 mm/h -13.4 -19.2 -13.6 -15.2 -14.9 -26.9 -15.0
ME (mm/h) 1-10 mm/h -1.32 -1.23 -0.96 -1.29 -1.87 -1.17 -1.84
< 1 mm/h -0.18 0.16 0.19 -0.08 -0.03 -0.13 -0.04
> 10 mm/h 95 96 96 95 92 97 92
RMSE (%) 1-10 mm/h 89 116 124 95 115 79 113
< 1 mm/h 166 268 274 197 326 131 313
POD ≥ 0.25 mm/h 0.26 0.47 0.48 0.32 0.37 0.55 0.38
FAR ≥ 0.25 mm/h 0.68 0.78 0.79 0.71 0.45 0.59 0.46
PR-ASS-1rate Winter IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009/10 radar radar radar radar radar gauge gauge gauge gauge gauge
> 10 mm/h 102 90 0 192 4,617 7 4,624
N. of samples 1-10 mm/h 202,975 33,129 2,187 238,291 257,296 2,277 259,573
< 1 mm/h 788,275 143,829 31,672 963,776 302,927 7,008 309,935
> 10 mm/h -12.6 -13.3 - -12.9 -12.4 -20.8 -12.4
ME (mm/h) 1-10 mm/h -0.99 -0.94 -0.74 -0.98 -1.43 -0.91 -1.43
< 1 mm/h -0.30 0.13 0.17 -0.22 -0.02 -0.10 -0.02
> 10 mm/h 100 90 - 95 82 96 82
RMSE (%) 1-10 mm/h 85 83 83 85 91 70 91
< 1 mm/h 114 188 214 128 217 91 214
POD ≥ 0.25 mm/h 0.21 0.58 0.55 0.27 0.50 0.69 0.50
FAR ≥ 0.25 mm/h 0.65 0.84 0.90 0.69 0.38 0.79 0.39
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 71
Table 22 - Comparative results of validation in several Countries/Teams split by season. PR-ASS-1 accumulated precip.
PR-ASS-1acc Spring IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge
N. of samples 578,568 500,001 34,646 1,113,215 383,791 16,373 272,453 672,617 12,735
ME (mm) -1.26 -1.21 -0.85 -1.22 -2.35 -1.42 -0.84 -1.72 -0.83
3 h RMSE (%) 133 135 132 134 120 95 114 117 135
POD 0.37 0.41 0.39 0.39 0.50 0.53 0.47 0.49 0.46
FAR 0.52 0.87 0.77 0.68 0.39 0.56 0.58 0.47 0.47
N. of samples 935,156 117,571 69,752 1,122,479
ME (mm) -1.25 -1.16 -0.45 -1.19
6 h RMSE (%) 165 168 163 165
POD 0.50 0.37 0.44 0.48
FAR 0.45 0.78 0.67 0.50
N. of samples 1,587,263 111,666 119,344 1,818,273
ME (mm) -1.43 -0.52 0.12 -1.27
12 h RMSE (%) 202 212 198 202
POD 0.59 0.48 0.50 0.58
FAR 0.36 0.71 0.59 0.40
N. of samples 44,387 2,638,946 97,142 179,840 2,960,315 1,301,940 1,324,862 2,626,802 52,835
ME (mm) -2.37 -1.66 -0.14 0.57 -1.49 -1.71 -0.28 -0.99 -0.62
24 h RMSE (%) 149 222 266 231 223 201 152 176 154
POD 0.70 0.65 0.58 0.55 0.64 0.70 0.70 0.70 0.67
FAR 0.29 0.27 0.64 0.51 0.30 0.22 0.24 0.23 0.16
PR-ASS-1acc Summer IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge
N. of samples 612,563 605,761 84,339 1,302,663 174,530 25,832 64,218 264,580 2,267
ME (mm) -1.98 -1.71 -1.28 -1.81 -3.86 -2.87 -3.31 -3.63 -4.04
3 h RMSE (%) 149 153 185 153 149 136 115 139 224
POD 0.31 0.28 0.32 0.30 0.33 0.34 0.13 0.28 0.19
FAR 0.67 0.80 0.74 0.73 0.62 0.60 0.76 0.65 0.69
N. of samples 1,023,481 215,851 145,045 1,384,377
ME (mm) -1.88 -1.64 -0.66 -1.71
6 h RMSE (%) 178 232 226 191
POD 0.43 0.42 0.43 0.43
FAR 0.56 0.68 0.61 0.58
N. of samples 1,718,760 187,756 224,410 2,130,926
ME (mm) -1.74 -1.56 -0.15 -1.56
12 h RMSE (%) 202 256 256 212
POD 0.54 0.50 0.51 0.53
FAR 0.46 0.57 0.50 0.47
N. of samples 45,473 2,878,938 163,302 321,363 3,409,076 835,637 373,510 1,209,147 12,263
ME (mm) -3.01 -1.58 -1.13 0.24 -1.41 -4.03 -4.06 -4.04 -5.56
24 h RMSE (%) 205 225 278 252 230 234 138 204 183
POD 0.51 0.63 0.59 0.58 0.62 0.57 0.26 0.47 0.30
FAR 0.32 0.35 0.49 0.37 0.36 0.44 0.43 0.44 0.43
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 72
PR-ASS-1acc Autumn IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge
N. of samples 663,053 570,127 38,332 1,271,512 354,182 21,645 245,757 621,584 15,620
ME (mm) -1.41 -1.23 0.27 -1.28 -3.20 -1.14 -1.12 -2.31 -1.52
3 h RMSE (%) 110 113 188 114 136 95 140 136 172
POD 0.40 0.55 0.51 0.47 0.45 0.55 0.39 0.43 0.35
FAR 0.49 0.78 0.81 0.63 0.42 0.48 0.50 0.45 0.41
N. of samples 1,081,399 136,820 89,305 1,307,524
ME (mm) -1.39 0.21 1.25 -1.04
6 h RMSE (%) 123 251 246 145
POD 0.49 0.61 0.63 0.51
FAR 0.40 0.71 0.69 0.45
N. of samples 1,825,397 119,138 170,141 2,114,676
ME (mm) -1.49 1.33 2.70 -0.99
12 h RMSE (%) 125 318 329 152
POD 0.56 0.66 0.73 0.58
FAR 0.30 0.66 0.56 0.34
N. of samples 30,055 2,992,214 102,990 265,989 3,391,248 1,270,619 1,063,787 2,334,406 52,031
ME (mm) -1.08 -1.66 2.96 4.26 -1.05 -3.81 -1.10 -2.58 -2.69
24 h RMSE (%) 151 132 394 408 162 221 168 197 199
POD 0.71 0.62 0.74 0.77 0.64 0.55 0.61 0.58 0.63
FAR 0.22 0.22 0.60 0.45 0.25 0.32 0.15 0.24 0.14
PR-ASS-1acc Winter IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast
Version 1.0 2009/10 radar radar radar radar radar gauge gauge gauge gauge gauge
N. of samples 827,677 269,312 13,859 1,110,848 565,733 8,318 589,034 1,163,085 34,203
ME (mm) -1.30 -0.85 0.63 -1.17 -1.79 -0.19 -0.47 -1.11 -0.34
3 h RMSE (%) 91 108 188 96 117 91 123 120 144
POD 0.31 0.64 0.57 0.39 0.55 0.70 0.51 0.53 0.54
FAR 0.40 0.84 0.92 0.51 0.37 0.72 0.44 0.41 0.32
N. of samples 1,423,592 118,897 45,457 1,587,946
ME (mm) -1.54 0.84 1.66 -1.27
6 h RMSE (%) 93 205 254 106
POD 0.38 0.73 0.74 0.42
FAR 0.29 0.79 0.85 0.34
N. of samples 2,459,129 104,526 110,591 2,674,246
ME (mm) -1.97 2.40 3.44 -1.58
12 h RMSE (%) 93 313 357 112
POD 0.44 0.85 0.86 0.47
FAR 0.20 0.76 0.74 0.24
N. of samples 48,643 3,930,494 89,813 231,470 4,300,420 1,944,113 2,481,177 4,425,290 111,401
ME (mm) -0.27 -2.68 4.14 5.33 -2.08 -1.33 -0.03 -0.60 0.27
24 h RMSE (%) 126 93 411 461 120 210 139 170 161
POD 0.83 0.50 0.90 0.87 0.53 0.77 0.77 0.77 0.75
FAR 0.36 0.12 0.72 0.64 0.16 0.19 0.13 0.16 0.10
5.2 Summary conclusions on comparative elements
In the various sections of Chapter 4 the Countries/Teams have concluded with highlighting the main
positive aspects of the product and the main failures, according to the experience on their area of
investigation. This Section does in no way bias the original conclusions in the different sections of
Chapter 4, but only refers to features stemming from observation of Tables 21 and 22.
It is noted that this validation cycle was the first performed by the H-SAF validation Units, the previous
having been performed internally to CNMCA. As for all other products, the comparison were made in
the native product grid, that in this case coincides with the grid points of the COSMO-ME model.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 73
Variability with geographical area
Rain rate (PR-ASS-1 00 h)
It may be observed that the performances are rather consistent across the various geographical areas,
especially for heavy (> 10 mm/h) and medium (1-10 mm/h) precipitation. This in spite of the fact that,
for this first cycle of “external” validation, the comparison methodology was not yet fully standardised.
Even for inner lands and coastal areas the performance are similar rather.
Accumulated (PR-ASS-1 03. 06, 12 and 24 h)
The performance is similar in spring, but in all other seasons sharply degrades in Hungary and Slovakia.
Variability with validation tool
Rain rate (PR-ASS-1 00 h) and Accumulated (PR-ASS-1 03. 06, 12 and 24 h)
The performances resulting from validation by radar and those by rain gauges are rather similar. This is
very important because User should not mind about which tool has been used for the validation: the
information on the performance is regarded as a property of the product, not of the ground truth.
Variability with season
Rain rate (PR-ASS-1 00 h) and Accumulated (PR-ASS-1 03. 06, 12 and 24 h)
The performances are rather similar across seasons. The FAR is rather high for rain rate in all seasons,
much better for accumulated precipitation. The POD is relatively high for accumulated precipitation,
especially in summer and autumn.
Variability with precipitation type (or intensity) or integration interval
Rain rate (PR-ASS-1 00 h)
Heavy and medium precipitation have very similar performances through all seasons and geographical
areas, whereas for light precipitation there is a substantial degradation in spring and autumn, and very
substantial in summer.
Accumulated (PR-ASS-1 03. 06, 12 and 24 h)
Slowly degrading with increasing integration interval.
Overall observation on compliance with User requirements
This final comment does not replace an in-depth analysis of the compliance between user requirements
and satellite-derived product performance, that would deserve a much detailed discussion. However,
with the understanding that this is just a rough overall assessment, Table 23 offers a view on the subject.
It is reminded that the verification is due for rate in the range 1-10 mm/h and accumulated over 24 h.
Table 23 - Simplified compliance analysis for product PR-ASS-1
Between target and optimal Between threshold and target Threshold exceeded by < 50 % Threshold exceeded by ≥ 50 %
PR-ASS-1 v1.0 Spring Summer Autumn Winter
Benchmark Requirement (RMSE %) radar gauge gauge radar gauge gauge radar gauge gauge radar gauge gauge
range thresh target optimal land land coast land land coast land land coast land land coast
Rate (1-10 mm/h) 100 50 25 125 97 119 117 95 113 85 91
Cumulate (24 h) 200 100 50 223 176 154 230 204 183 162 197 199 120 170 161
Compact presentation of validation results
Since it has been noted above that the variability of results with validation tool (radar or gauge), as well
as across the various Countries, and between inner land and coastal zones, are not very pronounced, it
may be useful to synthesis all results in a presentation that averages among the various situations only
leaving differentiation with season. This is provided in Table 24.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) Page 74
Table 24 - Synthesis of all validation results, including yearly average
PR-ASS-1 Version 1.0 Spring 2009 Summer 2009 Autumn 2009 Winter 2009/10 Yearly average
Instantaneous precipitation
> 10 mm/h 6,784 16,248 12,288 4,816 40,136
N. of samples 1-10 mm/h 419,371 453,280 463,961 497,864 1,834,476
< 1 mm/h 785,656 502,999 780,873 1,273,711 3,343,239
> 10 mm/h -16.7 -16.0 -15.0 -12.4 -15.4
ME (mm/h) 1-10 mm/h -1.44 -1.96 -1.49 -1.21 -1.52
< 1 mm/h -0.15 -0.12 -0.07 -0.17 -0.13
> 10 mm/h 92 96 93 82 93
RMSE (%) 1-10 mm/h 113 119 102 88 105
< 1 mm/h 184 302 221 149 197
POD ≥ 0.25 mm/h 0.30 0.21 0.34 0.34 0.31
FAR ≥ 0.25 mm/h 0.65 0.78 0.64 0.59 0.65
Accumulated precipitation
N. of samples 1,798,567 1,569,510 1,908,716 2,308,136 7,584,929
ME (mm) -1.40 -2.12 -1.62 -1.13 -1.52
3 h RMSE (%) 128 151 122 109 125
POD (≥ 1 mm) 0.43 0.30 0.46 0.46 0.42
FAR (≥ 1 mm) 0.60 0.72 0.57 0.46 0.57
N. of samples 1,122,479 1,384,377 1,307,524 1,587,946 5,402,326
ME (mm) -1.19 -1.71 -1.04 -1.27 -1.31
6 h RMSE (%) 165 191 145 106 149
POD (≥ 1 mm) 0.48 0.43 0.51 0.42 0.46
FAR (≥ 1 mm) 0.50 0.58 0.45 0.34 0.46
N. of samples 1,818,273 2,130,926 2,114,676 2,674,246 8,738,121
ME (mm) -1.27 -1.56 -0.99 -1.58 -1.37
12 h RMSE (%) 202 212 152 112 165
POD (≥ 1 mm) 0.58 0.53 0.58 0.47 0.53
FAR (≥ 1 mm) 0.40 0.47 0.34 0.24 0.35
N. of samples 5,639,952 4,630,486 5,777,685 8,837,111 24,885,234
ME (mm) -1.25 -2.11 -1.68 -1.31 -1.53
24 h RMSE (%) 200 223 176 145 179
POD (≥ 1 mm) 0.67 0.58 0.62 0.65 0.63
FAR (≥ 1 mm) 0.27 0.38 0.24 0.16 0.24
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) - Appendix Page 75
Appendix to PVR-06 (Instantaneous and accumulated precipitation at ground
computed by a NWP model)
Collection of validation experiment reports
(extracted from REP-3/07 dated 28 February 2010)
INDEX
2. Validation figures from the COSMO Consortium
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) - Appendix Page 76
2. Validation figures from the COSMO Consortium
2.1 Information source and method
Product PR-ASS-1 is basically the accumulated precipitation generated by the COSMO-ME operational
model, integrated over several intervals (3, 6, 12 and 24 hours) including 0 hours, i.e. the instantaneous
precipitation. The performances of the COSMO model are routinely monitored by the COSMO
Consortium, and several output features are systematically recorded. The H-SAF parameters are not
directly monitored, but CNMCA has used the records to extract a few significant performance
parameters, as follows:
only over continent, since only raingauge networks are used for comparison;
only accumulated precipitation, over 6, 12 and 24 h;
categorisation over three accumulated precipitation amounts: < 1 mm ; 1 to 10 mm ; > 10 mm
selected statistical scores:
- Frequency BIas (FBI)
- Probability Of Detection (POD)
- False Alarm Rate (FAR)
- Equitable Threat Score (ETS).
The statistical scores stem from the Contingency Table.
Contingency Table
Observed
yes no total
yes hits false alarms forecast yes
Forecast no misses correct negatives forecast no
total observed yes observed no total
where:
- hit: event forecast to occur, and actually occurred
- miss: event forecast not to occur, but actually occurred
- false alarm: event forecast to occur, but actually not occurred
- correct negative: event forecast not to occur, and actually not occurred
The statistical scores are defined as follows.
yesobserved
yesforecast
misseshits
alarmsfalsehitsFBI Range: 0 to infinity. Perfect score: 1
yesobserved
hits
misseshits
hitsPOD Range: 0 to 1. Perfect score: 1
yesforecast
alarmsfalse
alarmsfalsehits
alarmsfalseFAR Range: 0 to 1. Perfect score: 0
random
random
hitsalarmfalsemisseshits
hitshitsETS with
total
yesforecastyesobservedhitsrandom
ETS ranges from -1/3 to 1. 0 indicates no skill. Perfect score: 1. This score measures the fraction of
observed and/or forecast events that were correctly predicted, adjusted for hits associated with random
chance (for example, it is easier to correctly forecast rain occurrence in a wet climate than in a dry
climate). The ETS is often used in the verification of rainfall in NWP models because its "equitability"
allows scores to be compared more fairly across different regimes. Sensitive to hits. Because it
penalises both misses and false alarms in the same way, it does not distinguish the source of forecast
error.
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) - Appendix Page 77
2.2 Summary tables of accuracy
Products PR-ASS-1 have been available for H-SAF purposes since early 2008 and since then have been
regularly distributed. PR-ASS-1 actually provides five products:
precipitation rate, and accumulated precipitation over 3, 6, 12 and 24 h.
The following tables record certain accuracy scores for accumulated precipitation (AP), derived at the
CNMCA from the monitoring activity of the COSMO Consortium. It only refers to accumulation
periods of 6, 12 and 24 h over continental areas. Performances are provided for various classes of AP.
The tables are being produced for H-SAF at quarterly intervals, starting on September 2008.
Rolling statistics of comparisons of PR-ASS-1 accumulated precipitation against rain gauges.
Period: 1 September 2008 - 31 August 2009. Number of comparisons: 5000 per quarter. Continental areas.
Accuracy Class Integration over 6 h Integration over 12 h Integration over 24 h
score of AP Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug
> 10 mm 0.80 0.75 0.77 0.82 0.70 0.80 0.75 0.85 0.90 1.20 0.85 0.85
FBI 1-10 mm 1.10 1.25 1.00 0.97 1.00 1.20 1.10 1.21 1.00 1.20 1.10 0.94
< 1 mm 1.20 1.30 1.10 1.12 1.10 1.20 1.20 1.58 1.10 1.10 1.00 1.04
> 10 mm 0.30 0.20 0.35 0.29 0.20 0.30 0.25 0.35 0.30 0.50 0.35 0.37
POD 1-10 mm 0.50 0.40 0.55 0.32 0.50 0.50 0.55 0.44 0.55 0.65 0.60 0.45
< 1 mm 0.60 0.70 0.60 0.38 0.60 0.75 0.70 0.53 0.75 0.80 0.75 0.54
> 10 mm 0.80 0.85 0.75 0.64 0.20 0.30 0.25 0.59 0.65 0.75 0.55 0.57
FAR 1-10 mm 0.70 0.65 0.60 0.65 0.50 0.50 0.55 0.62 0.45 0.50 0.40 0.53
< 1 mm 0.50 0.45 0.40 0.66 0.40 0.38 0.40 0.66 0.35 0.28 0.30 0.48
> 10 mm 0.10 0.10 0.15 0.19 0.15 0.15 0.20 0.23 0.20 0.20 0.25 0.23
ETS 1-10 mm 0.17 0.20 0.20 0.19 0.25 0.30 0.28 0.23 0.35 0.35 0.35 0.26
< 1 mm 0.40 0.38 0.45 0.2 0.38 0.35 0.40 0.23 0.38 0.40 0.45 0.3
Period: 1 September 2009 - 30 November 2009. Number of comparisons: 5000 per quarter. Continental areas.
Accuracy Class Integration over 6 h Integration over 12 h Integration over 24 h
score of AP Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug
> 10 mm 0.84 0.97 0.92
FBI 1-10 mm 0.89 1.03 0.95
< 1 mm 0.92 1.09 0.99
> 10 mm 0.30 0.49 0.51
POD 1-10 mm 0.42 0.58 0.59
< 1 mm 0.54 0.68 0.68
> 10 mm 0.64 0.50 0.45
FAR 1-10 mm 0.53 0.44 0.38
< 1 mm 0.41 0.38 0.31
> 10 mm 0.19 0.31 0.32
ETS 1-10 mm 0.23 0.36 0.37
< 1 mm 0.35 0.42 0.43
Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) - Appendix Page 78
2.3 Main factors affecting accuracy
The ability of a NWP model to predict precipitation strongly depends on the structure and the objective
of the model. A short list of factors affecting quantitative precipitation forecasting follows:
COSMO-ME is a limited-area model (LAM). It is optimised for the small-synoptic scale (few tens
of kilometres wavelengths). for multi-purpose application at national level.
Although the grid-mesh of the model could be reduced to < 3 km. it is not a convection-resolving
model (CRM). Cloud microphysical parameters are carried forward in the model to an extent
limited to fulfil the multi-purpose national requirements.
In any event. a CRM would provide precipitation representation less biased than a LAM. but its use
in operational weather prediction would experience the intrinsic limits of predictability of the
atmosphere at the scale of convection.
The model handles precipitation better as concerns accumulated precipitation. less well as concerns
instantaneous precipitation. The information on precipitation is inferred through the energy and
water budget. that requires sufficiently large time-space scale to install and stabilise. For H-SAF.
accumulated precipitation is computed for 3. 6. 12 and 24 time intervals. and precipitation rate also
is provided. but verification for quality control is applied only for the 6. 12 and 24 hour intervals.
Of course. the quality of precipitation data is linked to the actual daily accuracy of the forecast. that
changes with the availability and quality of the observations for analysis and assimilation on the
specific date.