Validation practices for satellite-based Earth observation data across communities

Tijl Verhoelst (1), William Bell (2), Luca Brocca (3), Claire Bulgin (4), Jörg Burdanowitz (5), Xavier Calbet (6), Reik Donner (7), Darren Ghent (8), Alexander Gruber (9), Thomas Kaminski (10), Christian Klepp (11), Jean-

Christopher Lambert (1), Julian Liman (12), Gabriela Schaepman-Strub (13), Marc Schröder (12), and Alexander Löw (14)

(1) Royal Belgian Institute for Space Aeronomy, Brussels, Belgium (tijl.verhoelst@aeronomie.be), (2) European Centre for Medium-Range Weather Forecasts (ECMWF), Reading, UK, (3) Research Institute for Geo-Hydrological Protection - National Research Council, Perugia, Italy, (4) Department of

Meteorology, University of Reading, Reading, UK, (5) Meteorological Institute, Universität Hamburg, Hamburg, Germany, (6) Spanish Meteorological Agency, AEMET, Madrid, Spain, (7) Magdeburg-Stendal University of Applied Sciences, Magdeburg, Germany, (8) University of Leicester, Leicester, UK,

(9) KU Leuven, Leuven, Belgium, (10) The Inversion Lab, Hamburg, Germany, (11) Max Planck Institute for Meteorology, Hamburg, Germany, (12) Deutscher Wetterdienst, Offenbach, Germany, (13) Department of Evolutionary Biology and Environmental Studies, University of Zurich, Zurich,

Switzerland, (14) Ludwig-Maximilians-Universität München (LMU), Munich, Germany, deceased 2 July 2017

Context & Aims The validation of satellite-based measurements and their uncertainties is a challenge common to all

Earth Observation (EO) communities, each of which attempts to assess the fitness-for-purpose of the

satellite dataset for specific scientific or public-service applications, and to ensure the traceability of

the measurements to fundamental standards. We report here on the activities and outcomes of an

International Space Science Institute (ISSI) team which brought together land, ocean, and atmos-

phere validation experts, with the aim to share expertise and tools across EO communities. This work

lead to the publication of a review paper (Loew et al., Reviews of Geophysics v55, 2017), accessible

through the QR code above, or at https://doi.org/10.1002/2017RG000562.

Co-location mismatch Spatiotemporal co-location is one of the more challenging aspects of a validation exercise. Concerns to be ad-

dressed are:

The need to have co-located measurement close to each other relative to the spatiotemporal scale on

which the variability of the geophysical field becomes comparable to the measurement uncertainties.

If possible, differences in spatiotemporal resolution should be minimized.

The co-location criteria must take into account the need for a sufficient amount of co-located pairs for ro-

bust statistical analysis. This is often at odds with the 1st requirement and a compromise must be made.

Within the different communities, a wealth of techniques have

been developed to optimize the co-location, either on the origi-

nal measurement grids or using interpolation and up– and

downscaling techniques. Nevertheless, some co-location mis-

match usually remains, and various approaches exist to assess

representativeness errors and co-location uncertainty. A proper

consistency check between satellite and reference data takes in-

to account not only the uncertainties of each data set, but also

the uncertainties inherent to the imperfect co-location.

Methodology: the baseline Pair-wise comparisons after co-location and homogenization/harmonization;

Consistency check w.r.t. to ex-ante uncertainties, ideally including mismatch/

representativeness uncertainties;

Scalar metrics, differentiating between systematic and random effects;

Interpretation as a function of measurement influence quantities and geophysical

phenomena.

Methodology: advanced techniques Triple co-location and multiple triple co-location analysis (TCA)

Spectral methods (Fourier and Wavelet)

Field intercomparison and Functional Network Analysis

Consistency through process models

Indirect validation

Data assimilation

Self co-location

...

Recommendations Refine user requirements: They need to be specific, sufficiently differentiated, traceable and fully documented.

Follow metrological (i.e. measurement science) terminology and uncertainty expression and assessment methods.

Reference works are the Vocabulaire International de Métrologie (VIM) and the Guide to the expression of Uncer-

tainty in Measurement (GUM).

Ensure the traceability of reference measurements, including a detailed uncertainty budget.

Ensure the sustained availability of fiducial reference measurements from in-situ and ground-based networks. This

should be a continuous focal point for (space) agencies and service providers.

Go beyond scalar pair-wise comparison metrics and include the satellite and reference measurement uncertainties.

Be aware of scale mismatch: optimize co-location criteria and quantify any remaining co-location uncertainty

Establish and publish best practices, and share traceable, open-source validation software tools.

Examples for each are

available in the Appen-

dix of the review

paper.

User requirements The validation question is essentially: Are the data fit-for-purpose, do they meet the user requirements?

In all communities, a disconnect was found between the detailed output of the advanced validation

methodologies and the often simplistic and/or ambiguous user requirements.

User requirements need to be specific (e.g. about spatial and temporal domain), differentiated (e.g. be-

tween systematic and random effects) and traceable, i.e. with a well-documented origin.

Reference measurements The backbone of any validation exercise is the availability of reference measurements, where

“reference” implies:

Traceability, i.e. they can be linked through an unbroken chain of calibrations and compari-

sons to a metrological reference (Système International or community-agreed);

Full uncertainty characterization;

Availability;

At the network level: Sufficient coverage of the potential parameter space.

While the maturity of reference measurements varies across communities, their importance is real-

ized in all, as is evident from the increased interest from (space) agencies in supporting in-situ and

ground-based networks.

Terminology A first challenge within and across communities is the language used to report on validation methods

and results. Within several domains, an effort is ongoing to adopt the nomenclature used in the me-

trology (i.e. measurement science) community, which is described in detail in the Vocabulaire Inter-

national de Métrologie and the Guide to the expression of Uncertainty in Measurement (VIM &

GUM, JCGM, 2011,2012). See also the CEOS Terms, Definitions and Cal/Val Best Practices and the

WMO Quality Management Framework.