TASK TEAM ON WIGOS METADATA (TT-WMD)

FIRST SESSION

Manuel Bañón (mbanong@aemet.es)State Meteorological Agency (AEMET), Alicante, Spain

GENEVA, SWITZERLAND 11 – 15 MARCH 2013

The purpose of the WIGOS (1) initiative is to establish an upgraded environment for observing systems that will: Standardize observing requirements (standards, guidelines and best practices) defined for the user’s communityAvoid multiple, overlapping sources of information in WMO and WMO cosponsored Programmes

Metadata and Quality Management System (QMS)Ensure that the quality of observation is documentedEnsure the availability of associated comprehensive metadata for

better products and servicesEnsure implementation of basic principles of the observing

system/network targeted Quality Management System

Provide data specifications describing required data or productsIntegrate observing systems following the aims and rules defined within WIGOS frameworkStandardize access to data and productsProvide the tools for easy, single access for the user’s community to the set of standards, guidelines and best practices for the management of observing systems

(1) FRAMEWORK FOR WIGOS STANDARDIZATION OF THE SURFACE-BASED COMPONENTS. EC-WG/SG-WIGOS-2/Doc.

5.1

WIGOS metadata must be considered from user point of view, taking into account user requirements Meaningful use of observations requires knowledge about the conditions under which the observation was made; WIGOS metadata is what provides that supporting information.

User requirements be collected

(TT-WMD) Terms of ReferenceIn accordance with guidance and recommendations of Cg-XVI, EC and ICG-WIGOS-1:

1.To identify the information that is needed to allow the majority of users to use WIGOS observations in appropriate contexts and in a defensible way

2.To create the WIGOS Core Metadata Standard that allows the essential information to be exchanged unambiguously, regardless of the format used for the transfer

3.To define a mechanism for maintaining the WIGOS Core Metadata Standard, including how metadata might be provided that is additional to the Core and coordinate with the ICG-WIGOS Task Team on Regulatory Material (TT-WRM) on any appropriate documentation as needed for WIGOS related Manual(s) and Guide(s)

4.To implement within the WIGOS Core Metadata Standard, and the WMO Core Metadata Profile, a standard method of providing users with an indication of the quality of the data, and to do so in a way that distinguishes with the quality management of the data (“quality of the observation”) and ensuring that the user is able to identify which applications the data are suitable for (“classification of the observation”)

5.To coordinate regularly with the ICG-WIGOS as needed and report at least annually to the ICG-WIGOS on the progress

6.To complete its tasks and hand over additional requirements to its successor (if required) in time for approval by Cg-17

What is a Metadata Standard?

A Community agreed upon declaration that provides a structure to describe data with:

• Common terms to allow consistency between records• Common definitions for easier interpretation• Common language for ease of communication• Common structure to quickly locate information

In search and retrieval, standards provide:

• A reliable and predictable format for computer interpretation• A uniform summary description of the data set

Many metadata standards exist

• ISO 19115 Geographic information: Metadata

• Emphasis on geospatial data and services

• World Meteorological Organization Core Metadata Profile

(WMO) – a profile of ISO 19115

• Emphasis on meteorological data

• Content Standard for Digital Geospatial Metadata (CSDGM)

• Federal Geographic Data Committee (FGDC)

• Emphasis on geospatial data

• Directory Interchange Format (DIF)

• Emphasis on Earth science data

• Dublin Core Element Set

• Emphasis on web resources, publications

Definition of Metadata

Information to let you and others find, understand, and use your data both now and in the future

“Data about data”describes data

what are the data?where are they for?who is responsible?what are the units?

Need to agreewhat needs to be describedhow they will be organisedwhat they will be called

So thatnon-specialists can find dataapplications can handle data

You get the idea … so, what’s needed?

• Negotiated between you and your team (data producers), data developers and the data archive• Created / collected as part of workflows associated with lifecycle of the data

Based on key questions, e.g., :

• How will I and others find my data?• What will future scientists need to know, to understand and re-use the data?• What will data (product) developers need to create data services and products that use my data?• What does the data archive team need from the data producer team (as the ones who know the data best) in order to preserve it for the long term?

the WHO, WHAT, WHEN, WHERE, HOW, WHY about your data

Specific’s from different perspectives?

When submitting data:• Why was the data was created?• What limitations, if any, do thedata have?• What does the data mean?• Who should be cited if someonepublishes something that utilized your data?

When receiving data:• What are the data gaps?• What processes were used for creating the current data?• Are there any fees associated with the data?• In what scale were the data created?• What do the values in the tablesmean?• What software do I need in orderto read the data?• What projection is the data in?• Can I give this data to someone else?

METADATA (1)

Good metadata are needed to ensure that the final data user has no doubt about the conditions in which data have been recorded, gathered and transmitted, in order to extract accurate conclusions from their analysis.

Metadata have a key role in the process of creating such datasets, as the knowledge of the station history provides increased confidence in the statistical techniques employed to ensure that the only variations that remain in a climate time series are due to actual climate variability and change.

Meteorological data are influenced by a wide variety of observational practices. Data depend on the instrument, its exposure, recording procedures and many other factors. There is a need to keep a record of all these metadata to make the best possible use of the data. Additional information will be of great advantage for the data users, as well as for the providers.

METADATA (2)

A good metadata archive helps the NMHSs in asset management and other administrative procedures, as data existences and observing conditions are kept in good order. It also can be said that good metadata helps society to gain a better understanding of weather and climate related processes, as well as climate change.

GCOS Climate monitoring principles: “The details and history of local conditions, instruments, operating procedures, data processing algorithms and other factors pertinent to interpreting data (i.e. metadata) should be documented and treated with the same care as the data themselves.” (WMO 2002).

Metadata Types and Functions

• Provenance and Context• Physical and conceptual environment needed to

understand data• Important for use and re-use

• Discovery including• Citation to enable proper credit, authority and

identification• Access and use restrictions

• Preservation / Archiving• Facilitating utility of data over time

• Project Documentation• Accumulation of important facts, guidelines,

explanations about the project

13

Metadata requirements for climate (1)

STATION IDENTIFIERS: name, aliases, station number or code in other networks, opening/closing dates, type of station and station information contact

GEOGRAPHICAL DATA: latitude and longitude, elevation above mean sea level and relocations

LOCAL ENVIRONMENT: at different scales, it is recommended to keep track of several attributes as proximity and size of large water surfaces, urbanized areas and mountain ranges. At toposcale terrain slope, both steepness and direction, forests, crops and other roughness, nearby obstacles such as trees or houses (at airports: airplanes) and proximity to irrigation.

INSTRUMENT EXPOSURE: obstacles and shelter, ground cover (roughness, albedo, pavement, sand, etc.)

Metadata requirements for climate (2)

INSTRUMENTATION: instrument manufacturer, model of instrument, with size and identification, output type and sensitivity, transducer type, response time

DATA RECORDING AND TRANSMISSION: units and resolution,

range of recorded variations, response time and/or sampling

time, averaging period if applicable

oOBSERVING PRACTICES: observer, meteorological operational

procedures, checking instruments, replacing disposable items,

housekeeping, observed elements, observing times.

oCORRECTIONS MADE BY OBSERVER: conversion tables or

computer programs should be kept

Metadata requirements for climate (3)

• DATA PROCESSING: the corrections made to the data, the

procedures of quality control applied, the adjustments made to

ensure its homogeneity, and the data estimated and filled in,

after applying the interpolation procedure selected.

•UNITS: º Celsius o ºFahrenheit

•SPECIAL CODES: like missing data, wrong value, trace

precipitation, non-precipitation, or accumulated precipitation

•CALCULATIONS: time averaging (daily, monthly and so on) of

elements

•QUALITY CONTROL: Gross error checking, Tolerance tests,

Internal consistency check

•HOMOGENEITY ADJUSTEMENTS: elements, periods,applied test,

etc.

WMO

Information sources

Guide to climatological practices (WMO N° 100)

Guidelines on climate metadata and homogenisation (WMO/TD-No.1185 )

Climate Database Management Systems Specification Document

”Creating Documentation and Metadata”. Nancy J. Hoebelheinrich, Stanford University Libraries

My experience tells me that the basic metadata are very difficult to obtain. The technicians are reluctant to write and we need to specify very in a simple and clear that metadata are needed before and during the installation of a station and to be closely monitored during the operation thereof.

ThanksThanks