Eurostat

Expression language (EL) in Eurostat

SDMX - TWG Luxembourg, 5 Jun 2013Adam Wroński

Eurostat

Policy context

The initiative is based on the Communication from the Commission to the European Parliament and the Council on the production method of EU statistics (Document ESSC 2010/05/6). It calls for:

more harmonisation and standardisation of statistical methodologies for data validation within the ESS

harmonising the IT infrastructure and sharing IT tools as a way to facilitate the use of agreed statistical methods, leading to better quality and higher productivity in the processing of statistical data

Eurostat

1.1 EDIT – standard data editing tool

• Available in a stand-alone version, a server version and a freely accessible Web version 1-way SSL and ECAS protected

• The Web version allows editing statistical data by anyone registered without any software installation

• The tool relies on a EL capable of expressing complex rules

Eurostat

1.2 EDIT internal expression language

Custom EL designed specifically for data editing

Attempt to be as simple as possible and still enough flexible to fit the requirements of any known / analyzed domain

The programs describe the rules and are composed of a set of steps with inputs and outputs

Programs difficult to be written by non-programmers

Eurostat

1.3 EDIT use and future

Generic tool today offers more features than in the past. Some MSs preparing, testing or using Edit for their data

In some data collections the stakeholders, anticipating the opportunity, have started organising a common data checking procedure (incl. CVTS, SBS, BOP, FDI and ITS) are beginning to use agreed set of rules

ESS.VIP on Validation will give direction for the future

Eurostat

2.1 ITDG 2011 conclusions

A formal unambiguous EL was needed for encoding the rules so that they can be translated into specific data editing system rule syntaxes

Use of the generic software tools provided by Eurostat optional

Eurostat

3.1 ESS.VIP on Validation

• Ongoing project• Main task of the project: organize and optimize data editing

among MSs and Eurostat for ESS data collections. • Main deliverable: set of standards incl. common ESS EL• Statistician friendly user interface capable to generate rule

sets in EL and standardised documentation of the rules understandable by business users

Eurostat

3.2 Presentation to ESSC – May 2012

The ESSC generally supported the approach and the first steps of the planned project on broad principles and guidelines for a review of the data checking policy in the ESS

Eurostat would present a more detailed proposal on the project parameters to the ESSC meeting in 2013.

The project would focus on a common language and optional software

Eurostat

3.3 EL in Eurostat and the ESS

• A formal unambiguous EL was needed to allow rules encoding so that they can be translated into other ELs

For more efficient production chain with responsibilities clearly assigned to the different actors

Friendly to statisticians – if possible, the rules to be expressed in a human understandable way

To be able to treat both micro data and aggregate data;

To allow exchange of rules among ESS members

Eurostat

3.4 EL under development

• Information model:

The simpler the information model – the more flexible the language

Data model = bi-dimensional data sets consisting of rows and columns

To allow working with all types of incoming files

• Operators/functions/calculations: Statistical needs oriented Act on data model objects = data sets Allow expression of logical operators and computations

Eurostat

3.4. Beyond the EL• Formats description (e.g., variables)• Process description• Data set quality measure• Rule parameters

• EL expressions used in "if", "then", "else", weight, text message

Severity, code, name, message, descriptions Rule sets and execution e.g. conditional Rules storage (registry) Calculations can use the same language

Eurostat

Summary: Eurostat case for EL

• Savings• Common understanding• Easier exchange• Standardisation

Eurostat

EL: Status

• Draft - work on-going – in-depth analysis of the deliverable• A lot has been done but there is still distance to cover

• Mapping to SDMX and because of simple underlying model likely to other standards

• Related to EXL (starting point functionality) but augmented w/ ESS requirements

• Issue: we aim to have first implementable version by the Fall 2013

Eurostat

Thank you!