CORA – COmmon Reference Architecture · CORA – COmmon Reference Architecture Monica Scannapieco...

Eurostat

CORA –COmmonReferenceArchitecture

Monica Scannapieco Istat

Carlo Vaccari

Università di Camerino

Antonino Virgillito

Istat

Eurostat

Outline

• Introduction (90 mins)

• CORE Design (60 mins)

• CORE Architectural Components (90 mins)

• Illustration of CORE Platform (135 mins)

• Case studies (90 mins)

• CORE Follow-up (60 mins)

Eurostat

Introduction

Eurostat

CORE Generalities

• Principal Outcome: Environment for the definitionand execution of standard statistical processes

– Definition of a process in terms of availableservices

– Execution of the composed workflow

Eurostat

CORE Generalities

Service Repository

Process

View

“Plug and play” approach to process execution

Eurostat

CORE Generalities

“Plug and play” approach to process execution

Service Repository

Process View Data View

Eurostat

Why CORE?

STOP

Allocation(MAUSS – R)

Selection(SAS Script)

Estimation(ReGenesees)

START

Eurostat

Why CORE?

STOP




START

TechnologicalHeterogeneity

• Different technologies

• Different formats• …

Eurostat

Why CORE?

STOP




START

TechnologicalHeterogeneity

• Different technologies

• Different formats• …

DataHeterogeneity

• Different names for variables

• Variables as combinations of other variables

• ...

Eurostat

Why CORE?

• Technological heterogeneity can be solved by

solutions available on the market

CORE permits to solve both technological and data

heterogeneity in a single environment

Eurostat

CORE Vision

1. Abstract services: well-defined, technology-independentfunctionalities implemented by different IT tools;

2. Statistical process: workflow defined in terms of availableservices;

3. Data model: standardization of the semantics/format ofservices data, i.e. definition of the domain entities involved asinput/output between services.

Eurostat

CORE Vision

1. Abstract services: well-defined, technology-independent functionalities implemented by IT tools

Allocation

Selection

Estimation

(MAUSS – R) (ReGenesees)

(SAS Script)

Eurostat

CORE Vision

3. Data model: standardization of the semantics/format of services data

Allocation

Selection

(MAUSS – R)

(SAS Script)

<schema name="DEMO_DD">

<entity name="SamplePlan">

<property name="VAR"/>

<property name="SIZE"/>

. . .

</entity>

</schema>

3.1 Domain descriptor (DD)

3.2 Mapping to/from DD

DD

Eurostat

CORE Design Tasks - 1

• Design of services

• Definition of integration APIs (IAPIs)

• Data conversion from/to CORA model to/fromtool specific format

• Graphical front ends for designing schemas and mappings

Eurostat


• Design of processes

• How to define and execute processes withinCORE

• Modelling language

• Execution

• Visual interfaces design

• Design of a service repository

Eurostat


• Design of exchanged data

• Definition of data models and formats (plainXML/XSD, SDMX…) to be used for data exchanges

• Definition of metadata necessary for processexecution

• SDMX Relationships

Eurostat

CORE Design

Eurostat

CORE Design: Services

• Abstract services: specify a well-definedfunctionality in a technology-independentway

• An abstract service can be implementedby one or more concrete services, i.e. IT tools

• Examples: sample allocation, record linkage, estimates and errorscomputation, etc.

Eurostat


• GSBPM classification

• Documentation purpose

• Provided that a CORE service can be linked to IT tools, GSBPM tagging enables the performance of a search e.g. retrieving“all the IT tools implementing the 5.4 Impute

subprocess of GSBPM proposal”

Eurostat


• Service inputs and outputs

• Specified by logical names

• Characterized with respect to their “role” in data exchangeNon-CORE: if they are not provided by/to otherservices of the process, but are only “local” to a specific service

CORE: they are passed by/to other services and hencethey do need to undergo CORE transformations

Eurostat

CORE Design: Data and Metadata

• They are specified as service inputs and outputs

• Logical names link them to previously specifiedservices

• Non-CORE data only need the file system pathwhere they can be retrieved

Eurostat

CORE Design: CORE Data

• The specification of CORE data is provided by 3 elements:

• Domain descriptor

• CORE data model

• Mapping model

Eurostat

Domain Descriptor: Model

• Entity

• Like “entities” in Entity Relationships

• Entity properties

• Like “attributes” in Entity Relationships

• Very simple (meta-)model

Eurostat

Domain Descriptor: Example

<schema name="DEMO_Domain_Descriptor">

<entity name="SamplePlan">

<property name="STRATIFICATION_VAR"/>

<property name="STRATUM_SAMPLE_SIZE"/>

<property name="STRATUM_POPULATION_SIZE"/>

</entity>

<entity name="Enterprise">

<property name="IDENTIFIER"/>

<property name="STRATIFICATION_VAR"/>

<property name="WEIGHT"/>

<property name="SAMPLING_FRACTION"/>

<property name="ENTERPRISE_FLAG"/>

<property name="EMPLOYEES_NUM"/>

<property name="VALUE_ADDED"/>

<property name="AREA"/>

</entity>

</schema>

Eurostat

Domain Descriptor Role

• Role of the Domain Descriptor (DD): fromservice-to-service data mapping to service-to-global data mapping

Eurostat

CORE Data Model: Role

• Specified once and valid for all processes

• Extensible, i.e. core tag, data set kind, column kind can be modified

• Adds more semantics to data

• Example of usage: mapping to othermodels

Eurostat

CORE Data Model

• Rectangular data set

• CORE tag:

• Data set level (mandatory)

• Column level (optional)

• Rows level (optional)

• Data set kind

• Column kind

Eurostat

CORE Data Model Role

• Specified once and valid for all processes

• Extensible, i.e. core tag, data set kind, column kind can be modified

• Adds more semantics to data

• Example of usage: mapping to othermodels

Eurostat

Mapping Model

• Rectangular data assumption

• Mapping is intended to be specified with respect to Domain

Descriptor

• Columns are to be mapped to properties of an entity

• It contains the specification of how CORE data model

concepts are associated to data

Eurostat

CORE Logical Architecture

Eurostat

CORE GUIs

• Process design

• Ad-hoc customization of an existing tool (Oryx)

• Service data flow

• Service design

• Set of interfaces for the definition of services and

related data flow

• Data design

• Set of interfaces for the specification of domain

descriptors and mapping files

Eurostat

Use Case Specification

• CORE (Principal) Users

33

Eurostat

Use Case Specification: Tool Management

34

Eurostat

Use Case Specification: Service Management

35

Show Tools' List

Add Service

Modify Service

Delete Service

«uses»

Show Services' List

Statistical User Service Management«uses»

«uses»

«uses»

«uses»

«uses»

Select Service

SelectTool

«uses»

Eurostat

Use Case Specification: Process

36

Eurostat

Process design: Oryx

• Oryx is an academic open source frameworkfor graphical process modeling

• Based on web technology

• Extensible via a plugin mechanism and new stencil sets

• Supports BPMN and other processmodeling languages

• Programming language Javascript and Java, internal data format based on RDF

Eurostat

Stencil Set

• Set of graphical objects and rules that specify how to relate those graphical objects to others

• Additional properties that can later be used by other applications or Oryx extensions (e.g. setting element colors and visibility)

• Can be used to build process models

Eurostat

The CORE Stencil Set

• Graphical representation of CORE processes

• Easy-to-use editor (desktop feeling)

• Easy-to-extend source (JSON)

• Defined from BPMN

• Guarantees complete BPMN compliance

Eurostat

Integration APIs

• Purpose: wrapping a tool by a CORE

service

• Translates inputs and outputs of the tool in a

completely transparent and automatic way

Eurostat

Repository

• Processes and their instances

• Services with their GSBPM and CORE classifications

• Tools and their runtime features

• Data with their logical classification within CORE processes

Eurostat

Database design: Overview

42

Eurostat

Database Design: Principal Entities

43

• Service & Tool

Eurostat


-id

-name

-GSBPMtag

-coretag

-version

-namespace

service

-id

-name

-definition

process

0..*

0..*

44

• Service & Process

Eurostat


45

• Operational Data

Eurostat

Process Engine

• Official statistics processes can be viewed from two

perspectives:

• Functional: they are data-oriented, reflecting a common

feature of scientific workflows

• Organizational: they are workflow-oriented, have the

complexity of real production lines, with the need for

harmonizing the work of different actors

Eurostat

Process Engine

• Hence our process engine has two layers …

DATA FLOW CONTROL SYSTEM

WF ENGINE

Complex control flows

� Syncronizing constructs, cycles,

conditions, etc.

� E.g.: Interactive multi-user

editing imputation

Simple control flows

� Sequence of tasks is composed

by connecting the output of one

task to the input of another

� Data intensive operations

Eurostat

Worflow Engine Selection Process

• CORE workpackage (WP$) led by INSEE

• Business Process Management (BPM) platforms:

• Bonita (http://www.bonitasoft.com/)

• Activiti (http://www.Activiti.org/)

• ActiveVOS (http://www.activevos.com/)

Eurostat

Worflow Engine Selection Process

Eurostat

SDMX Relationships

• Both propose an information model

• CORE information model

– takes explicitly process dimension into account

– Data dimension spanning over the whole statistical process

• SDMX information model

– focused on data exchange (though processes are also considered)

Eurostat

SDMX Relationships

• CORE information model

– Deals with both microdata and macrodata

• SDMX information model

– Mainly deals with macrodata

Eurostat

SDMX Relationships

1. Can we use SDMX for micro and macro data exchanges in a CORE process?

– Need for mapping of information models

2. What about metadata?

– CORE: Data and metadata managed at the same way

– SDMX: distinction between structural metadata and reference metadata. Possibility of having domain knowledge codified through concepts

Eurostat

SDMX Relationships

• Choices and steps:

• Conversion from CORE XML to CSV in order to use SDMX conversion tools

• Starting from the CORE file structure it was created a SDMX DSD (Data Structure Definition)

• SDMX data format : cross-sectional

• Once prepared the DSD, we proceeded to convert the CORE file using the SDMX Converter tool

Eurostat

SDMX Relationships

• CORE-to-SDMX Conversion Proof-of-Concept

• Setting:

• Italian Time-Use Survey

• Data Structure Wizard and the SDMX Converter

• Compute Estimates and Sampling Errors” (as the aggregated data dissemination phase)

• Choices and steps:

• Conversion from CORE XML to CSV in order to use SDMX conversion tools

• Starting from the CORE file structure it was created a SDMX DSD (Data Structure Definition)

Eurostat

SDMX Relationships

• The experiment has shown the feasibility of the conversion to SDMX format of a data file obtained as a CORE output

• Not automated conversion:

• Manual mapping of the CORE output’s fields to the dimensions and attributes of the SDMX DSD

• SDMX does not manage more than measure, it was necessary the verticalization of the CORE output file in order to convert it to the SDMX cross sectional

Eurostat

Architecture Deployment

• Web-based architectured centered on a

centralized component

– CORE Environment

• Different CORE deployments can co-exist

– Intra- or Inter- organization

• Services can be remotely executed

– Support is needed in the form of a

distibuted component for tool execution

and data transfer

Eurostat

Types of service runtime

• Batch

– Tool executed by a command line call

– Can be automated

• Interactive

– User interact with the tool through a tool-provided GUI

– Cannot be automated

• Web service

– No tool – procedure distributed on a web service actived

by a programming language call

– Can be automated

Eurostat

CORE Technical Architecture

GUI Definition Repository

Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent

Batch-Interactive runtime

Web service runtime

Web container

Run on the machine on which the tool is deployed.Is responsible for: -Preparing the input -Gathering the output-Activating the tool

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


The process enginesignals a service must be executed

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


Service definition isextracted from the repository, as well asthe required datasetsand the correspondingmappings

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


Datasets are convertedaccording to the mapping

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


Converted datasets are transferred to the remote runtime

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


The tool is activated by the runtime agent

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


The output datasets are gathered and sent back to the CORE environment

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


Datasets are convertedback to CORE format according to the mapping

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


Converted datasets are stored in the repository

Eurostat



Integration APIs

Process Engine

Runtime

CORE Environment

Web service client

Remote activation

Runtime

Runtime agent


The process continues its execution

Eurostat

Scenario 1

• Remote execution command line/GUI

– Physical layers: CORE env, Service

– AGENT

Eurostat

Scenario 2

• Remote execution web service

– Physical layers: CORE env, Service

Eurostat

CORE Scenario

Eurostat

Why a Process Scenario?

• Helps to clarify ideas and to asses their feasibility

• Forces to make newly proposed solutions concrete

• Can/will be used as empirical test-bed during the whole implementation cycle of the CORE environment

71

Eurostat

How did we build the Scenario?

• Rationale for our Scenario:

• Naturality: involves typical processing steps

performed by NSIs for sample surveys

• Minimality: very easy workflow (no

conditionals, nor cycles), can be run without a

Workflow Engine

• Appropriateness: incorporates as muchheterogeneity as possible: heterogeneity isprecisely what CORE must be able to get rid of

72

Eurostat

Spreading Heterogeneity over the

Scenario

• The Scenario incorporates both:

• Data Heterogeneity

Via data exchanged by CORE services belonging to the scenario process

• Technological Heterogeneity

Via IT tools implementing scenario sub-processes

73

Eurostat

Data Heterogeneity

• The Scenario entails different levels of data

heterogeneity:

• Format Heterogeneity: CSV files, relational DB tables, SDMX XML files involved

• Statistical Heterogeneity: both Micro and Aggregated Data involved

• “Model” Heterogeneity: some data refer to ordinary real-world concepts (e.g. enterprise, individual, …), some other to concepts arising from the statistical domain (e.g. stratum, variance, sampling weight, …)

74

Eurostat

Technological Heterogeneity

• The Scenario requires to wrap inside

CORE-compliant services very different IT

tools:

• simple SQL statements executed on a relational DB

• batch jobs based on SAS or R scripts

• full-fledged R-based systems requiring a human-computer interaction through a GUI layer

75

Eurostat

The Scenario at a glance

76

START

ComputeStrata

Statistics

Allocate the

Sample

Selectthe

Sample

Compute Estimates and Sampling Errors

CalibrateSurvey Data

CollectSurvey Data

Check and Correct

Survey Data

Store Estimatesand Sampling

Errors

Convert to

SDMX

STOP

ALLOCATION

ESTIMATION

Eurostat

Sample Allocation Subprocess

• Overall Goal: determine the minimum

number of units to be sampled inside

each stratum, when lower bounds are

imposed on the expected level of

precision of the estimates the survey

has to deliver

• Two statistical services are needed:

• Compute Strata Statistics

• Allocate the Sample

77

START

Compute

Strata Statistics

Allocate the Sample

AL

LO

CA

TIO

N

START

Compute

Strata Statistics

Allocate the Sample

AL

LO

CA

TIO

N

Eurostat

Compute Strata Statistics

Service• Goal: compute, for each stratum,

the population mean and standard deviation of a set of auxiliary variables

• IT tool: a simple SQL aggregated query with a group-by clause• NSIs usually maintain their sampling frame(s) as Relational DB tables

• Integration API: must support Relational/CORE transformations

• CORA tag: “Statistics”

78

START

ComputeStrata Statistics

Allocate the Sample

AL

LO

CA

TIO

N

START

ComputeStrata Statistics

Allocate the Sample

AL

LO

CA

TIO

N

Eurostat

Allocate the Sample Service

• Goal: solve a constrained

optimization problem to find and

return the optimal sample

allocation across strata

• IT tool: Istat MAUSS-R system

• implemented in R and Java, can be run either in batch mode or interactively via a GUI

• Integration API: must support

CSV/CORE transformations

• MAUSS handles I/O via CSV files


79

START

Compute

Strata Statistics

Allocate the Sample

AL

LO

CA

TIO

N

START

Compute

Strata Statistics

Allocate the Sample

AL

LO

CA

TIO

N

Eurostat

Sample Selection Subprocess

• Goal: draw a stratified random sample of units from the sampling frame, according to the previously computed optimal allocation

• IT tool: a simple SAS script to be executed in batch mode

• Integration API: CSV/CORE transformation• SAS datasets have proprietary, closed format �� we’ll not support direct SAS/CORE conversions

• CORA tag: “Population”• output stores the identifiers of the units to be later surveyed + basic information needed to contact them

80

Selectthe Sample

Selectthe Sample

Eurostat

Estimation Subprocess

• Overall Goal: compute the

estimates the survey must

deliver, and asses their

precision as well

• Two statistical services are

needed:

• Calibrate Survey Data

• Compute Estimates

and Sampling Errors81

Compute Estimates

and Sampling Errors

Calibrate

Survey Data

ES

TIM

AT

IONCompute Estimates

and Sampling Errors

Calibrate

Survey Data

ES

TIM

AT

ION

Eurostat

Calibrate Survey Data Service

• Goal: provide a new set of weights (the “calibrated weights”) to be used for estimation purposes

• IT tool: Istat ReGeneseessystem• implemented in R, can be run either in batch mode or interactively via a GUI

• Integration API: can use both CSV/CORE and Relational/CORE transformations

• CORA tag: “Variable”

82

Compute Estimates

and Sampling Errors

Calibrate

Survey Data

ES

TIM

AT


and Sampling Errors

Calibrate

Survey Data

ES

TIM

AT

ION

Eurostat

Estimates and Errors Service

• Goal: use the calibrated weights to compute the estimates the survey has to provide (typically for different subpopulations of interest) along with the corresponding confidence intervals

• IT tool: Istat ReGenesees system

• Integration API: can use both CSV/CORE and Relational/CORE transformations

• CORA tag: “Statistic”

83

Compute Estimates

and Sampling Errors

Calibrate

Survey Data

ES

TIM

AT


and Sampling Errors

Calibrate

Survey Data

ES

TIM

AT

ION

Eurostat

Store Estimates Subprocess

• Goal: persistently store the

previously computed survey

estimates in a relational DB

• e.g. in order to subsequently

feed a data warehouse for

online publication

• IT tool: a set of SQL statements

• Integration API: Relational/CORE

transformation again

• CORA tag: “Statistics”84

Store Estimates

and Sampling Errors

Store Estimates

and Sampling Errors

Eurostat

Convert to SDMX Service

• Goal: retrieve the aggregated data

from the relational DB and directly

convert them in SDMX XML format

• e.g. to later send them to

Eurostat

• IT tool: ???

• Integration API: must support

SDMX/CORE transformations


85

Convert to

SDMX

STOP

Convert to

SDMX

STOP

Eurostat

Scenario Open Issues

• Besides I/O data, CORE must be able to handle “service

behaviour parameters”. How?

• e.g. to analyze a complex survey, ReGenesees needs a

lot of sampling design metadata, namely information

about strata, stages, clusters identifiers, sampling

weights, calibration models, and so on

• Enabling the CORE environment to support interactive

services execution is still a challanging problem

• we plan to exploit MAUSS-R and/or ReGenesees to test

the technical feasibility of any forthcoming solution

• How to implement a SDMX/CORE converter?

86

Eurostat

Demo Scenario

• Involves 3 typical processing steps performed by NSIs for sample surveys:

• Sample Allocation

• Sample Selection

• Estimation

• It has been used as empirical test-bed during the whole implementation cycle of the CORE environment

87

Eurostat

Rationale for the Scenario

• Minimality: very easy workflow (no conditionals,

nor cycles), can be run without a Workflow

Engine

• Appropriateness: addresses heterogeneity issues

• heterogeneity is precisely what CORE must be ableto get rid of

88

Eurostat

Spreading Heterogeneity over the

Scenario

• The Scenario incorporates both:

• Data Heterogeneity: Via data exchanged by CORE services belonging to the scenario process

• Technological Heterogeneity: Via IT tools implementing scenario services

A batch job based on a SAS script

Two full-fledged R-based systems

89

Eurostat

The Scenario at a glance

90

START

MAUSS-R

ALLOCATION

SAS SCRIPT

SELECTION

STOP

ReGeneseesSystem

ESTIMATION

Eurostat

Sample Allocation Service

• Overall Goal: determine the

minimum number of units to be

sampled inside each stratum,

when lower bounds are imposed

on the expected level of precision

of the estimates the survey has to

deliver

• IT tool: Istat MAUSS-R system

• implemented in R and Java


91

START

MAUSS-R

ALLOCATION

Eurostat

Sample Selection Service

• Goal: draw a stratified random

sample of units from the

sampling frame, according to the

previously computed optimal

allocation

• IT tool: a simple SAS script to

be executed in batch mode

• CORA tag: “Population”

92

SAS SCRIPT

SELECTION

Eurostat

Estimates and Errors Service

• Goal: compute the

estimates the survey has

to provide (typically for

different subpopulations of

interest) along with the

corresponding confidence

intervals

• IT tool: Istat ReGenesees

System

• R-based

• CORA tag: “Statistics”93

STOP

ReGeneseesSystem

ESTIMATION

Eurostat

CORE Follow up

Eurostat

CORE in Istat

• CORE is an Action of the Istat strategic plan Stat2015

• Period 2013-2015

• Objective: Usage of CORE platform in production scenarios ofIstat

• Plan for 2013:

• Implementation of engineering activities

• Usage of CORE to support sharing of generalized software functionalities– currently studying how to

• Usage of CORE in dissemination flow of the corporate architecture in conjunction with an ETL tool (Kettle) – currentlystudying how to

Eurostat

Development of CORE Services for ESS: Issues

• CORE is strictly related to the “Shared Services” technical cross-cutting issue of the ESS VIP (Vision Infrastructure Project) Programme

• Period 2013-2018

• Role: Supporting standardisation of the communication

protocol among standard statistical services

Eurostat

Issue 1: Relationship between CORE and SOA

• Hints for answering issue 1:

• CORE adopts a SOA design approach

• CORE services can be deployed as Web Services

• CORE do “imply”/”include” SOA technologies

• SOA technologies does not “imply”/”include” CORE

Eurostat

Issue 2: Relationship between CORE and GSIM


• CORE did not have the purpose of defining yet another information model

• CORE takes into account the need for an information model

• Introduced only for demonstration purposes

• Hence from a design perspective CORE is open to adopt a full-fledged information model like GSIM

• CORE Model slot/CORE Domain Descriptor slot

Eurostat

Issue 3: Relationship between CORE and DDI/SDMX


• DDI/SDMX provides “logical” information models

• GSIM serves a documentation purpose

• DDI/SDMX serve (mainly) a representation purpose

• CORE could be integrated with DDI/SDMX by:

• Mapping rectangular datasets representation of CORE data to such models

• Mapping in principle feasible as CORE model “less expressive”

Eurostat

Issue 4: CORE Deployment Issues in the ESS –SOA supporting platform


• Need for designing a CORE deployment for the ESS

• Service repositories

• Data exchanges

• Security issues

• Performance issues

• ...

CORA – COmmon Reference Architecture · CORA – COmmon Reference Architecture Monica Scannapieco...

Documents

Transcript of CORA – COmmon Reference Architecture · CORA – COmmon Reference Architecture Monica Scannapieco...