1Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Challenges in the provision of large data sets to multiple user communities

14-16 July 2014, Boulder CO

MONITORING WEATHER AND CLIMATE FROM SPACE

2Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Mission Statement

We are the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT).

EUMETSAT is an intergovernmental organisation and was founded in 1986. Our purpose is to supply weather and climate-related satellite data, images and products

• 24 hours a day, • 365 days a year

to the National Meteorological Services of our Member and Cooperating States in Europe, and other users worldwide.

www.eumetsat.int

3Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETSAT Headquarters (Darmstadt, Germany)

4Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETSAT Data Services - High Level Overview

Users

Discover, search, register, order and subscribe

Subscribe to services

Discover/search products

Register to messages

Distributed search, remote order

Order/ subscription products

Subscribe to services

GEONETCast Catalogue

EUMETCast/ GEONETCast

Unified Meteorological

Archive(U-MARF)

EUMETSAT Partner Agencies/

GMES, WMO

User Notifcations(UNS)

EUMETCast / GEONETCast

Other Dissemination

Services

Registration

Discovery (Product Navigator)

Order Followup

Access to applications & data

EUMETSAT EO Portal

5Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETSAT Data Services - Data Flows

EUMETSAT GROUND INFRASTRUCTURE

6Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Single Access Point

One stop shop

Data discovery

User registration (>10.000)

Subscription for data services: Near-real time data

Online Data Access

Long Term Archive

Additional Image Gallery (WMS/ WCS)

User Notifications

Operational Real-Time Services Status Indicator

EUMETSAT Earth Observation Portal

7Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETCast Dissemination Near Real Time data delivery to users in 3 continents via Satellite

Users (2/2014)

4

3

2-10

11-50

51-200

201-500

1 User

500+

0 User

3,280 users worldwide,4168 stations5 new users per month

EUMETCast AmericasSES 6, C-BAND

EUMETCast AfricaEUTELSAT 5 WEST A, C-BAND

EUMETCast EuropeEUTELSAT 9 A, KU-BAND

8Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETCast Dissemination:Integrating terrestrial dissemination

9Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

MULTICAST ACCROSS GÉANT and NREN INFRASTRUCTURE

GEANT/NRENs

EUMETCast Platform (MMDS)

EUMETCast User Stations

Reference Stations

Multicast ...Multicast

Multicast

Exploring the multicast usage on GÉANT infrastructure in Europe and their world wide connectivity;

2011 Proof of concept on terrestrial multicast on GÉANT test bed

2012 3 month feasibility study on terrestrial multicast via GÉANT

2013 Multicast test to KMA Korea

2014 Multicast test to NOAA (Boulder)

2014 Start of 2 year prototype emulating operational scenarios of multicast across Europe and our overseas partners using GÉANT, NREN, I2 and Teint infrastructure.

10Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETCast Terrestrial real-time monitoring

- Multiple Multicast channels- SSM mode- 20-30Mbit/s- FEC ¾- Closed loop monitoring based on successful file receipt- Packet/channel based ack/nack retransmission possible but not yet enabled

Acknowledgements: N-WAVE&NOAA Alex Hsia, David Hartzell

11Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETCast Dissemination:Low cost, off the shelf Receiving Station

DVB PCI Card

EKU

AntennaLaptop/ PC

SoftwareDVB USB

DVB Router

Typical total cost of a station < 2000 Euros

12Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

EUMETSAT Long Term Archive

Archived data starting 1981 All EUMETSAT Satellite data

including reprocessed data > 1 Petabytes in the archive

for MTP, MSG, Metop Retrieval/Ingestion: 10 to 1 Long-Term Data Preservation

in practice since 1995; Disaster resilient - 3rd Copy Access via EUMETSAT EO

Portal

13Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

What we are working on…

14Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Outlook – Enhance Future abilities in terms of

• Scalability• Flexibility,• Responsiveness to emerging needs,• Resource efficiency• While aiming for:

• Better sharing of resources,• Avoidance of data islands,• Reducing costs

• Fit the organisational structure

EUM/TSS/VWG/14/762641, 7 July 201415

Many aspects to consider => Challenge

Viewing Angle

16Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Outlook - Product delivery and management

• Generation of user tailored products• Efficient for write once, read many times• Suitable products for user needs and

dissemination infrastructure (bandwidth)• Avoid/ minimise processing where possible• Efficient for slicing

• Usage of Format evolutions• Secure the raw data (golden copy)• Initially smaller products which increase over

time• Reprocessing to re-build formats

17Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Outlook - Bringing data to Users

• Adaptive and flexible services to deliver data to users/ communities (e.g. EUMETCast)

• Community driven formats optimised for dissemination mean

• Serving large user groups from different communities

• Managing data policies in accordance with data provider requirements via a data policy and legal framework support by the related technical mechanisms

18Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Outlook - Bringing Users to data

Standardisation activities in relation to Discovery, Access and Retrieval (DAR) of data

Strong focus on interoperability and ease of user experience

One stop shop experience for discovering services and data

Facilitating the Interoperability of portals by exploiting standards (OGC, INSPIRE)

Enhance Long Term Archive, Online Data Access

Outreach activities Exploiting capabilities to execute user algorithms on data

Post-Processing, Hosted Processing

19Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Outlook - Information to Users

• Delivering Information to users rather than data

• Processing where the data is located

• Service Interfaces instead Facility interfaces

• Application of (private) cloud principles

20Improving Data Mobility & Management for International Climate Science, Michael.Schick@eumetsat.int & Lothar.Wolf@eumetsat.int

Thank you for your attention