1 Improving Data Mobility & Management for International Climate Science,...

1Improving Data Mobility & Management for International Climate Science, [email protected] & [email protected]

Challenges in the provision of large data sets to multiple user communities

14-16 July 2014, Boulder CO

MONITORING WEATHER AND CLIMATE FROM SPACE


Mission Statement

We are the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT).

EUMETSAT is an intergovernmental organisation and was founded in 1986. Our purpose is to supply weather and climate-related satellite data, images and products

• 24 hours a day, • 365 days a year

to the National Meteorological Services of our Member and Cooperating States in Europe, and other users worldwide.

www.eumetsat.int


EUMETSAT Headquarters (Darmstadt, Germany)


EUMETSAT Data Services - High Level Overview

Users

Discover, search, register, order and subscribe

Subscribe to services

Discover/search products

Register to messages

Distributed search, remote order

Order/ subscription products

Subscribe to services

GEONETCast Catalogue

EUMETCast/ GEONETCast

Unified Meteorological

Archive(U-MARF)

EUMETSAT Partner Agencies/

GMES, WMO

User Notifcations(UNS)

EUMETCast / GEONETCast

Other Dissemination

Services

Registration

Discovery (Product Navigator)

Order Followup

Access to applications & data

EUMETSAT EO Portal


EUMETSAT Data Services - Data Flows

EUMETSAT GROUND INFRASTRUCTURE


Single Access Point

One stop shop

Data discovery

User registration (>10.000)

Subscription for data services: Near-real time data

Online Data Access

Long Term Archive

Additional Image Gallery (WMS/ WCS)

User Notifications

Operational Real-Time Services Status Indicator

EUMETSAT Earth Observation Portal


EUMETCast Dissemination Near Real Time data delivery to users in 3 continents via Satellite

Users (2/2014)

4

3

2-10

11-50

51-200

201-500

1 User

500+

0 User

3,280 users worldwide,4168 stations5 new users per month

EUMETCast AmericasSES 6, C-BAND

EUMETCast AfricaEUTELSAT 5 WEST A, C-BAND

EUMETCast EuropeEUTELSAT 9 A, KU-BAND


EUMETCast Dissemination:Integrating terrestrial dissemination


MULTICAST ACCROSS GÉANT and NREN INFRASTRUCTURE

GEANT/NRENs

EUMETCast Platform (MMDS)

EUMETCast User Stations

Reference Stations

Multicast ...Multicast

Multicast

Exploring the multicast usage on GÉANT infrastructure in Europe and their world wide connectivity;

2011 Proof of concept on terrestrial multicast on GÉANT test bed

2012 3 month feasibility study on terrestrial multicast via GÉANT

2013 Multicast test to KMA Korea

2014 Multicast test to NOAA (Boulder)

2014 Start of 2 year prototype emulating operational scenarios of multicast across Europe and our overseas partners using GÉANT, NREN, I2 and Teint infrastructure.


EUMETCast Terrestrial real-time monitoring

- Multiple Multicast channels- SSM mode- 20-30Mbit/s- FEC ¾- Closed loop monitoring based on successful file receipt- Packet/channel based ack/nack retransmission possible but not yet enabled

Acknowledgements: N-WAVE&NOAA Alex Hsia, David Hartzell


EUMETCast Dissemination:Low cost, off the shelf Receiving Station

DVB PCI Card

EKU

AntennaLaptop/ PC

SoftwareDVB USB

DVB Router

Typical total cost of a station < 2000 Euros


EUMETSAT Long Term Archive

Archived data starting 1981 All EUMETSAT Satellite data

including reprocessed data > 1 Petabytes in the archive

for MTP, MSG, Metop Retrieval/Ingestion: 10 to 1 Long-Term Data Preservation

in practice since 1995; Disaster resilient - 3rd Copy Access via EUMETSAT EO

Portal


What we are working on…


Outlook – Enhance Future abilities in terms of

• Scalability• Flexibility,• Responsiveness to emerging needs,• Resource efficiency• While aiming for:

• Better sharing of resources,• Avoidance of data islands,• Reducing costs

• Fit the organisational structure

EUM/TSS/VWG/14/762641, 7 July 201415

Many aspects to consider => Challenge

Viewing Angle


Outlook - Product delivery and management

• Generation of user tailored products• Efficient for write once, read many times• Suitable products for user needs and

dissemination infrastructure (bandwidth)• Avoid/ minimise processing where possible• Efficient for slicing

• Usage of Format evolutions• Secure the raw data (golden copy)• Initially smaller products which increase over

time• Reprocessing to re-build formats


Outlook - Bringing data to Users

• Adaptive and flexible services to deliver data to users/ communities (e.g. EUMETCast)

• Community driven formats optimised for dissemination mean

• Serving large user groups from different communities

• Managing data policies in accordance with data provider requirements via a data policy and legal framework support by the related technical mechanisms


Outlook - Bringing Users to data

Standardisation activities in relation to Discovery, Access and Retrieval (DAR) of data

Strong focus on interoperability and ease of user experience

One stop shop experience for discovering services and data

Facilitating the Interoperability of portals by exploiting standards (OGC, INSPIRE)

Enhance Long Term Archive, Online Data Access

Outreach activities Exploiting capabilities to execute user algorithms on data

Post-Processing, Hosted Processing


Outlook - Information to Users

• Delivering Information to users rather than data

• Processing where the data is located

• Service Interfaces instead Facility interfaces

• Application of (private) cloud principles


Thank you for your attention

1 Improving Data Mobility & Management for International Climate Science,...

Documents

Transcript of 1 Improving Data Mobility & Management for International Climate Science,...