SESIP-0716-DP

Big Data and Cloud Computing

Expert Panel

Assessing Cloud-Native Architectures for EOSDIS

Applications

Dan Pilone - dan@element84.com

The material is based upon work supported by the National Aeronautics and Space

Administration under Contract Number NNG15HZ39C

https://ntrs.nasa.gov/search.jsp?R=20160009335 2020-07-31T01:59:30+00:00Z

SESIP-0716-DP

2

15+ years of Earth Science Data

Deletion of

MODIS

Collection 4

Products

Landsat-7 Data

Migrated from

LPDAAC to USGS

Addition of

AQUARIUS and

ICEBRIDGE

products

Addition of

ALOS PALSAR

& UAVSAR

products

ICESat MEASURES 2006 MEASURES 2012TERRA AQUA AURA

Addition of SNPP

CERES&OMPS

Total EOSDIS Accumulated Data Archive Volume

(Petabytes)

2

SESIP-0716-DP

3

EOSDIS Archive Growth Estimate

(Prime + Extended)

2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025

Cumulative Archive Size (PB) 13.8 20.0 27.0 34.8 42.7 65.0 118.0 170.5 223.1 275.6 328.2

Archive Growth Rate (PB) 4.9 6.2 7.0 7.9 7.9 22.4 52.9 52.6 52.6 52.6 52.6

0.0

50.0

100.0

150.0

200.0

250.0

300.0

350.0

400.0

PB

Archive Growth Rate (PB) Cumulative Archive Size (PB)

SESIP-0716-DP

EOSDIS Vision 2020 (cloud relevant)

- Data Analysis at Scale- Users are able to analyze the entire data record for any data variable

over any arbitrarily defined area.

- Data and Processing Mobility- Data and processing move transparently as necessary to achieve optimal

performance.

- Dataset Upgrading- High-value datasets are upgraded as necessary to fully support in the

rich capabilities available in the data systems.

- Virtual Collections...- ...can be organized/oriented around a science problem.

- Combining Data, Combining Tools- NASA data can be combined* with data from other agencies, nations and

other entities

- Tools and services within the community are easy to combine.

*combined = compared / merged / fused / assimilated

SESIP-0716-DP

6

So what does this impact?

SESIP-0716-DP

7

Infrastructure, Scaling, Monitoring,

Security, …

SESIP-0716-DP

8

Development model, Compliant

Deployment, Operations, …

SESIP-0716-DP

9

Application Architecture

SESIP-0716-DP

10

CMR in the Cloud

SESIP-0716-DP

11

GIBS in the Cloud?

SESIP-0716-DP

12

Systems Architecture

SESIP-0716-DP

13

Cumulus I&A Prototype

Data

Pipeline

Cloud Based Storage

Data

Validation

Reformattin

g

Ingest Archive Access

Metadata

Generation

CMR

Submission

MEaSUREs

Projects

HTTP

Distribution

OPeNDAP

Reprojection

Access

Projects

ICEBridge

Flight Path Val

FRB

Generation

Daily

Composites

SMAP

Unique…

DAAC 1

DAAC 2

Cloud based

Services

Cloud based

Services

Cloud based

ServicesYou?

SESIP-0716-DP

14

Exploiting cloud computing sets the

stage for EOSDIS as a platform,

enabling an AppStore model for

services and micro-services

meeting end-user needs.

SESIP-0716-DP

What does this mean?

• Need widespread knowledge of cloud

architectures and patterns (inc. NGAP,

serverless, 12 factor applications)

• Rapid pace of change

• Potentially dramatic change in data usage

patterns

• Need for Cloud Native data formats

• Need for Cloud Native services

SESIP-0716-DP

Outstanding Questions…

• Existing prototyping efforts have been restructured to systematically and realistically investigate and test commercial cloud environments for essential EOSDIS and DAAC operations.– Isolated prototypes have demonstrated unrestricted cloud

environments offer performance and cost benefits under specific situations, but can operational NASA science data systems function effectively in a commercial cloud environment?

– Do commercial clouds offer increased efficiency within a NASA environment and/or provide additional functionality? How do costs compare to opportunities and risks?

– How can NASA efficiently partner with commercial cloud providers to improve services and usage of data and services?

16

SESIP-0716-DP

17

Thank you!

Dan Pilone // dan@element84.com

SESIP-0716-DP

18

This material is based upon work

supported by the National

Aeronautics and Space

Administration under Contract

Number NNG15HZ39C.