ESA UNCLASSIFIED - For Official Use

December 2018

Data Flow Operations Concepts –Product Lifecycle Evolution

Slide 2

EO Product Lifecycle – Systematic Production

Downlink AcquisitionOnboard Sensing Processing

Archiving

Dissemination

Publication DeletionReprocessing

Archiving

Dissemination

Publication

A Simplistic Data Provider View of

Systematic Production

Auxilliary Data Collection

Slide 3

Downlink AcquisitionOnboard Sensing Processing

Archiving

Dissemination

Publication DeletionReprocessing

Archiving

Dissemination

Publication

Processing

Archiving

Dissemination

Publication DeletionReprocessing

Part of a complex

value adding chain

EO Product Lifecycle – Systematic Production

Slide 4

EO Product Lifecycle - Processing

Downlink AcquisitionOnboard Sensing Processing

Archiving

Dissemination

Publication DeletionReprocessing

Product Consolidation Data Quality Correction

Improvements in Sensor

Understanding

Improvements in Scientific

Understanding

Cost / Quality / Schedule

Level 0 Level 1 Level 2

Auxilliary Data Collection

Result of many trade off analysis and experiences Cost / Quality / Schedule

Slide 5

EO Product Lifecycle – Current Production Baselines

Catalogue Completeness

Well indentifiedAlgorithm and

Processor Versions

Data Format andArchive Integrity

Quality Control and Disclaimers

Fully documented Input/Output

Relying on the correct and reliable configuration of complete processing chain

Slide 6

EO Product Lifecycle – Current Implementation of Catalogue Completeness - APIs and listsProducts() -> uuid, md5, id/filename … metadataDeletedProducts() -> uuid, md5, id/filename … deletionCauseCatalogueView: .csv lists

Slide 7

EO Product Lifecycle – Current Implementation -Ingestion vs. Synchronisation in the DHR Network

BE FE

DHR

HUBX

Product UUID, checksum aremaintainedvia synchronisation… Integrity and provenance generically assured

No assurance of provenance

S1A_GRD_XYZ

S1A_GRD_XYZ

Slide 8

EO Product Lifecycle – Current Implementation –Already Suffering

BE FE

DHR

LTA Retrieval

Already suffering- .zip processing of archive data differs,

MD5 no longer uniquePossibilities are “workarounds”- Alternative MD5’s for one UUID - UUID generation based on core

components of the productAs many drawbacks as benefits

S1A_GRD_XYZ

S1A_GRD_XYZ

Slide 9

EO Product Lifecycle – Evolution to On Demand Processing

Downlink AcquisitionOnboard Sensing Processing

Archiving

Dissemination

Publication DeletionReprocessing

Archiving

Dissemination

Publication

On Demand

On Demand Processing – freeing the obligation to manage the full long term archive of all L1/L2

Auxilliary Data Collection

Slide 10

To sustain the current demand the distribution function will continue to need to manage rolling archive cache of latest “fresh” data

Full catalogue of processed data will remain available

～No long term archiving of all product levels, ….～As upgrade of processors are introduced they will be immediately

available for ad-hoc historic data requests, complete baselines no longer assured

？Bug fixes “versions” of data products will invalidate definitive list？No “guarantee” the same production for subsequent processing using

“same” input and processors (incl. aux data)

EO Product Lifecycle – Evolution to On Demand Processing - Impacts

Slide 11

Quite a few Provenance Starting Points

e.g. Provenance Management for Modern Distributed Systems. In Proceedings of USENIX Theory and Practice of Provenance, London, England, July 9–13, 2018

Centralised vs. Decentralised Approaches

Slide 12

Initial RequirementsBaseline RequirementsProvide per product proof of provenance for Sentinel dataflows

a record of all processing steps (ACQ, PRO, DISSEM, LTA ..)

Provide a secure, definitive reference for valid Sentinel data transformations

Attributable to contracted/trusted partnersAllow trusted partners to invalidate individual productsAllow trusted partners to invalidate entire product baselinesNo “lock-in” for trusted partnersAllow to document reason of invalidityAllow users to “certify” the validity of individual productsAllow users to “certify” the validity of lists of productsAllow users to identify product replacementsFully support the Sentinel on-demand production scenariosDistributed – Deployable in multiple facilities and cloud

infrastructures

Slide 13

Initial Requirements

Potential RequirementsApplicable to mission planningAllow to identify production gapsAuditable timelineInput for end-to-end dashboardExtendable to per product proof of provenance for Collaborative and Downstream individual products

Service Requirements

High availability, Robust to maintenance, network shortfall

Scalable - Sentinel-Scale, 100’s*10^6 products, many transformations per product

Including all components/operations necessary for management of digital signature

Capable to support data migration from initial sources (.csv etc.)

Open Source, API libraries / scriptable interfaces for integration to processor workflows

Slide 14

Way Forward

Product Lifecycle Operations Concept TN - 01/2019For comment and review - 02/2019

Hot topic for future prototype and interoperability demonstration