CEOS Data Cube Open Source Software Status

Brian KilloughCEOS Systems Engineering Office (SEO)

WGISS-40Harwell, Oxfordshire, UKSeptember 30, 2015(remote presentation)

General CEOS Data Cube Architecture

A flexible architecture that supports infinite user applications, increasing and diverse datasets, local or cloud-based deployment, and automated ingestion of new datasets.

Open Source software to allow free and open access, Advanced Programming Interface (API) access, future data and capability growth, and commercial opportunities.

Applications • Web-based• Mobile

Data and Application

Platform

API Toolset

• Access Control• Data Management• Job Management

• Data Cube Access• Analytics• Visualization

API

API

Data Cube

Analysis-Ready Data

Products

SatelliteSpace Agency

High-Level Data Cube Requirements1. Free and open access to software and APIs

2. Documented processes for data cube generation, new data ingestion, and application interfaces

3. Cloud-based or local deployment4. “Analysis Ready” data products for satellite data layers

(fully processed – calibration, geolocation, radiometric)5. Preservation of native data product grid formats6. Computational flexibility to allow reprojection into “nested grid”

formats for multiple dataset interoperability and spatial consistency7. Reference user interface that supports data cube statistics/analysis

and optical image preparation (e.g., mosaics).8. Enable user development of applications through flexible APIs9. Architecture flexibility and standards to support Data Cubes for local,

regional or country scales

Kenya Project

The Kenya Data Cube Project is led by NASA-SEO and the Australian Government. Supported by USGS and the Clinton Foundation (CCI and SLEEK).

Two operating versions of the Kenya Data Cube ... One is deployed on the Amazon Cloud and one is deployed on a local computer.

11.5 TB of uncompressed Landsat data (7500+ Landsat 7+8 scenes). Pixel information was extracted, reformatted into a UTM grid, and compressed into a 3.6 TB Data Cube. Application algorithms can be run directly on the compressed Data Cube.

Demonstrated an operating sample user interface for mosaic creation at the recent SDCG meeting. Planning to add more data layers in the coming months ... SRTM, ALOS PALSAR Mosaics, MODIS.

Colombia Project

The Colombia Data Cube Project is led by NASA-SEO and the Colombian Government. Supported by CSIRO (Australia, CEOS Chair Team) and IDEAM (Institute of Hydrology, Meteorology and Environmental Studies).

A mini-cube (4 Landsat path-row regions) will be delivered on October 14-15. Includes all historic Landsat data back to 2000. This meeting will define the objectives and detailed project development plan for 2016.

Colombian researchers are eager to expand the Data Cube to the full country and to develop enhanced user interfaces to support forest and agriculture applications. Additional datasets will also be added.

Kenya Data Cube User Interface Demo

Map Panel

Tool Panel

• Selectable data products (L7, L8)

• Selectable time periods (years, months)

• Selectable region (user defined, KML, Shapefile)

• Selectable product (mosaic)

• Selectable output image (true color, false color, NDVI)

Open Source Software

The NASA-SEO, CSIRO Australia and Geoscience Australia are managing the CEOS Data Cube Open Source Software.

Website: http://www.datacube.org.auIncludes background on Australia Geoscience Data Cube, Wiki and GitHub (open source repository)

Software is released using the Apache 2.0 license. This is a very permissive license that provides an express grant of patent rights from contributors to users and allows open distribution and commercial use.

NASA internal approval for open source release took 2 months. This includes approvals from softwware engineering, legal and export control.

NASA will release its first version of code within a week: Landsat “ingestor”, Kenya Data Cube and a reference user interface (mosaic creation).

Future Plans

• Work with CEOS Space Agencies to develop plans for sustained provision of Analysis Ready Data (ARD)

• Sentinel-1/2 and SPOT-5 are priorities.

Reference User Interface

Data Cubes

Analysis-Ready Data

Products

• Develop prototype reference user interfaces for the Kenya and Colombia projects.

• Consider features such as: mosaic creation, data cube analytics and visualization, forest cover classification and change detection, and agriculture NDVI and change detection

• Develop an API for users to create their own user interfaces

• Develop prototype Data Cubes for Kenya and Colombia projects. Test local and cloud-based deployment options.

• Expand data layers to include: Landsat, SRTM, ALOS, MODIS, SPOT-5, Sentinel-1/2.

• Improve data cube ingestor software (creation) and data cube application software (analysis)