Brian Killough / NASA, CEOS SEOSDCG-8 Session 7, Agenda Items 26 and 27

GFOI Space Data Services

SDCG-8DLR, Bonn, Germany

September 23rd-25th 2015

Agenda

• 3-Year Work Plan Tasks and Outcomeso #6: Ensured On-going Coverage (+ Archive Characterization)o #7: Interoperable Data Discovery Toolso #8: Assembly and Delivery of Core Datao #10: Cloud-Computing Pilot Projectso #11: Model National GFOI Data Services System

• CEOS Data Cube Summary Report - Reported by Matt Jondrow

#6: Ensured On-going Coverage (+ Archive Characterization)

• The SEO has completed 22 country reports to characterize Landsat coverage. These were delivered to countries at SDCG and SilvaCarbon meetings. 2 more reports are being delivered at SDCG-8 (Peru and Vietnam).

• Feedback from countries has been limited, but those that have responded, were positive. The SEO has provided additional detailed support to countries (Kenya and Colombia) to identify low-cloud scenes and order surface reflectance products.

• Future ... The SEO can develop systems analysis tools, user guides and tutorials to allow countries to do their own analyses. There is also a recent discussion about creating a “GFOI Data Services Portal”. In addition, the SEO can continue to offer custom services.

Country Report Summary

Two detailed “country reports” were completed for SDCG-8 (Peru and Vietnam).“Country Reports” contain a summary of Landsat acquisitions over the country since 2000 (see example tables/charts on the next page).The reports include details about cloud cover and data processing levels to allow countries to assess the acceptability of available scenes for forest mapping and reporting.Analyses were completed using the “COVE Coverage Analyzer” tool and a custom queries of the Landsat archive system for data processing status.

4

Example Tables/Charts

SDCG-7Sydney, Australia

March 4th – 6th 2015

5

TOP TableSummary of Landsat 5/7/8 acquisitions per year (2000 through 2014).

Middle TableDetailed Path-Row summary of low-cloud (<10%) scenes and medium-cloud (<60%) scenes in 2014 and 2015 for Landsat-7 and Landsat-8.

Bottom ChartLandsat acquisitions with <20% cloud cover summarized by year since 2000.

Year L5 L7 L8 Total L5 L7 L8 Total 2000 1336 1362 0 2698 275 241 0 5162001 1339 1492 0 2831 250 244 0 4942002 244 1563 0 1807 26 226 0 2522003 733 1146 0 1879 205 196 0 4012004 1509 1324 0 2833 299 266 0 5652005 1332 1423 0 2755 329 357 0 6862006 1358 1266 0 2624 294 294 0 5882007 1138 1138 0 2276 250 251 0 5012008 1342 1235 0 2577 303 286 0 5892009 1334 1246 0 2580 264 266 0 5302010 784 1353 0 2137 239 259 0 4982011 856 1308 0 2164 206 243 0 4492012 0 1626 0 1626 0 347 0 3472013 0 1513 1170 2683 0 287 281 5682014 0 1782 1677 3459 0 294 330 6242015 0 1125 1204 2329 0 152 215 367Total 13305 21902 4051 39258 2940 4209 826 7975

<20% Cloud CoverAll Scenes

Coverage Analyzer Example

SDCG-7Sydney, Australia

March 4th – 6th 2015

6

Peru ExampleThe COVE Coverage Analyzer can easily

produce the figure on the left.These figures are valuable for

identifying path-row regions with cloud issues.

Due to extreme cloudiness, there are several regions in the north-west of Peru (Ecaudor border)that did not acquire any images with <20% cloud cover in 2014.

2014 summary of total Landsat-7 and Landsat-8 scenes with <20% cloud cover. The maximum for any path-row region is 23*2 = 46 scenes. The south border had 29 scenes in one location.

#7: Interoperable Data Discovery Tools

• The COVE Tool (Coverage Analyzer) now includes Landsat 7/8, SPOT 1-6, Pleaides-1A/1B, Radarsat-2, ALOS-1.

• There have been few requests for searches of Envisat, ERS, so adding those archive connections is low priority.

• Adding an archive link for Sentinel-1A and Sentinel-2A is highly desired, but a mechanism does not exist to get this data from ESA/EC.

• The SEO vision is that the COVE tool can provide a “one-stop” location to perform coverage assessments and “discovery” of valid images. Once discovered, users can find links to the data in COVE, or be directed to other sites (CEOS OpenSearch from WGISS) to find the data.

• Data discovery is also part of the (Space Data Management System (SDMS) scene-based tools.

#8: Assembly/Delivery of Core Data

• The new Global Data Flow Study is focused on this topic. (see additional chart).• There are 3 methods of data delivery (business-as-usual data over the internet or

on drives, delivery using cloud-computing, or delivery in-country using Data Cubes (see additional chart).

• After discussion with FAO at SDCG-7, the SEO took an action to develop a “GFOI Space Data Guide” that provides details and contacts for data access, details and contacts for coverage analyses (SEO), and links to resources (COVE, tutorials).

The scale of the data problem • Kenya: 4 TB of Landsat 7/8 data since 2000. Adding 4.5 TB per year for Landsat and

Sentinel-2A. Storage needs are ~22 TB for 3 years. Cost of cloud-based storage (Amazon EBS) and processing (Amazon EC2) is ~$29,000 U.S. per year.• Colombia: 11 TB of Landsat 7/8 data since 2000. Adding 8 TB per year for Landsat and

Sentinel-2A. Storage needs are ~42 TB for 3 years. Cost of cloud-based storage and processing is ~$53,000 U.S. per year.• 70 GFOI Countries: Consider all Landsat 7/8 and Sentinel-2A data = 200 TB per year, 1.3 PB

of new data by 2020. This does not count historic data needed for baseline analyses.• 50% of the countries have a required annual data volume of 0.6 to 3.4 TB (Landsat and

Sentinel, combined). The mean volume is 2.8 TB and the median volume is 1.2 TB.

We have a BIG data problem with non-sustainable costs. The global data flow study will explore these problems and potential solutions.

Global Data Flow Scenarios

Satellite Level-1Data

Space Agency In-Country Users

Satellite Analysis Ready Data (ARD) Products

• Hard drive delivery or internet download

• Countries perform data analysis on ARD locally

• Countries use scene-based tools or a Data Cube (countries create their own Data Cubes using CEOS open source tools or utilize CEOS support toinitialize a baseline Data Cube)

Space Agency Processinginto Level-2 SR products

• Hard Drive Deliveryor Internet Download

• Countries perform processing and data analysis locally

Intermediate Data Storage and Processing(Cloud-based or Data Hubs)

• Scene-based tools• Hosted Data Cube and tools

• Country analyses of ARD performed on Cloud or Data Hub• Countries download analysis

products via Internet

Business-as-Usual

BAU

Scenario-2

Scenario-1

#10: Cloud Computing Pilot Projects

• KSAT/AMA delivered a scene-based SDMS to FAO on November 5, 2014. FAO has developed a similar tool called “SEPAL” and is using it in-country for forest management. The SEO has advanced its scene-based SDMS and will make it available for testing in Kenya and other GFOI countries.

• The SEO delivered an SDMS to Colombia (TanDEM-X DEM) on December 3, 2014. This data is being used with SRTM for forest cover research.

• The SEO has developed a full-country Kenya Data Cube with Landsat 7/8 data since 2000. This development includes the use of analysis-ready surface reflectance products from USGS, modified ingestion software from GA/CSIRO, and a new reference user interface from AMA. All of the software and tools will be available on an open source site.

• The SEO is starting a Colombia Data Cube project. By late 2015, a mini-cube will be completed for 4 path-row regions. By the end of 2016, a full-country cube will be completed with multiple data layers and a user interface that supports multiple applications (GFOI and GEOGLAM).

12

SEO SDMS (scene-based)

High-Level Data Cube Requirements

• Free and open access to software and APIs• Documented processes for data cube generation,

new data ingestion, and application user interfaces• Cloud-based or local deployment• Use of “Analysis Ready” data products• Preservation of native data product grid formats• Computational flexibility to allow reprojection into “nested grid” formats

for multiple dataset interoperability and spatial consistency• Baseline user interface that supports data cube statistics/analysis and

optical image preparation (e.g. mosaics).• Enable user development of applications through flexible APIs• Architecture flexibility and standards to support Data Cubes for local,

regional or country scales

Data Cube Datasets (priority order of addition)• Landsat-5/7/8 (Surface Reflectance) from USGS. Products available, by order.• SRTM ... NASA Space Shuttle Digital Elevation dataset (Feb 2000) now available globally at

1-arcsec (30 meters) resolution. • ALOS (PALSAR L-Band SAR). Free and open annual mosaics from 2007-2010 provided by

JAXA. Working with Ake Rosenqvist (GFOI) to demo. • MODIS (Terra, Surface Reflectance, 8-day Level-3 500-m Global, MOD09A1).

Working with Alyssa Whitcraft (GEOGLAM) to demo. Also will work with GA/CSIRO to utilize their experience with this dataset.• SPOT-5 (10-meter multi-spectral). Plans to obtain SPOT-5 Take-5 data over Kenya sample

site (processing complete in Sept 2015). Over time, CNES plans to process/release all data.• Sentinel-1A: C-band SAR data, plans to add sample images to prototype in late 2015• Sentinel-2A: Launched June 21, data available in late 2015• RapidEye: SDCG-8 action to obtain sample data for Data Cube ingestion testing.• In-Situ Data: Working with Kenya SLEEK team to investigate approaches for adding in-situ

climate data. May also consider global rain gauges and other ground-based data for demo.• Radarsat-2 (C-Band SAR): Dataset is restricted, so we are not planning to add this data to

any of the prototypes. We will consider adding accommodations to the ingester to support this dataset, so others can use the data.

#11: Model National GFOI Data Services

• It is expected that the prototypes in Kenya and Colombia would be ideal testing locations toward a model national GFOI data services system.

• FAO will be using SEPAL in many countries, so feedback from its use would help support requirements for a model national system.

QUESTIONS• What is the process for integration with the MGD?• Is it possible to have a common country engagement strategy among GFOI and

FAO? For example, where do we use SEPAL, SDMS or Data Cubes?• How can we tailor our data services for the future to better meet the needs of

GFOI?