JL2-SESIP-0717

Data Are from Mars,

Tools Are from Venus

H. Joe Lee (hyoklee@hdfgroup.org)

The HDF Group

This work was supported by NASA/GSFC under

Raytheon Co. contract number NNG15HZ39C

All images used in this presentation are from

autodraw.com for public use.

JL2-SESIP-0717

2

No “Earth” in title?

“Men Are from Mars, Women are from Venus”

- John Gray

JL2-SESIP-0717

3

Are data from Mars?

Why can’t I use Earth tools?Correct geo-referencing

JL2-SESIP-0717

4

Are tools from Venus?

Why can’t I open Earth data?

JL2-SESIP-0717

5

Data Producers from Mars

• I want my data slim and efficient.

• How can I save money in managing data?

Tool Developers from Venus• I make my tool work for popular data first.

• Can I make money by supporting your data?

JL2-SESIP-0717

6

Result: Frustrated Users

JL2-SESIP-0717

7

We (HDFEOS.org) can help.

JL2-SESIP-0717

8

• Hierarchical Data Format (HDF)

• Network Common Data Format (netCDF)

• Geospatial Tagged Image File Format

(GeoTIFF)

• Keyhole Markup Language (KML) /

zipped KML (KMZ)

• Comma-separated values (CSV)

• etc.

We identify gaps in File Formats

JL2-SESIP-0717

9

• hdf

• netcdf-C

• netcdf-Java

• HDF – Earth Observing System (hdf-eos)

• Climate and Forecast Metadata (CF) conventions

• Geospatial Data Abstraction Library (GDAL)

• etc.

We identify gaps in Libraries

JL2-SESIP-0717

10

• Microsoft Excel

• Esri ArcGIS

• Google Earth

• MATLAB

• Python

• Interactive Data Language (IDL)

• Panoply

• Integrated Data Viewer (IDV)

• HDFView

• h5dump

• Etc.

We identify gaps in Tools

JL2-SESIP-0717

11

• Open-source Project for a Network Data

Access Protocol (OPeNDAP)

• Web Map Service (WMS)

• Web Map Tile Service (WMTS)

• Web Coverage Service (WCS)

• etc.

We identify gaps in Services

JL2-SESIP-0717

12

• File conversion

• Libraries and tools usage

• NASA HDF product specific examples

• Demo services (e.g., Hyrax*, THREDDS**)

AND we provide Solutions…

*Hyrax is the data server from OPeNDAP.

**Thematic Real-time Environmental Distributed Data Services

JL2-SESIP-0717

13

• Make HDF5 data work with

– GDAL

– netCDF

– Hyrax/THREDDS

• Don’t forget a few key CF conventions.

• Follow DIWG* recommendations.

Suggestions for data producers

*Data Interoperability Working Group

JL2-SESIP-0717

14

• Download and test NASA HDF products.

• Support them natively.

• Support augmentation.

– VRT* in GDAL

– NcML** in netCDF

• Support 3D visualization for data in the air.

Suggestions for tool developers

*Virtual Dataset in XML format

**netCDF Markup Language

JL2-SESIP-0717

15

3D Visualization

JL2-SESIP-0717

16

• Try OPeNDAP first. CSV may be enough.

• Try netCDF conversion / augmentation.

• Correct metadata with NcML / VRT.

• Try GEE* instead of GDAL.

• Use CMR** wisely.

Suggestions for end-users

*GDAL Enhancement for ESDIS project

** Core Metadata Repository

JL2-SESIP-0717

17

• Hadoop / Spark (streaming) / Dask

• Parquet / Arrow

• Elastic Search / Kibana

How about Big (fast) data?

JL2-SESIP-0717

18

Streaming Swath

Hadoop + Spark Streaming + Elasticsearch & Kibana

S3 + AWS Lambda + Elasticsearch & Kibana

JL2-SESIP-0717

19

• scikit-learn / keras / h2o.ai

• Please contact us at

eoshelp@hdfgroup.org if you’d like to see

examples on machine learning.

Future: (Deep) Machine Learning?

JL2-SESIP-0717

20

Deep Learning: Amazon

Rekognition

JL2-SESIP-0717

21

Alexa, what does MODIS see now?

Alexa: I can recognize “Alps.”

JL2-SESIP-0717

22

This work was supported by

NASA/GSFC under Raytheon Co.

contract number NNG15HZ39C