Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014


Big Open Data: Transformation through Public Data Sets

Ariel [email protected]


Agenda• Open Data Overview (Ariel Gold, AWS)• NASA NEX (Dr. Tsengdar Lee, NASA)• Data Distribution Model (Ariel Gold, AWS)• Q&A


What is Open Data?

Data released to the public in ways that make it easy to

discover, access, and use.


Why are governments making their data open?

Five main goals seen across different policies and programs.


Reduced backlog in manual submissions while improving accuracy and reducing costs

Enabled ingestion of new open data sources to improve mission delivery

Shifted from quarterly reports to developer-focused APIs, raw data, and documentation


Release Definition & Policy Launch Open Data CatalogDo Things w/ Open Data

Release New (Big) Open Data

Repeat w/ Increased Focus

Impact

Gather Customer Feedback

Open Data Adoption Curve


How do you drive impact with (big) (open) data?

Build communities and tools around sustainable open data sets.


AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014

OpenNEX as an Open Data Analytical Service

Tsengdar Lee, [email protected]


NASA EARTH EXCHANGE (NEX)

Virtual collaborative-for-global-change scienceCOLLABORATION

Web & System

COMPUTINGNASA HECC

Pleiades 180k+ coresEndeavour 1.5k+ cores, 6TBD-wave Quantum 512 Qbit

DATA REPOSITORY800 TB, 126 PB

network links


NEX Provides a Complete Work Environment “Science as a Service”

COLLABORATION

Over 400 Members

CENTRALIZED DATA REPOSITORY

Over 1300 TB of Data

COMPUTINGScalable Diverse Secure/Reliable

KNOWLEDGE

WorkflowsMachine ImagesRe-useable software


• 33 CMIP5 models available

• All 4 RCPs (2.6, 4.5, 6.0, 8.5) and historical runs

• 30 arc-second (800m) spatial resolution, monthly time-step, 1950-2099

• Max/min temperature and precipitation

• Statistical downscaling (bias-corrected spatial disaggregation; Maurer et al., 2007)

Thrasher et al., 2013, Eos

Creating NEX-DCP30 Downscaled Climate Projections at 30 Arc-Second Resolution for the National Climate Assessment


• Sam Goward, PI

• Detecting 30m

resolution forest disturbance annually • National runs complete• Data

distribution being worked

North American Forest Dynamics (NAFD)

13


April 2010

October 2010

• David Roy, PI• Completed processing 140,000 scenes for first year of global Landsat (monthly and annual)• Public release later in June

Web-Enabled Landsat Data (WELD)

14


California Drought Monitoringat 250m Scale

Changes in High Latitude

Ecosystems - Greening


A tale of two droughts/Amazon 2005 & 2010

TRMM MODIS

Workflows Improving Research Productivity

Faster (24 months vs. 6 months), consistent (same analytical methods, quality flags) and reproducibleSamantha et al., GRL, 2011

Xu et al., GRL, 2011


NEX and AWS

Sandbox on Public Cloud- Prototyping - Development

NAS HECC- Large-Scale processing- NCA Example

COLLABORATION

(more public members)

COMPUTINGHECC + AWS1

DATA REPOSITORY(AWS OpenData, with

“Market driven” Cache)

AWS


2014 International Space Apps Challenge

46 countries participated 735 virtual participants 69 virtual team projects

Space Apps kicked off Thursday night in Doha, Qatar at midnight and ended in Seattle, WA at 6 p.m. Pacific—76 hours of around-the-world

hacking on NASA data.


2014 International Space Apps: Most Popular Challenges

Earth Watch: Where on Earth (44 projects), Climate & Neighborhood (22), Cool It (21)

Robotics: Exomars Rover is My Robot (42)Asteroids: Asteroid Prospector (38)Space Tech: Space Wearables (25), Alert-Alert (24)Space Flight: Growing Food for A Martian Table (22), SpaceT (20)


2014 International Space Apps:OpenNEX Data Challenges

Climate and My Neighborhood: 22 projects

Climate Alert: People’s Choice Local Award Winner

Climatehood: Best Use of Data Local Award Winner

Ways to Die: Most Inspiring Local Award Winner

NASA-NEX-Climate-Changes: People’s Choice Local Award Winner

21

OpenNEXWorkshops and Challenges

7/1 – 11/15/2014• Virtual Workshop teaches how to use OpenNEX

AMIs and Public Data sets on AWS.• Collaborate within arms length

- Users work with vm templates including prepackaged tools, they instantiate, extend and own accordingly.

- Core data is read only public data sets on S3• Bring citizen scientists together around data,

services and knowledge• Extract more value out of assets, i.e. data and

knowledge, via the group • http://nex.nasa.gov/OpenNEX



Thank You


Thank YouTsengdar Lee, Ph.D.

[email protected]


How do you drive impact with (big) (open) data?

Build communities and tools around sustainable open data sets.


National Agriculture Imagery Program


Before• Pay $5k • Wait weeks to get data on disks • Technical and GIS expertise to

make data usable

After• No or minimal cost• Get access immediately • No technical or GIS expertise to

start using

Significantly increase openness of a 48TB aerial imagery data set at a low and sustainable cost ($1,600/month)

National Agriculture Imagery Program


Managing the Cost of Data Distribution

• Amazon Simple Storage Service (S3) “requestor pays”

• Eliminate variable costs• Multiple configuration options

depending on your use case


Insert NAIP Demo Slides (or link to live pages)


Q & AFor more information on AWS and open data, visit:

http://aws.amazon.com/government-education/open-data/






Thank YouAriel Gold

[email protected]

Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014

Technology

Transcript of Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014