The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data...
-
Upload
domenic-stanley -
Category
Documents
-
view
213 -
download
0
Transcript of The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data...
The Unified Access Framework (UAF)The Unified Access Framework (UAF)Building NOAA’s Global Earth Observation Building NOAA’s Global Earth Observation
Integrated Data Environment (GEO-IDE) Integrated Data Environment (GEO-IDE) one step at a timeone step at a time
Steve Hankin (PMEL), Kevin Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), Lewis O’Brien (PMEL/JISAO), Lewis McCulloch (HQ/TPIO) and the McCulloch (HQ/TPIO) and the
NOAA UAF teamNOAA UAF team
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
Dec. 2010Dec. 2010 22
Enterprise-wide integration of dataEnterprise-wide integration of datais a very hard problem! is a very hard problem!
(i.e. adopting shared approaches (i.e. adopting shared approachesto achieve data interoperability) to achieve data interoperability)
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 33NOAA/UAF-gridNOAA/UAF-grid
Why?Why?
Different fields have different Different fields have different concepts of ‘data’concepts of ‘data’
… … and have developed solutions that and have developed solutions that make sense make sense to them.to them.
Getting people (and organizations) Getting people (and organizations) to change habits is difficult!to change habits is difficult!
Dec. 2010Dec. 2010 44
NOAA-worldNOAA-world
NOAA/UAF-gridNOAA/UAF-grid
weather forecast weather forecast (time critical)(time critical)
fisheries management fisheries management (regulatory concerns)(regulatory concerns)
nautical chartingnautical chartingclimate, ocean, atmosphere researchclimate, ocean, atmosphere research
… … the list goes on … the list goes on …
NOAA management has been reluctant to invest NOAA management has been reluctant to invest funds in Agency-wide data integrationfunds in Agency-wide data integration
Dec. 2010Dec. 2010 55
The accepted approach:The accepted approach:build a “system of systems”build a “system of systems”
““Wrap” existing systems with loosely Wrap” existing systems with loosely coupled, standardized servicescoupled, standardized services
a Service Oriented Architecturea Service Oriented Architecture
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 66
Global Earth ObservationGlobal Earth Observation
Integrated Data EnvironmentIntegrated Data Environment
GEO-IDE Con-ops GEO-IDE Con-ops outlines such a planoutlines such a plan
2005-06, 66 pages2005-06, 66 pages(by “DMIT”)(by “DMIT”)
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 77
Seed fundingSeed funding ٨٨ available last yearavailable last year
Question:Question:How do you approach a very big How do you approach a very big problem with a very small amount of problem with a very small amount of money?money?
finallyfinally
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 88
1.1. Generate use casesGenerate use cases
2.2. Define requirementsDefine requirements
3.3. Write a Concept of OperationsWrite a Concept of Operations
4.4. … … and an Implementation Planand an Implementation Plan
5.5. Assemble volunteer teams to implement Assemble volunteer teams to implement
NOT !NOT !
NOAA/UAF-gridNOAA/UAF-grid
Tried and true approach …
Dec. 2010Dec. 2010 99
An alternative (‘agile’) approachAn alternative (‘agile’) approach
Don't Solve ProblemsDon't Solve Problems-- Copy Success -- Copy Success
""Switch: How to Change Things When Change Is HardSwitch: How to Change Things When Change Is Hard", ", Chip and Dan Heath (psychologists), 2010Chip and Dan Heath (psychologists), 2010
NOAA/UAF-gridNOAA/UAF-grid
What “success” did UAF chose to copy?What “success” did UAF chose to copy?
Year 1 focused on gridded datasets.Year 1 focused on gridded datasets.
Dec. 2010Dec. 2010 1010
Projects: (too many to name)Projects: (too many to name)
DataDataformats:formats:
netCDF GRIB HDF
ServiceServicestack:stack:
netCDF-CF-DAP-THREDDS-WMS
Applications:Applications: Matlab ArcGIS Ferret
GrADS Google Earth IDV LAS ERDDAP …
Users: (too many to name)Users: (too many to name)
…
Dec. 2010Dec. 2010 1111
UAFUAF(*)(*)-grid-grid(initial focus: gridded data)(initial focus: gridded data)
– netCDF netCDF – CF CF – OPeNDAP OPeNDAP – THREDDS THREDDS – WMS & WCSWMS & WCS
(*) Unified Access (*) Unified Access FrameworkFramework
---- file format/API/data file format/API/data modelmodel
---- metadata conventionsmetadata conventions
---- network transportnetwork transport
---- network topologynetwork topology
---- OGC rendering and OGC rendering and transporttransport
NOAA/UAF-gridNOAA/UAF-grid
THREDDS “network topology”THREDDS “network topology”a tree defined as distributed XMLa tree defined as distributed XML
Dec. 2010Dec. 2010 NOAA/UAF-gridNOAA/UAF-grid 1212
NO
MAD
S
GEO-IDE/UAF
NOAA NOAA Affiliated
NMFSOAR NWS NESDIS
NO
DC
NG
DC
GFD
L
PMEL
AOM
LO
CO
PFEG
ND
BC
ESRL
Coas
twat
ch
IOOS National Partners
IOOS Regional Partners
NAV
O
AOO
S
NAN
OO
S
CEN
COO
S SCCO
OS
PACI
OO
SG
LOS
NER
ACO
OS
MAC
OO
RA SECO
ORA
CARI
COO
S GCO
OS
Dec. 2010Dec. 2010 1313
Who is using this approach?Who is using this approach?
ModelersModelers– IPCC, GFDL, NCAR, … IPCC, GFDL, NCAR, …
Satellite programsSatellite programs– GHRSST, PathFinder, CoastWatch, …GHRSST, PathFinder, CoastWatch, …
NCEP weather and ocean forecastsNCEP weather and ocean forecasts– GRIB files served via NOMADSGRIB files served via NOMADS
Coastal (“HF”) radarCoastal (“HF”) radar
A growing list of observations programsA growing list of observations programs– Argo, OceanSites, …Argo, OceanSites, …
Pending adoption by OGCPending adoption by OGC
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 1414
Some rough spotsSome rough spots
compliance with CF conventions is inconsistentcompliance with CF conventions is inconsistent
individual files often not aggregated into logical individual files often not aggregated into logical datasetsdatasets
metadata are often in need of enhancementmetadata are often in need of enhancement
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 1515
… … but excellent repair tools exist:but excellent repair tools exist:NcML and “virtual files”NcML and “virtual files”
e.g. Improve CF compliance by adding “standard_name” e.g. Improve CF compliance by adding “standard_name” attribute to GRIB dataattribute to GRIB data
<variable name="vorticity"><variable name="vorticity"> <attribute <attribute name="standard_name" name="standard_name" value="atmosphere_absolute_vorticity“ value="atmosphere_absolute_vorticity“ /> /></variable></variable>
The file, itself, is untouched. But access to the file through the The file, itself, is untouched. But access to the file through the services is made to conform to standards.services is made to conform to standards.
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 1616
NcML for aggregationNcML for aggregation
e.g. Aggregate three 1-year files of the same (say) model run e.g. Aggregate three 1-year files of the same (say) model run
<aggregation type="joinExisting“<aggregation type="joinExisting“ dimName="TimeAxis"> dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/>
</aggregation></aggregation>
The files are untouched, but services see a single logical file.The files are untouched, but services see a single logical file.
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 1717
The emerging UAFThe emerging UAF
The UAF THREDDS catalog:The UAF THREDDS catalog:
~500 “clean” datasets~500 “clean” datasets.. growing as the team aggregates .. growing as the team aggregates and configures using ncMLand configures using ncML– team members become coachesteam members become coaches
lasting solutions must be implemented lasting solutions must be implemented at the data provider sitesat the data provider sites
The UAF TeamThe UAF Team– 8-10 active participants8-10 active participants– self-selected (volunteers)self-selected (volunteers)
1 year into the project …1 year into the project …
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 1818
How to reach users?How to reach users?
Through Through theirtheir preferred tools preferred tools (without downloading files)(without downloading files)
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 1919
Model 1: UMASS-ECOMModel 1: UMASS-ECOM Model 2: UMAINE-POMModel 2: UMAINE-POM Data: SST 2008-Sep-08 07:32Data: SST 2008-Sep-08 07:32
Desktop access in MatlabDesktop access in MatlabNOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 2020
Access in ArcGISAccess in ArcGISusing the Environmental Data using the Environmental Data
Connector (EDC)Connector (EDC)NOAA/UAF-gridNOAA/UAF-grid
Web browserWeb browser examples … examples …
Dec. 2010Dec. 2010 2424
Take differences
Google Earth
Scripts
Live Access Server Live Access Server (LAS)(LAS)
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 2626
WMS can be WMS can be used by many used by many
GIS applicationsGIS applications
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 2727
Browser access using ERDDAP Browser access using ERDDAP ……and from there …and from there …
Dec. 2010Dec. 2010 2828
.mat.mat
REST URL access to data subsetsREST URL access to data subsetsin several simple formatsin several simple formats
(accessible through home-grown scripting of many types)(accessible through home-grown scripting of many types)
.kml
.mat
.nc
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 2929
How can users find the datasets?How can users find the datasets?(less mature)(less mature)
1.1. Web crawlersWeb crawlers• extract metadata from files (and elsewhere)extract metadata from files (and elsewhere)• ingest into a data basesingest into a data bases
and from there:and from there:
2.2. formal metadata records generatedformal metadata records generated (ISO 19115) (ISO 19115)
3.3. data discovery portals harnesseddata discovery portals harnessed(e.g. RAMADDA, GeoNetwork, GI-Cat, …)(e.g. RAMADDA, GeoNetwork, GI-Cat, …)
4.4. metadata made available for harvesting metadata made available for harvesting – GCMD, Geospatial One-Stop, GEOSS, data.gov GCMD, Geospatial One-Stop, GEOSS, data.gov
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 NOAA/UAF-gridNOAA/UAF-grid 3030
UAF home page
Instructions for end users: how to access data through their favorite applications
UAF is experimenting UAF is experimenting to see how to see how documentation may documentation may be shared by other be shared by other projects…projects…
Dec. 2010Dec. 2010 NOAA/UAF-gridNOAA/UAF-grid 3131
• netCDF Java Tools• LAS• ERDDAP• GODIVA
From the THREDDS catalog “viewers” may also be called up
Dec. 2010Dec. 2010 NOAA/UAF-gridNOAA/UAF-grid 3232
File-internal metadata accessible directly from THREDDS catalogs
Dec. 2010Dec. 2010 3333
The coming year’s target:Integrating “1-d grids” (observations)
Collections of time series & profiles in netCDF-CF format are already on-line
NOAA/UAF-gridNOAA/UAF-grid
Dec. 2010Dec. 2010 3434
UAF team met just 3 weeks ago (Seattle)
NOAA/UAF-gridNOAA/UAF-grid
Preliminary plans:
Some initial target collections of in situ obs:• Argo, OceanSites, ICOADS• stream gauge data from USGS• atmospheric profilers (NCDC)• …
Common denominators for accessing the collections:• UAF year-1 techniques allow 1-at-a-time access• ERDDAP tables (ASCII and OPeNDAP Sequence)
• a gateway for IOOS/SOS, RDBMS, DAPPER, …
Year 1 metadata techniques largely applicable to obs
Still largely volunteer-powered