060314 Ispra Htap Presentations Husar 060314 Ispra
-
date post
13-Sep-2014 -
Category
Technology
-
view
11 -
download
0
description
Transcript of 060314 Ispra Htap Presentations Husar 060314 Ispra
Work Group Meeting onHTAP-Relevant IT Techniques, Tools and Philosophies:
DataFed Experience and Perspectives
Rudolf B. HusarCAPITA, Washington University, St. Louis, MO
Ispra, JRC, March 14. 2004
• Data handling approach• Software tools• Participant involvement• Data dissemination• Integration problems• Summary
Policy Guidance for HTAP IT
Terry Keating, HTAP Task Force Co-Chair:
• Transparency of the HTAP Process– Acceptance of the Tech Report will depend on openness
• Inclusiveness and Ease of Participation– Facilitate participation by smaller contributors
Current Air Quality Information ‘Ecosystem’
AQ info is distributed over many ‘dimensions’: Geography, Content, Agency, Program…AQ info content includes: emissions, ambient & satellite data and model outputsInfo is provided and consumed by different agencies, (NASA, NOAA, EPA…)Providers have different access protocols, formats, and information usage conditions
Lack of Interoperability
Poor data & model utilization
Less informed decision making
GEOSS Info Flow ArchitectureA General Framework Accepted by Members
Model
Model
Data
Data
Federated Network for Air Quality Data and Processing ServicesProject Team:
Software Architecture: R. HusarSoftware Implementation: K. Höijärvi
Data and Applications: S. Falke, R. Husar
Data Handling Approach
• The challenge is to design a general supportive infrastructure• Simply connecting the relevant provides and users for each info product is messy
Federated Data System for Air Quality
• The info system infrastructure needs to facilitate the creation of info products
AQ Compliance
Nowcast/Forecast
Status & Trends
Find Data Gaps
ID New Problems
………
Info Needs
Reports
• Providers supply the ‘raw material’ (data and models) for ‘refined’ info products
EmissionSurface Satellite
Model
Single Datasets
Providers
Wrappers
Where?
What?
When?
Federate Data
Structuring
• Structuring the heterogeneous data into where-when-what ‘cubes’ simplifies the mess
Slice & Dice
Explore Data
Viewers
• The ‘cubed’ data can be accessed and explored by slicing-dicing tools
Programs
Integrate
Understand
• More elaborate data integration and fusion can be done by web service chaining
Non-intrusive Linking & Mediation Data UsersData Providers
Data Handling Approach:
DataFed
Approach: Mediation Between Users and Data ProvidersDataFed assumes autonomous data management (a la Internet)Non-intrusive third-party data wrapping for unified web service (WS) accessEnd-user programming of applications through chaining of WS components
ApplicationsBuilding browsers and analysis tools for distributed monitoring data Serve as data and service resources for user programs (science, GIS tools)Support application projects, e.g. FASTNET, Exceptional Event Rule
Typical DataFed AQ Analysis Tools
Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis
CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions
Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets
Quebec SmokeJuly, 7, 2002
SeaWiFS Satellite
Aerosol Chemical
Air Trajectory
Map Boarder
Web Service Composition
Software Tools:
Demo: Networked Data Access, Processing and Fusion
AeroCOM – VIEWS SO4
• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_VIEWS• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_V• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=VIEWS_KYU
Networking
VisualizationGoal: 4D, User-selectable layers
Image below completed in 1998
Satellite Data Layers:
Land Reflectance (SeaWiFS)
Fire Pixels (ASTR Night)
High Cloud (GOES, Meteosat, GMS2)
Aerosol AOT (AVHRR)
4D Dynamic Visualization Demonstrate interactions, allow exploration
‘Google Earth’ for Earth Science is now possible
Partners
• NASA
• NOAA
• EPA
• USGS
• DOE
• NSF
• Industry…
Earth Science Information Partners
Air Quality Cluster
1. Serve as facilitator to Earth Science information community. 2. Promote efficient flow of ES data from collection to end-use. 3. Improve quality and usability of ES data and info systems 4. Expand the use of Earth science information
Participant Involvement
Data Dissemination & Use – Service Based
• Provide Catalog Services (Publish, Find data)• Allow Data Access (‘Bind’) – Use International Standards • Add tools for exploration and analysis
Near Real Time Data IntegrationDelayed Data Integration
Surface Air Quality AIRNOW O3, PM25 ASOS_STI Visibility, 300 sitesMETAR Visibility, 1200 sitesVIEWS_OL 40+ Aerosol Parameters
SatelliteMODIS_AOT AOT, Idea ProjectGASP Reflectance, AOTTOMS Absorption Indx, Refl.SEAW_US Reflectance, AOT
Model OutputNAAPS Dust, Smoke, Sulfate, AOTWRF Sulfate
Fire DataHMS_Fire Fire PixelsMODIS_Fire Fire Pixels
Surface MeteorologyRADAR NEXTRADSURF_MET Temp, Dewp, Humidity…SURF_WIND Wind vectorsATAD Trajectory, VIEWS locs.
WCS - Interoperable Data Access Service
netCDF – Machine independent encoding
ncML – XML data description
CF – Naming, structure convention
OGC Web Coverage Service - Interoperable Data Access
Query Data Syntax + Semantics
Coverage (parameter)
BBOX
Time Range
netCDF Example Profile
• SERVICE=wcs ‘service• REQUEST=GetCoverage,VER=1.0 ‘service method• COVERAGE=AIRNOW.pmfine ‘what• CRS=EPSG:4326 ‘projection• BBOX=-125,22,-61,51,0,0 ‘where• TIME=2005-06-6T15:00:00Z ‘when• FORMAT=netCDF ‘return format
GALEON Interoperability ExperimentGeo-interface for Atmosphere, Land, Earth, and Ocean netCDF
Unify Earth Science & GIS Data Flows
Strong European Participation
IT – S. Nativi, L. Bigagli
UK – Andrew Wolf
DE – Peter Bauman)
B. Domenico
B. Domenico
GALEON UNIDATA
U Florence/CNR-IMMA WCS Server
OGC WCS Demonstration: THREDDS_GFS 4Dim Dataset
Lat/Lon Box Elev Range Time RangeMap: BBOX=-180,-90,180,90, 1350,1350& TIME=2005-12-06/2005-12-06/PT3HTime: BBOX=-34,49.05,-34,49.05, 1350,1350& TIME=2005-12-05/2005-12-08/PT3HElev: BBOX=-34,49.05,-34,49.05, 0,18000 & TIME=2005-12-06/2005-12-06/PT3H
The form of the WCS query is the same for all slices through the data cube (views) The only difference in the views is the thickness of the slices in each dimension Return grid is in multiple formats (NetCDF, CSV, GML, PNG, … )
Map View Services WCS Query
Time View Services WCS Query
Elevation View Services WCS Query
Summary
• Current systems data & model analysis are heterogeneous
• Standardization is a key need for agile IT systems
• Non-intrusive mediators can achieve virtual standardization
• Technologies are currently available for dynamic NETWORKING
We eager to share our networked data, tools and methods
OGC WCS Demonstration: Grid, Image, Station Data Types
Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDFCoverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-tableCoverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFFCOVERAGE=sst& BBOX=-126,24,-65,52,0,0 &TIME=2001-01-01,2001-01-01&FORMAT=NetCDF
UNIDATA – THREDDS/GALEON WCS
DataFed GALEON WCS
U Florence, It GALEON WCS
DataFed GALEON WCS
Grid
Grid
Image
Station
Services WCS Query
Services WCS Query
Services
WCS Query
Services
WCS Query
Single Data Model for All AQ Data
Most Views are slices through a cube of data organized by lat, lon, altitude, and time (X,Y,Z,T)
Multidimensional Data Cube
OGC Web Coverage Service (WCS) Specification
• HTTP GET/POST based interfaces• Services have XML service descriptions (“Capabilities”, “Description”)• Filter parameters allow selection of subsets of source data• Output formats advertised by each service instance
OGC WCS getCoverage SchemaSuitable for wrapping with SOAP envelope, WSDL access, loose coupling
WCS is for "coverages" – information representing space-time-varying phenomena
WCS describes, requests and delivers coverages in spatio-temporal domain
WCS version 1.1 is limited to grids/"simple” coverages with homogeneous range sets
AQ Monitoring Network Data Storage and Delivery through OGC Protocols
Relational Data Model
Star Schema
WMS
WCS SOS
Sen
sorMLW
FS
Observations
Sta
tion
In
fo.
Para
m/S
en
sor/M
eth
od
Data View ServicesW
MS
Stations Par-Meth
Observations
SO
S
WCS Data Access
Technology Support for ‘Integrated Solutions’
Air Quality Information System
through
Data Access through Adapters
DataFed SOAP,HTTP Get
OGC WCS HTTP Get, Post
OGC WMS HTTP Get
Station-Point SQL Server,
Files…
Sequence Image, file
nDim Grid OpenDAP NetCDF,
…
Other Traject., Event, Pic
Sources Diverse formats
Many data models
Data Wrapper Data into geo-cubes
Queries to views
Virtual Data Cube Global geo-cube data model Makes queries data-neutral
Others? e.g.
OpenDAP
Output Protocol dependent
User specified
GeoTable CSV,XLS,GML
GeoGrid GML,NetCDF..
GeoImage GeoTIFF,
PNG..
Other MS Dataset..
Query Adapter Maps query to protocol
User selects protocols