Outline
Recent near real-time in-situ observations are aggregated to a Recent near real-time in-situ observations are aggregated to a ‘Xenia’ schema relational database(RDB) via its import XML ‘Xenia’ schema relational database(RDB) via its import XML format ‘ObsKML’ schema. From this database of recent format ‘ObsKML’ schema. From this database of recent observations a variety of file formats, web services and observations a variety of file formats, web services and applications can be driven. By suggesting a minimally applications can be driven. By suggesting a minimally common observation oriented XML and RDB schema, common observation oriented XML and RDB schema, developed scripts and products can benefit from and build developed scripts and products can benefit from and build around these shared schema. For more info contact around these shared schema. For more info contact [email protected]@gmail.com
KeywordsKeywordsScripting languages Scripting languages Perl PHPPerl PHPRelational Database Relational Database Sqlite PostgresSqlite PostgresMapping Engine/Interface Mapping Engine/Interface Mapserver OpenLayers TileCacheMapserver OpenLayers TileCacheOpenGeospatial Consortium(OGC) services OpenGeospatial Consortium(OGC) services WMS/WFS/SOSWMS/WFS/SOSGoogle Earth/Maps Google Earth/Maps KML/KMZKML/KMZGoogle Code/AnalyticsGoogle Code/Analytics
Problems Xenia intended to address
• Grants for research instrumentation which will be collecting observation data while lacking a data management/sharing component beyond basic file storage
• Low-volume data(< 100,000 records per hour) in-situ observational platforms or system arrays (e.g. 1 to 1000 platforms collecting 10-20 observations per hour) collecting data at any geographic scale (local,regional,national,etc)
• Bridging the gap between raw data collection and the organization and sharing of data using previously developed products, services and standards(leveraging earlier work against new data providers)
• Fostering a standardization of products and services via a common openly shared technical infrastructure(common database schema and product support scripts)
Problems Xenia not intended to address
High-volume data (millions of records per hour) such as gridded model outputs, hf radar, etc. High-volume data problems at this time are better addressed using traditional file processing techniques where data management can suggest output file formats(such as images, shapefiles, etc) and metadata that are conducive to search and usage needs.
Xenia Relational Database Schema
Hierarchy = organization->platform->sensor->multi_obs(observation data)
Data flows – ObsKML
ObsKML (Observations KML)
A simple XML encoding of observation metadataassociated with a KMLPlacemark.
Default XML import/exportformat for Xenia databaseInstances.
Postgres or Sqlite database paths
Xenia data organized by time interval
Latestpast several hours
New Data
Recent0-6 weeks
Archival3+ weeks to 1-2 years
Possibly table separated by year,month,etc
Archival file1-2+ years
Files separated by product/year/month
Julian weekly divided database files(sqlite) of all observations are available http://tinyurl.com/6ctz66
Xenia aggregation, replication, redundancy
XeniaA,B,C,D,E,F
XeniaA,B,C
XeniaD,E,F
XeniaA
XeniaB
XeniaC
XeniaD
XeniaE
XeniaF
Xenia BackupA,B,C,D,E,F
Xenia BackupD,E,F
Overlapping/redundant systems could aggregate/replicate via SQL or XML
Flow/event monitoring
Around 10,000 observations are collected each hour from a variety of data providers. The data flows are monitored for both successful import and towards provided report metrics or notifications when problems or events occur.
Sensor count graph showing the hourly number of successful sensor imports per provider
More detailed platform/observation table showing daily counts color coded to highlight specific sensor issues
Applications – National Weather Service
Latest regional platform observations from Xenia database are styled to html tables and presented in fixed map interface alongside NWS forecast warnings/advisories and other map layers of interest
http://forecast.weather.gov/mwp/
Applications – SECOORA interactive map
Latest regional observation maps are accessed via WMS(Web Mapping Service) and merged into an OpenLayers(browser javascript) map interface. Observation database requests are supported via WMS GFI(Get Feature Info)
http://secoora.org/maps/dynamic
Applications - CarolinasRCOOS
Map image layers supported via MapServer WMS with increased map response time via TileCache which tiles and caches image requests for repeated reference. Site feature info supported via Xenia database instance styled output html table accessed via WMS GFI.
http://carolinasrcoos.org
KML/KMZ – Latest obs by platform
KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer
Latest observation data organized by all observations per platform
http://tinyurl.com/664wtx
KML/KMZ – latest obs by obsType
KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer
Latest observation data organized by observation type for all platforms
http://tinyurl.com/664wtx
All observations carry
* observation type(obsType)
* unit of measure(uom)
* measurement value
obsType+uom = measurement type(m_type)
Color styled low to high value blue/green/red
Development/documentation support
Google Code provides very simple, free browser based access to the basic project support tools such as a subversion(svn) code/documentation sharing/versioning, wiki documentation and issue tickets.
Google Analytics provides free browser based access to website usage statistics via a javascript project token included in webpage access.
Lessons Learned
• Web GIS interface: php/mapscript -> openlayers/javascript/ajax, tilecache • Latest in-situ obs: RDB – sensor/obs hard-coded/table approach -> obs non-specific infrastructure(xenia,obskml) • Sensor/observation specific metadata: loosely link a variety of more specific metadata files/schemas to common infrastructure• Hourly gridded data: RDB quickscat, model layers -> ogr/sql+shapefiles • Full RDB to file-based RDB: postgresql+postgis -> sqlite+ogr, sqlitegeo? • Self hosted to third party hosted: third party website hosting for static content/context, common processing/storage. Google code/analytics for basic shared development
support/documentation.• ROA(Resource Oriented Architecture)/REST style simpler declarative alternatives to complex query SOA(Service Oriented Architecture)
File formats and web services• File formats (CSV,shapefile,netCDF,KML/KMZ,Sqlite DB)
http://secoora.org/data
Free desktop analysis tools
MapWindows(shapefile/GIS) http://www.mapwindow.com
ncBrowse(netCDF) http://www.epic.noaa.gov/java/ncBrowse
ODV(ctd/csv) http://odv.awi.de
Sqlite DB http://sqlite.org
• Web services (OGC WMS/WFS/SOS)
http://secoora.org/data
http://code.google.com/p/xenia/wiki/XeniaSOS
Credits
2C.Calloway,2J.Cleary,1J.Cothran,4J.Donovan,3J.Dorton,1M.Fletcher,C.Galvarino,1S.King,2S.Haines,3L.Leonard,1D.Porter,3X.Qi,1D.Ramage,2H.Seim,4V.Subramanian,1S.Walker,4
R.Weisberg
1University of South Carolina, 2University of North Carolina at Chapel Hill, 3University of North Carolina at Wilmington, 4University of South Florida
2C.Calloway,2J.Cleary,1J.Cothran,4J.Donovan,3J.Dorton,1M.Fletcher,C.Galvarino,1S.King,2S.Haines,3L.Leonard,1D.Porter,3X.Qi,1D.Ramage,2H.Seim,4V.Subramanian,1S.Walker,4
R.Weisberg
1University of South Carolina, 2University of North Carolina at Chapel Hill, 3University of North Carolina at Wilmington, 4University of South Florida
Top Related