7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS...
-
Upload
magnus-rich -
Category
Documents
-
view
213 -
download
0
Transcript of 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS...
7 Nov 2007 2007 Geospatial Interoperability Summit
Iowa Environmental Mesonet: Using Open Source GIS Tools and Web
Services to Disseminate Environmental Data
Daryl HerzmannIowa StateUniversity
7 Nov 2007 2007 Geospatial Interoperability Summit
Data, data, everywhere data
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
The Desperate Challenge
• Disparate Formats– Grib– METAR– Text CSV– Free form text– ADAS III– SHEF– Excel (gasp!)
• Disparate Network Protocols– HTTP– FTP– LDM IDD (tcp)– TCP stream– Modems
The data comprising the IEM is made available to us in the following formats and network protocols.
7 Nov 2007 2007 Geospatial Interoperability Summit
Why an Open Source solution?
1. They are the best tools out there. UNIX mentality -> Do something small and do it well.
2. Transparency. We are a public institution doing public funded research.
3. Excellent Support• Public bug trackers• Community forums• IRC• Motivated developers
7 Nov 2007 2007 Geospatial Interoperability Summit
Open Source SolutionPostGIS / PostgreSQLSpatial Database
Map ServerMapping Engine +Scripting
Red Hat Linux Operating System
PythonData processing andnetwork services
GDALSwiss Army Knifeof all things GIS
PHPDynamic Website
7 Nov 2007 2007 Geospatial Interoperability Summit
Key Component: PostGIS
• Early IEM code (2001) was a series of flat files in Shapefile format and database of observations. This was a disaster.
• PostGIS provided the opportunity to manage data, metadata, and the GIS data in one location.– Network scalability– Transaction / stored procedure support– Speed / functionality with merge of GEOS library
7 Nov 2007 2007 Geospatial Interoperability Summit
Look at all of my data!
• 1 billion observations• 1 terabyte of data
– Archives of NEXRAD, GOES Satellite, precipitation datasets, web cameras!, Weather Service products, …….
• Now what? Swim in all of my data?
7 Nov 2007 2007 Geospatial Interoperability Summit
Here’s an idea, lets create a webpage where they can download the data!
• Daryl:– Creates webpage
• End user:– Finds my page– Fills out form– Saves data, attempts to
figure out format– Attempts to find geo-
referencing for their GIS.– Hopefully loads data into
app– Commits numerous errors
along the way
7 Nov 2007 2007 Geospatial Interoperability Summit
The Open Geospatial Consortium (OGC) Way
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
In 2002, we built OGC services, but they didn’t come….
• Why?– Where were the cool
clients?– ESRI’s OGC support
wasn’t there yet– It all felt slow as
compared with local disk access
– A bit ahead of its time
7 Nov 2007 2007 Geospatial Interoperability Summit
In 2007, they have come!
• Thanks to cool clients– Google Maps– OpenLayers– ESRI Interop Extension– Map Server and others
• ~ 2 million hits per day (~20 “maps” per second)
7 Nov 2007 2007 Geospatial Interoperability Summit
Supporting the Load
• 12 systems comprise IEM web farm
• Various levels of redundancy, but no spatial database redundancy yet
• Roughly 30 GB in , 60 GB out each day
• Linux IPVS provides redundant entry point and load balancing
7 Nov 2007 2007 Geospatial Interoperability Summit
Available OGC Services
7 Nov 2007 2007 Geospatial Interoperability Summit
http://mesonet.agron.iastate.edu/ogc/
• CONUS GOES (Infrared, Visible)
• CONUS NEXRAD Base Reflectivity, Storm Total Precip, 1 Hour Precip
• NWS issued advisories,watches, and warnings
• Iowa Rainfall Estimates
• Iowa Road Conditions
• IEM Data Services (temp, winds, etc)
7 Nov 2007 2007 Geospatial Interoperability Summit
Raw NEXRAD Composites
Have a large amount of noise due to ground clutter. A simple GDAL scriptcomparing radar echos to its elevation quickly gets rid of most noise.
7 Nov 2007 2007 Geospatial Interoperability Summit
Cleaned Composite
7 Nov 2007 2007 Geospatial Interoperability Summit
Some IEM developed GIS webapps
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
4 May 2003, heavy rains on freshly tilled soils
25 Sep 2005, heavy rains on mature crops
7 Nov 2007 2007 Geospatial Interoperability Summit
GIS + webcams
IEM developed software to have the cameras automatically track the sun and the moon. Can track NEXRAD identified storms as well!Done using FOSS GIS tools.
7 Nov 2007 2007 Geospatial Interoperability Summit
What others have done.
7 Nov 2007 2007 Geospatial Interoperability Summit
7 Nov 2007 2007 Geospatial Interoperability Summit
Lessons Learned
• Adopting OGC services can greatly reduce the complexity of serving out your data and using your own data!
• Performance hasn’t been an issue, most of the overhead is with HTTP.
• For Map Server, try to limit the number of mapfiles you maintain!
7 Nov 2007 2007 Geospatial Interoperability Summit
Future Work
• Increase Metadata!
• Help support NetCDF in GDAL (?)
• Work with end users on getting IEM data into their platform.