Webex presentation to FL DBA and Data Webex presentation to FL DBA and Data ChiefsChiefs
Matt Petkewich, USGSMatt Petkewich, USGS
March 4, 2010March 4, 2010
1
Impacts of Everglades Depth Estimation Network (EDEN) on
NWIS in Florida
Everglades Depth
Estimation Network (EDEN)
2
EDEN Daily Water-Level
Surface
3
4
Computation of water depth
DEM Water level surface
Ground Surface
Water SurfaceWater Surface – Ground Surface = Water Depth
Water depth surface
4
5
http://sofia.usgs.gov/eden
EDENweb directs users to NWIS
NWIS PAGE FOR A GAGE
5
EDEN DATA TeamMember Location Responsibility
Pamela Telis FL WSC Project leader
Heather Henkel FL GD – St. Pete
Water-level surfacing; GIS, database, and web work
Bryan McCloskey
FL GD – St. Pete
Water-level surfacing; database and web work, meteorological data
Paul Conrads SC WSC Hindcasting data/Gapfilling
Matthew Petkewich
SC WSC Water-level data QA; processing; and Gapfilling
Nate Matthews SC WSC Water-level data QA
Carlton Wood SC WSC Water-level data QA
Brian Reece/Joe Vrabel
TX WSC Programming of scripts to process data
Mike Holmes “Retired”/contractor
Helps as needed on NWIS/cooperator data issues 6
7
Procedure used for processing:3 days of data - daily
3 months of data - quarterly12 months of data - annually
8
Daily processing (3 days of data):USGS – 84 sites
ENP (USNPS) – 53 sitesSFWMD (FL005) – 113 sites
9
Time Action
5:02 a.m. File of South Florida Water Management District (SFWMD) data received
7:00 a.m. File of Everglades National Park Service (ENP) data received
8:15 a.m. uploadEDEN.pl : ENP and SFWMD data pushed into NWIS
8:15 - 11:45 a.m.
EDEN data (ENP, SFWMD, and USGS) reviewed using NWIS web to locate any bad data
8:15 – 11:45 a.m.
Bad data edited (deleted/shifted) in ADAPS , if required
12:00 p.m. ftpedenoutwat.sh: data processed for input to EDEN Server; EDEN Server to create median input files, reject files, and water surface maps
10
Quarterly processing (3 months of data):
USGS – 84 sitesENP (USNPS) – 53 sites
SFWMD (FL005) – 113 sites
11
Annual processing (12 months of data):USGS – 84 sites
ENP (USNPS) – 53 sitesSFWMD (FL005) – 113 sites
12
EDEN workspace
13
Current data processing scripts:• uploadEDEN.pl (at 8:15 a.m.):
• batch uploads ENP and SFWMD data into an archive and NWIS
• sends out a summary email to data team• perl environment used; no external system calls
• edenman_cronjob_1.sh – Calls the following 3 scripts:• ftpeden.sh – retrieves and archives cooperator data
from FTP• enpcron.sh – parses and drops off ENP data• sfwmdcron.sh – parses and drops off SFWMD data
…NWIS UPLOAD…
• run_rsEDEN.sh - calls rs utility• run_stdstoreEDEN.sh – calls std_store utility
14
EDEN workspace
15
NWIS web and ADAPS:
16
Current data processing scripts:ftpedenoutwat.sh (at 12:00) – retrieves 3
previous days of data from NWIS in OUTWAT RDB format. Creates files for processing of WL surfaces.
17
Procedure used for processing Quarter/Annual Data:
• 1-4 quarters of data received from cooperators• Cooperator data from period of interest deleted to allow clean reload of new data• The new datasets from the cooperators uploaded in NWIS• QA procedures include filling missing data with
estimated data using data from adjacent sites• Estimated data is used for EDEN project only and is not loaded into NWIS
18
Schedule of Annual Server Activities
Date Activity
Daily Process 3 days worth of data
Mid-Feburary Process 4th quarter data
Mid-May Process 1st quarter data
Early June Process previous water year (4 quarters of data)
Mid-August Process 2nd quarter data
Mid-November Process 3rd quarter data
Periodic (as needed) Re-process individual/multiple quarters
19
20
Top Related