Webex presentation to FL DBA and Data Webex presentation to FL DBA and Data ChiefsChiefs

Matt Petkewich, USGSMatt Petkewich, USGS

March 4, 2010March 4, 2010

1

Impacts of Everglades Depth Estimation Network (EDEN) on

NWIS in Florida

Everglades Depth

Estimation Network (EDEN)

2

EDEN Daily Water-Level

Surface

3

4

Computation of water depth

DEM Water level surface

Ground Surface

Water SurfaceWater Surface – Ground Surface = Water Depth

Water depth surface

4

5

http://sofia.usgs.gov/eden

EDENweb directs users to NWIS

NWIS PAGE FOR A GAGE

5

EDEN DATA TeamMember Location Responsibility

Pamela Telis FL WSC Project leader

Heather Henkel FL GD – St. Pete

Water-level surfacing; GIS, database, and web work

Bryan McCloskey

FL GD – St. Pete

Water-level surfacing; database and web work, meteorological data

Paul Conrads SC WSC Hindcasting data/Gapfilling

Matthew Petkewich

SC WSC Water-level data QA; processing; and Gapfilling

Nate Matthews SC WSC Water-level data QA

Carlton Wood SC WSC Water-level data QA

Brian Reece/Joe Vrabel

TX WSC Programming of scripts to process data

Mike Holmes “Retired”/contractor

Helps as needed on NWIS/cooperator data issues 6

7

Procedure used for processing:3 days of data - daily

3 months of data - quarterly12 months of data - annually

8

Daily processing (3 days of data):USGS – 84 sites

ENP (USNPS) – 53 sitesSFWMD (FL005) – 113 sites

9

Time Action

5:02 a.m. File of South Florida Water Management District (SFWMD) data received

7:00 a.m. File of Everglades National Park Service (ENP) data received

8:15 a.m. uploadEDEN.pl : ENP and SFWMD data pushed into NWIS

8:15 - 11:45 a.m.

EDEN data (ENP, SFWMD, and USGS) reviewed using NWIS web to locate any bad data

8:15 – 11:45 a.m.

Bad data edited (deleted/shifted) in ADAPS , if required

12:00 p.m. ftpedenoutwat.sh: data processed for input to EDEN Server; EDEN Server to create median input files, reject files, and water surface maps

10

Quarterly processing (3 months of data):

USGS – 84 sitesENP (USNPS) – 53 sites

SFWMD (FL005) – 113 sites

11

Annual processing (12 months of data):USGS – 84 sites

ENP (USNPS) – 53 sitesSFWMD (FL005) – 113 sites

12

EDEN workspace

13

Current data processing scripts:• uploadEDEN.pl (at 8:15 a.m.):

• batch uploads ENP and SFWMD data into an archive and NWIS

• sends out a summary email to data team• perl environment used; no external system calls

• edenman_cronjob_1.sh – Calls the following 3 scripts:• ftpeden.sh – retrieves and archives cooperator data

from FTP• enpcron.sh – parses and drops off ENP data• sfwmdcron.sh – parses and drops off SFWMD data

…NWIS UPLOAD…

• run_rsEDEN.sh - calls rs utility• run_stdstoreEDEN.sh – calls std_store utility

14

EDEN workspace

15

NWIS web and ADAPS:

16

Current data processing scripts:ftpedenoutwat.sh (at 12:00) – retrieves 3

previous days of data from NWIS in OUTWAT RDB format. Creates files for processing of WL surfaces.

17

Procedure used for processing Quarter/Annual Data:

• 1-4 quarters of data received from cooperators• Cooperator data from period of interest deleted to allow clean reload of new data• The new datasets from the cooperators uploaded in NWIS• QA procedures include filling missing data with

estimated data using data from adjacent sites• Estimated data is used for EDEN project only and is not loaded into NWIS

18

Schedule of Annual Server Activities

Date Activity

Daily Process 3 days worth of data

Mid-Feburary Process 4th quarter data

Mid-May Process 1st quarter data

Early June Process previous water year (4 quarters of data)

Mid-August Process 2nd quarter data

Mid-November Process 3rd quarter data

Periodic (as needed) Re-process individual/multiple quarters

19

20