WMP Technical Working Group – 14 Feb 2012 …and the Watershed Management Program!

WMP Technical Working Group – 14 Feb 2012

…and the Watershed Management Program!


Last meeting we discussed the difficulties handling large numbers of ArcHydro-generated catchments, specifically including:

1- processing time to get the “micro-catchments”,2- time to review “micro-catchments”, and3- difficulty constructing the HydroNetwork for “micro-catchments”, and4- problems with Preferential Flow Path and catchment grouping.

Recent Developments:

1- Some empirical data on processing times and some surprising findings,2- Ideas to cut down on review time,3- A fix for the HydroNetwork construction, and4- no real progress on catchment grouping (yet.)

Well… three out of four isn’t bad!


1 – Some Empirical Data

Experimental Conditions:

Dell T3500 – Win7-64 SP1; Intel Xeon Dual Core (3.2GHz); 4GB RAM ArcGIS 10: Builds 2414, 2800 (SP1), or 3200 (SP2)ArcHydro 2: Builds 2.1.0.16, 2.1.0.19, 2.1.0.22, 2.1.0.27)

Notes:

Some combinations of ArcGIS 10 service packs and ArcHydro 2 demonstrated incompatibilities. ArcGIS 3200 andArcHydro 2.1.0.16 and ArcHydro 2.1.0.19 produced unexpected errors. This was corrected with 2.1.0.22.

We find the most stable combinations to be ArcHydro 2.1.0.22 and .27with ArcGIS 2414 or 3200. We have not tested ArcGIS SP3

Time trials included processing up to HydroNetwork (Drainage ConnectivityCharacterization tools)


Brooker Creek (Hillsborough County portion)Basin Area ~23.5 sq. milesDEM Size = 56,000,000 cells

SWF13023 SWF13023Sink Fill Area (in US Acres) 4356 sq ft NoFIllNumber of Catchments/Sinks 1,703 418,542

Sink PreScreening 0:04:07 0:04:00Sink Evaluation 0:05:55 0:48:45Fill Sinks 0:00:34 0:00:55Flow Direction with Sinks 0:04:20 0:12:30Flow Accumulation 0:02:13 0:01:16CatchmentGridDelineation 0:00:53 0:00:53Catchment Polygon Processing 0:00:21 0:13:34Drainage Point Processing 0:00:28 0:16:22Drainage Boundary Definition 0:00:48 1:24:18Sub-Total Processing Time 0:19:39 3:02:33

Drainage Connectivity (HydroNet) 0:03:26 7:08:47Preferential Flow 0:03:11 0:00:00

TOTAL PROCESSING TIME 0:26:16 10:11:20ArcHydro statistics:

BoundaryDrainage lines – 5,248 vs. 1,231,134 Sink Points – 1703 vs. 418,542Hydro Edges – 10,059 vs. 2,453,996HydroJunctions – 6,934 vs. 1,600,863


Just a real quick calculation…

1 Day =

24 hours =

1440 minutes =

86,400 seconds

So…to review (just look at) each catchment inBrooker Creek for just 1 second each….

418,542 catchments (seconds) / 86,400 seconds/day =

4.85 DAYS

That is non-stop, doing ABSOLUTELY NOTHING else!


Pemberton Creek – Hillsborough CountyPemberton Creek

Basin Area ~65 sq. miles

DEM Size = 156,000,000

ArcGIS Build 2414 SP0 2414 SP0

ArcHydro Build 2.1.0.19 2.1.0.19

Computer SWF13325 SWF13325

Sink Fill Area (in US Acres) 0.1 acre fill No Fill

Number of Catchments/Sinks 1158 965,076

Sink PreScreening 0:13:51 0:13:06

Sink Evaluation 0:18:38 1:50:21

Fill Sinks 0:02:49 0:02:50

Flow Direction with Sinks 0:10:24 8:06:46

Flow Accumulation 0:10:30 0:03:17

CatchmentGridDelineation 0:03:50 0:03:05

Catchment Polygon Processing 0:00:23 0:30:25

Drainage Point Processing 0:00:59 0:34:32

Drainage Boundary Definition 0:00:41 3:15:02

Sub-Total Processing Time 1:02:05 14:39:24

Drainage Connectivity (HydroNet) 0:03:55 17:03:00

Preferential Flow

TOTAL PROCESSING TIME 1:06:00 31:42:24


Lines Time

North1 228663 1:21:04North2 290735 1:41:32North3 211685 1:15:01Central4 254676 1:32:49Central5 236254 1:24:37Central6 287534 1:45:03Central7 166714 1:02:16Central8 98580 0:37:25Central9 159914 0:58:55Central10 208352 1:15:44Central11 154406 0:58:30Central12 146900 0:56:46South13 153296 0:57:53South14 200880 1:15:25

Total 17:03:00

Pemberton Creek – ArcHydro HydroNetwork “Chunks”

The Drainage Connectivity Characterization Tool works on a selected set of Catchments(= ArcHydro “Chunk”)


Pemberton Creek Catchments


Raleigh-Rogers-Starvation Lake Watershed

Basin Area ~ 69.5 sq. miles

DEM Size = 156,000,000 cells

SWF13023 SWF13023 SWF13023 SWF13023 SWF13023

Sink Fill Area (in US Acres) 1 acre 0.5 acre 0.25 acre 4356 Sq. Ft. No Fill

Number of Catchments/Sinks 1,916 2,097 3,785 80,147 865,418

Sink PreScreening 0:12:06 0:11:54 0:11:54 0:11:53 0:11:59

Sink Evaluation 0:05:47 0:05:57 0:06:47 0:14:48 1:38:35

Fill Sinks 0:02:03 0:02:40 0:02:37 0:03:16 0:03:44

Flow Direction with Sinks 0:12:04 0:11:10 0:11:10 0:09:02 13:01:00

Flow Accumulation 0:06:39 0:06:10 0:06:01 0:04:40 0:03:45

CatchmentGridDelineation 0:02:30 0:02:29 0:02:23 0:02:01 0:02:42

Catchment Polygon Processing 0:00:34 0:00:34 0:00:44 0:04:07 0:28:05

Drainage Point Processing 0:00:55 0:00:55 0:00:58 0:03:27 0:37:51

Drainage Boundary Definition 0:01:11 0:01:15 0:01:42 0:17:21 3:44:39

Drainage Connectivity (HydroNet) 0:05:25 0:05:28 0:08:01 1:33:11 16:01:12

Preferential Flow 0:00:00 0:00:00 0:00:00 0:00:00 0:00:00

Total Processing Time 0:49:14 0:48:32 0:52:17 2:43:46 35:53:32

Raleigh-Rogers-Starvation Lake Watershed


Pinellas-Anclote Watershed


Watershed Size (sq. Miles) = 63.3

DEM size (number of Cells) = 180,325,425

Processing Machine SWF13322 SWF13322 SWF13023 SWF13023

ArcGIS Build/Service Pack 2800 SP0 2800 SP0 2414 SP0 2414 SP0

ArcHydro Build 2.1.0.16 2.1.0.16 2.1.0.19 2.1.0.19

Sink Fill Area (in US Acres) Half acre Fill Quarter Acre Fill 4356 SqFt 625 Sq Ft

Number of Catchments/Sinks 1,999 3,312 1,659 2,332

Sink PreScreening 0:17:56 0:17:50 0:16:54 0:15:38

Sink Evaluation 0:09:24 0:09:56 0:18:20 1:33:18

Fill Sinks 0:04:27 0:03:28 0:03:39 0:03:48

Flow Direction with Sinks 0:14:17 0:13:36 0:17:07 0:16:01

Flow Accumulation 0:09:52 0:09:33 0:12:44 0:10:24

CatchmentGridDelineation 0:05:49 0:05:04 0:06:07 0:05:25

Catchment Polygon Processing 0:00:34 0:00:42 0:00:31 0:00:37

Drainage Point Processing 0:00:59 0:01:01 0:01:07 0:01:53

Drainage Boundary Definition 0:01:10 0:01:33 0:01:01 0:01:15

Drainage Connectivity (HydroNet) 0:06:07 0:08:22 0:05:31 0:06:54

Preferential Flow 0:00:00 0:00:00 0:00:00 0:00:00

Total Processing Time 1:10:35 1:11:05 1:23:01 2:35:13



Watershed Statistics (625 sq. ft. fill): 2,332 Sinks/Catchments 6,971 Drainage Boundary Lines 9,281 HydroJunctions13,495 HydroEdges



Anclote Watershed

AncloteBasin HydroJunction

HydroEdgeCatchment

1:30,000 – 625 sq. ft. fill



Anclote Watershed

AncloteBasin HydroJunction

HydroEdgeCatchment

1:30,000 – No Fill



1:1200 No Fill

Watershed Statistics (No Fill) 1,146,540 Sinks/Catchments 3,371,256 Drainage Boundary Lines 3,733,211 HydroJunctions 5,156,619 HydroEdges

ArcGIS Build/Service Pack 2414 SP0 Chunk Lines TimeArcHydro Build 2.1.0.19

North1 430288 2:34;59Sink Fill Area (in US Acres) No Fill North2 333308 1:59:40Number of Catchments/Sinks 1,146,540 North3 147605 0:55:26

North4 122112 0:45:18Sink PreScreening 0:16:59 Central5 341814 2:08:15Sink Evaluation 2:11:28 Central6 230942 1:28:11Fill Sinks 0:04:00 Central7 240075 1:28:59Flow Direction with Sinks 14:37:55 Central8 227590 1:25:44Flow Accumulation 0:03:47 Central9 150866 0:57:18CatchmentGridDelineation 0:03:58 South10 140568 0:53:31Catchment Polygon Processing 0:36:22 South11 213308 1:20:04Drainage Point Processing 0:44:59 South12 281958 1:45:03Drainage Boundary Definition 3:59:51 South13 510882 3:10:05Drainage Connectivity (HydroNet) 18:17:34Preferential Flow 0:00:00

Total Processing Time 40:56:53

That is the equivalent of 17.7 days of Catchment evaluation!


Conclusions: 1- Some fill is necessary to produce a “reasonable” number of ArcHydro

catchments for human evaluation. Results indicate that between0.1 acre-fill (4360 sq. ft.) and 5x5 DEM cell (625 sq. ft) fill seemsappropriate,

2- Filling to one of these thresholds yields ~ 1000 – 3000 ArcHydro Catchments for human evaluation,

3- As the Drainage Connectivity Characterization tool now operates onselected sets, smaller areas of interest (AOIs) can be selectedfrom the DEM and micro-catchments can be generated for those AOIs. Chunking and appropriate workflows can be usedto re-integrate AOIs into the HydroNetwork,

4- There are several relationships between the number of elements andprocessing times. Adding catchments/sinks increases the timefor Flow Direction computation but decreases the time for FlowAccumulation computation,

5- The “chunking” workflow permits computing HydroNetworks for LARGEwatersheds. Empirical data suggest that the processing takes~ one hour per 150000 DrainageBoundary lines.


Part 2 – ArcGIS Data Reviewer (formerly known as Data ReViewer)


Current StatusSince 2008, the District has been using a third-party ESRI extension, DataQE, forinsuring consistent Data Quality.

Advantages:

1- Works in the ArcGIS Environment,2- User customizable dictionary and rule set,3- Ruleset can be easily distributed to multiple users,4- Reasonable learning curve and cost point.

Disadvantages:

1- Doesn’t make very good use of the geodatabase,2- Doesn’t include any topological/geometric checks,3- Doesn’t track edits and audits made to the database,4- Doesn’t include any comment functionality (or tracking),5- Doesn’t record information regarding reviewers and editors,6- Doesn’t indicate severity of errors.

http://www.datalogic-systems.com/dataqe/index.htm


Status under Evaluation for Adoption

Prior to ArcGIS 10, ESRI bundled several products into the Production Line Tool Set (PLTS) package, including:

Workflow ManagementJob TrackingVersion ManagementData Quality Management

Although these are still available as a bundled product, in 2011, ESRI beganoffering the Data Quality Management portion as either a “stand alone” or concurrently licensed extension as:

ArcGIS Data Reviewer



Advantages: 1- Tightly integrated into ArcGIS

2- Reviewer Checks include,

Database Validation ChecksPolygon ChecksPolyLine ChecksTable ChecksTopology ChecksZ-Value ChecksSpatial Parameter EvaluationFeature on Feature ChecksDuplicate Geometry Checks



Advantages continued,3- Maintains QC calls in feature classes and geodatabase tables,

Feature Dataset Reviewer Tables



Advantages continued,4- Maintains Detailed information in the REVTABLEMAIN, such as:

Check types, who made it, severity of error, who repaired it,when/who checked the repair, etc.



Advantages continued,5- Easy to construct and distribute the rule set. User can select all or a subset of rules to run.



Advantages continued,6- Facilitates manual review by incorporating a user-defined grid

for review,7- User formatted, flexible report generator, and8- Insures that corrections are made and QCed!9- ESRI Maintained and managed software; will always be

“in step” with ArcGIS.

Disadvantages,

1- Higher cost point than DataQE (a standard ESRI Extension)for a single-use license ($2500 vs $899)

2- Steeper learning curve than DataQE (additional flexibilityand more options than DataQE)

3- Requires modifications to current workflows (1st reviewer, editor, 2nd reviewer workflow)

4- Additional data tables (and feature classes) are added to GWISto manage.


Moving Forward

1- District Staff have developed rule sets for GWIS including:

2- District Staff are developing and testing workflows for implementation on WMP Projects,

3- District Staff have attended an ESRI Training Session and are developingGWIS-centric Training Materials,

4- Mapping & GIS has deployed ArcGIS Data Reviewer for ArcGIS 10

5- Mapping & GIS will phase out support of DataQE, starting in January 2012 and ending in December 2012. Mapping & GIS will not supportDataQE after January 2013.


QUESTIONS AND/OR COMMENTS ???

WMP Technical Working Group – 14 Feb 2012 …and the Watershed Management Program!

Documents

Transcript of WMP Technical Working Group – 14 Feb 2012 …and the Watershed Management Program!