FME User Conference Presentation
Transcript of FME User Conference Presentation
![Page 1: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/1.jpg)
Data Validation using FME Server
Clay Wise, OCP, GISPEnterprise ArchitectHampton Roads Sanitation District
![Page 2: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/2.jpg)
HRSD (Hampton Roads Sanitation District
Who? What you should come away with
Whistler, BC to Virginia Beach, Virginia = 5,107 km or 3,173 miles
![Page 3: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/3.jpg)
HRSD Mission Statement
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(!(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(
!( !(!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(
!(
!( !(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!( !(!(
!( !(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(
!(!(
!(
!(
!(
!(
#
#
#
#
#
3SRP
HO
LLAND
RD
DAM NECK RD
LON
DO
N B
RID
GE
RD
SHIPPS CORNER RD
SW
ALLO
W D
R
WARWICK DR
TAYLOR FARM RD
HAWK AVE
SK
YLA
RK
DR
OL
D C
LUB
HO
US
E R
D
BAY
HE
AD
DR
BLU
E JA
Y D
R
LYN
NH
AVE
N P
KY
DR
AK
ES
MIL
E R
D
DORY DR
FINCH AVE
RAVEN AVE
CRUSADER CIR
STALLS WAY
PASTURE LN
OLD GLORY RD
NEW LAND DR
SO
NIC
DR
CA
NA
RY
DR
CA
PE
AN
N W
AY
DELAWARE XING
MONDRIAN LOOPVAN LOEN D
R
SCARBOROUGH WAY
WH
ITE
BIR
CH
LN
CR
EEKSIDE D
R
CAPE ANN CT
WIN
DY
WILLO
W C
T
PEELE CTSAXON PL
ENG
AGEM
ENT
CT
SPRUCE PINE RD
DUNNBURY CT
DUNNEBROOK DR
VERMEER DR
FOREST GREEN DR
RU
DD
Y O
AK
CT
WIMBLEDON W
AY
SADDLE CT
REDGROVE CT
MANOR GLEN CT
BARLBOROUGH WAY
YORKBOROUGH WAY
DAM NECK RD
HO
LL
AN
D R
D
CAPE ANN CT
Cardinal Estates
Chesapeake Landing
Oceana East Industrial Park
Oceana South Industrial Park
HRSD SHIPPS CORNER PRS
3TP
3TP
3TP
3TP
3TP
3TP3TP
3TP3TP
3TP
3TP
3TP
3TP
SUFFOLK
MATHEWS
CHESAPEAKE
ISLE OF WIGHT
VIRGINIA BEACH
GLOUCESTER
KING WILLIAM
KING AND QUEEN
MIDDLESEX
JAMES CITY
HAMPTON
YORK COUNTY
NORFOLK
NEWPORT NEWS
POQUOSON
PORTSMOUTH
WILLIAMSBURG
Mission Prevent pollution, protect public health, and support community
development by providing wastewater collection and treatment
![Page 4: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/4.jpg)
Quick Facts
Treatment of Sanitary Sewer for over 1.7 million people.
More than 500 miles of pipes, 6-66 inches in diameter
102 Pump Stations 9 Major Treatment
plants 231 million gallons per
day is treated Serving 17
cities/counties
![Page 5: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/5.jpg)
Regulatory Requirements
Context: Sanitary Sewer Overflows Task: Regional Hydraulic Model
Mostly Pressurized System 14 Jurisdictional Data Inputs 14 Jurisdictional GIS Database Designs Numerous Data Inputs
Over 1,800 jurisdictional pump stations Over 6,500 miles of jurisdictional sewer mains Over 120,000 sewer structures (manholes, vaults, etc…)
Challenge: Data Sharing Jurisdictional Hydraulic Model and general GIS Data Inputs to
HRSD HRSD Regional Hydraulic Model Data to Jurisdictions Data Validation & Standardization, Data Updates & Corrections
![Page 6: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/6.jpg)
Data Validation Tool
What are we validating?
Every attribute (Field) Specific values
(material, locality code) How many were correct
out of the total Is the data spatially
correct? Is the geometry valid? Are their connection
points valid?
![Page 7: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/7.jpg)
Validation Checks
Connection Points Where a jurisdiction connects to HRSD Force Main
system. Checking to see if the IDs match between their submittal and the connection point table we maintain.
Spatially correct Zero length lines Self intersecting polygons **With 14 jurisdictions we have run into all levels
of GIS users. This check should be in your validation process regardless of how simply or complex this process is.
![Page 8: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/8.jpg)
How do you upload and download to FME Server?
FME Workbench Desktop product to develop and publish
data validation workspaces to FME Server Once published the workspace can be run
through a web browser End users do not need an FME license to
run the translation Can also download the posted workspace
to make changes
DOWNLOAD.mp4
![Page 9: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/9.jpg)
How are we validating data?
FME Workbench Desktop product to construct/develop the
validation process Simple upload to publish process to FME
Server
FME Server
Upload &Download
![Page 10: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/10.jpg)
The “nuts and bolts” of data validation
Example Workspace
Tester Transformers
Custom Transformers (Green)
![Page 11: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/11.jpg)
Testers
Looking for:
NULL values
Specific Values (Locality Code)
Range (Elevations)
![Page 12: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/12.jpg)
Showing the results
Using the drop down menu you shouldset this parameter to an attribute that contains unique ids. Even if you don’t want to show these ids on the report it is still required.
cfg01PumpStations:Calls out a specific variable in the report web page.
![Page 13: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/13.jpg)
Showing the results (continued..)
cfg01PumpStations:Calls out a specific variable in the report web page.
<li class="error_message">cfgFailCount of cfgFeatCount features in your dataset have blank PSID attribute values.<ul><li>The ids of the failed features are: cfgFailedIds.</li></ul></li><br/>
![Page 14: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/14.jpg)
Web page editing
I use Dreamweaver (CS3), but you can use cheaper software such as UltraEdit to edit the HTML.
cfg01PumpStations: This is the same variable we called out in the previous slide. After the information goes through the validation process it is placed in this variable. You can change them to whatever name suits you as long as they match the between the workbench file, and the webpage.
![Page 15: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/15.jpg)
Web page editing
Color coding the message displayed. Here I simply have red for an error, and green if all the features passed.
cfg01Nodes: This is the same type of variable I have spoken about in the previous slides.
![Page 16: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/16.jpg)
Results
cfgPipes01
cfgPipes02
cfgPipes04
cfgPipes03
Results are grouped according to the needs of the modelers
Green for passed, Red for failure
Connection points in blue are included with the summary checks
RUN.mp4
![Page 17: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/17.jpg)
Data Upload
Once all the data is correct, a data download page will appear
The jurisdiction will then be able to upload their data through our firewall
![Page 18: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/18.jpg)
Identity Management
Giving the jurisdictions access to our data
Oracle Identity Management
Granular protection inside the HRSD firewall
Single Sign on Reduce Risk Used for many
other processes than just validation
Virginia BeachSuffolkChesapeakeHamptonNorfolkPoquosonPortsmouthWilliamsburgGloucesterIsle of WightJames CityKing and QueenKing WilliamMathewsMiddlesexYork
![Page 19: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/19.jpg)
Suffolk FME Server Example
Suffolk will log in using OIM
They will then find the GIS Parcels, and Water Meters that they maintain
They will then run FME Server
FME Server will join their GIS data to the water consumption data, and output a zip file for their use
![Page 20: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/20.jpg)
Lessons learned
Good first run of the Validation Tool in winter of 2008 Modelers provided feedback for the second run
Geometry Check – Complete Some jurisdictions didn’t have very clean data (i.e. Self
intersecting polygons, zero length lines) Brief summary at the beginning of each feature class –
Complete Summary of linear feet of pipe- Complete Connection point validation – Complete
Positive: HRSD has control over all changes.
![Page 21: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/21.jpg)
Benefits
HRSD has control over all changes
HRSD is able to quickly see where jurisdictions are failing to meet the requirements Instead of weeks of manually checks on each feature class, the
data can be processed in minutes. Sharing the reports with the jurisdictions
Efficient process for keeping data current, and valid. Simple form to fill out. Non-GIS personnel can run the application
over the web. Get to come to cool conferences like this one and talk about it!
![Page 22: FME User Conference Presentation](https://reader035.fdocuments.in/reader035/viewer/2022062308/5555f833d8b42a8a5f8b4fbf/html5/thumbnails/22.jpg)
Questions
Questions?
For more information: Clay Wise: [email protected] HRSD: www.hrsd.com Data validation: fmepedia.com
FME Server > F > FME Server Examples>D > Data Validation with FME Server