2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for...
-
Upload
barry-gibson -
Category
Documents
-
view
216 -
download
1
Transcript of 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for...
![Page 1: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/1.jpg)
2005 Ohio GIS Conference
September 21-23, 2005
Marriott North Hotel
Columbus, Ohio
Geoprocessing for
Animal Premises ID
Luanne HendricksState of Ohio OIT/GISSC InternColumbus State Community College
![Page 2: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/2.jpg)
Overview
• Objective
• Source Data & Desired Outputs
• Timeline
• Tools and Automation
• Process
• Statistics
• Observations
![Page 3: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/3.jpg)
Objective
Geoprocessing
Input:Source Data
from County Auditors
Output: - Normalized Parcel Data - Unique AG Owners
![Page 4: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/4.jpg)
Output - Deliverables
• Normalized Parcel/Point Geodata – agricultural ( 100 <= LUC <= 199)
– dairy (LUC = 103, 113)– residential ( 510 <= LUC <= 520, LUC = 560)
• Normalized Tabular Data (Access DB)– Table of unique ag owners with owner_id– Table of parcel data with owner_id
• Time Estimate to regenerate data annually
![Page 5: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/5.jpg)
Example: Locate Residential Parcels of Ag Land Owners
![Page 6: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/6.jpg)
Example: Select Parcels owned by Owner ID = 2894
![Page 7: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/7.jpg)
Owner to Parcel Table Example
![Page 8: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/8.jpg)
![Page 9: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/9.jpg)
Source Data – Quantity/Quality• Large volume of data
– approx. 5 million source records– some counties had 40-50 fields of data– approx. 5 GB of data
• Multiple source files per county
• Parcel, Point, CAMA data
• Non-standardized data fields
• Variable completeness
![Page 10: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/10.jpg)
Example: Non-Normalized Source vs. Normalized Output
![Page 11: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/11.jpg)
Processing – High Level View
Data Collection from Counties
Normalize Source Data
Generate Owner Ids for Parcel Records
Generate Owner Table
Match Dairy Addresses to Parcel Table
Create Project for User
![Page 12: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/12.jpg)
TimelineFirst Pass
Effort Several PT HC - Approx. 1 FT HC
Tasks Data Collection & Geocoding
Normalizing Owner IDs Dairy Match
Create Project
Month January February March April May
Second Pass
Effort 1 PT HC 1 FT HC
Tasks Identify Original Source used
Manual Normalizing
Automation
Normalizing
Owner IDs
Owner Ids
Dairy match
Project
Month May June July August Sept.
![Page 13: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/13.jpg)
Need Automation Strategy
• Need to automate process for:– Repeatability– Ease of modification– Testability– Traceability
• ...As well as speed
![Page 14: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/14.jpg)
Tools Processing Tasks
ArcToolBox- Model Builder Script development
- Python
- VBscript
Pre-Normalization- Joining source files, - adding key id, -copying to working directory
Pre-Owner ID Generation- Address Standardization- Rejoin Data file to Shapefile
MS Access - VBA- Queries- SQL- Form Interface
- Normalization
- Owner ID & Owner Table
- (Dairy Match)
![Page 15: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/15.jpg)
Processing Detail - Example
Pre-normalization steps in Model-Builder for a county with 2 source files – shape and CAMA that need to be joined. This county is now ready for normalization in Access. Slightly different steps are needed for point files andcounties with a single source parcel shapefile.
![Page 16: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/16.jpg)
Processing Detail - Example Continued
Model-Builder has limitations – you can’t loop through these steps for a list ofcounties. But this model can be converted to script and coded to process alist. Additional field-name mapping steps needed due to “coarse-grained” geoprocessing object.
Loop thrucnty list.
Delete Temporarytable view & layer
Get FieldsMake Field Map
![Page 17: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/17.jpg)
Example of Geoprocessing Tool Limitations
When you join fields in the geoprocessing environment, and create a new Feature Layer shapefile, field names are [original layer name].[field name] truncated to 10 characters. Renaming is not done automatically for you as itis when you join and create a new layer manually in ArcMap.
![Page 18: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/18.jpg)
Python Script Example
![Page 19: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/19.jpg)
Access Form Interface Used for Normalization
![Page 20: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/20.jpg)
Example: Non-Normalized Source vs. Normalized Output
![Page 21: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/21.jpg)
Normalization Mapping Table
![Page 22: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/22.jpg)
Processing – Owner IDs
Data Collection from Counties
Normalize Source Data
Generate Owner Ids for Parcel Records
Generate Owner Table
Match Dairy Addresses to Parcel Table
Create Project for User
![Page 23: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/23.jpg)
Owner ID and Owner Table Generation
![Page 24: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/24.jpg)
Standardized vs. Un-standardized
![Page 25: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/25.jpg)
Owner ID Algorithm
• Aggregate on Lastname, Firstname
• Standardize addresses
• For each Lastname,Firstname group, choose the address - OWNADD1, MAILADD1, or SITEADD, that produces the best set of matches
![Page 26: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/26.jpg)
Statistics
ORIG_REC = Total AG + Total ResidentialNOAD = # Records with no address informationADD_REC = Total # of AG + Total Residential associated with more than 1 parcelFINL_REC = Total # of AG + Total Residential associated with at least one AG pclOWNR = # of Records in the Owner TableNMD_AG = Aggregate of OWNNAM1/MAILADD1 and OWNADD1/MAILADD1
as a sanity check and to compare how effective the processing was
![Page 27: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/27.jpg)
Testing
• Use Statistics– Numbers make sense– Numbers add up, e.g.:
• All records in Parcel table assigned an ownerid• # Records in Owner Table = # Aggregated on Owner Id in
PCL table
• Visual Inspection– Visually inspect how Owner Ids were assigned– Create shapefile and view data in project– Spot check source vs. processed data in shapefiles
![Page 28: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/28.jpg)
Status
• 53 counties normalized
• 40 counties have owner ids/owner table
• Dairy matching - to do
• Final project – to do
![Page 29: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/29.jpg)
Example Project – Work in Progress
![Page 30: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/30.jpg)
Observations and Conclusions (1)
• After initial development, Automation speeds process
• For example, using Form Interface to normalize:
Data NormalizationTime Data
Volume
Manual
1st pass
6 day 1X
Ag only
Auto
2nd pass
1 day 5X
Ag + Res
![Page 31: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/31.jpg)
Observations and Conclusions (2)
• Automation:– speeds process after initial development investment– enables repeatability of process– makes modification and redo less painful– increases data consistency– reduces errors– accurately documents process– increases future capability to do similar processing –
tools are reusable
• Automation is cost effective
![Page 32: 2005 Ohio GIS Conference September 21-23, 2005 Marriott North Hotel Columbus, Ohio Geoprocessing for Animal Premises ID Luanne Hendricks State of Ohio.](https://reader035.fdocuments.in/reader035/viewer/2022062717/56649e495503460f94b3d076/html5/thumbnails/32.jpg)
Observations and Conclusions (3)
• This job would be easier if:– Data was maintained in small standard
components:• Last Name, First Name, MI as separate fields• Address components – SiteNum, SiteDir, SiteStr• There was a standard for field names of
components