Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam...
-
Upload
dwain-reeves -
Category
Documents
-
view
214 -
download
0
Transcript of Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam...
![Page 1: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/1.jpg)
Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land ManagementBy Adam Ridley, GIS Specialist
Cottonwood Field Office – Idaho BLM
![Page 2: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/2.jpg)
2
Overview
► BLM Mission and Structure
► Idaho BLM GIS Architecture and Directory/File System
► Data Management Challenges– 4 Components
► Background and Issues
► Coding Approach
► Timeline
► Results
► Questions
![Page 3: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/3.jpg)
3BLM Mission & Structure
Washington Office
Idaho State Office
Cottonwood Field Office
11 Additional Field Offices
12 Other State/Region
Offices
Many Local Field Offices
Bureau-wide
State Office
Field Office
Mission:To sustain the health, diversity, and productivity of America’s public lands for the use and enjoyment of present and future generations.
Photo Credit: Craig Johnson, Fishery Biologist
![Page 4: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/4.jpg)
4Idaho BLM
► 12 million acres in Idaho
► 1 National Conservation Area
► 1 National Monument
► 16 Wild and Scenic Rivers covering 318 miles
► 8 Wilderness Areas covering 541,812 acres
► 4 National Historic Trails covering 452 miles
► 1 National Scenic Trail covering 13 miles
![Page 5: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/5.jpg)
5
Idaho BLM GIS Architecture
► Esri-based enterprise GIS using SDE
► Data pertains to Cultural Resources, Forestry, Range, Recreation, Wildlife, etc.
► Largely rural state, limited telecom infrastructure
► History of bandwidth issues and network instability
► Local file servers at each Field Office; local ArcGIS Desktop installations; local data
► Limited Web GIS development
Photo Credit: Ryan Turner, Park Ranger
![Page 6: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/6.jpg)
6
Directory/File System
Server RootGIS Root
Project Example
![Page 7: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/7.jpg)
7
Data Management Challenges
► Challenge 1: Layer files reference incorrect fields or have broken links
► Challenge 2: Local corporate datasets become decadent due to problems with replication
► Challenge 3: Metadata is not typically included in the one-way replication process
► Challenge 4: Changes to corporate data or archiving projects breaks links in map documents
![Page 8: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/8.jpg)
8
► Layer files function by storing a snapshot of a layer’s properties
► Idaho BLM maintains a library of layer on each local server
► Roads, Fires, Boundaries, Ownership, etc.
► Layer files link to both replicated and local reference data
► Changes to reference data can break layer files
Layer File ValidationChallenge 1
![Page 9: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/9.jpg)
9
Replica CurrencyChallenge 2
► Replication – Esri process to manage changes across an enterprise GIS.
► One-way replication
► State Office to Field Offices
► Runs Nightly
► Uncertainty results from:
► Not all datasets are set for regular replication.
► Nightly replication can fail for a variety of reasons
► Replication failures not reported to Field Office GIS staff
State Office server
Field Office server
One -wayReplication
![Page 10: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/10.jpg)
10
Metadata CurrencyChallenge 3
► Similar issues to Replica Currency
► Metadata changes not captured through replication process
► Updates to metadata occur when replica is rebuilt or when pushed manually.
► Either scenario infrequent at best
![Page 11: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/11.jpg)
11
MXD ValidationChallenge 4
► MXDs are an Esri file format – maintain symbolized data and map layouts
► MXDs store file paths one of two ways:
► An absolute path (“R:\loc\GIS\PROJECTS\Heckman CE\Heckman Inclusions Oct_2014.shp”)
► A path relative to the location of the MXD (“.\HeckmanInclusionsOct_2014.shp”)
► Links break if:
► MXD or referenced data is moved
► Referenced data is renamed
How do we maintain viability of MXDs over long term and support archiving?
► Final Data favors absolute paths; project data favors relative paths
![Page 12: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/12.jpg)
12
Why Code? Why Python?
► Ability to schedule the script for off-hours execution
► Reduces likelihood for schema locks
► Less network competition
► Consistency in task execution
► Personnel time and associated costs with manual process
► Department of Interior (DOI) and BLM IT policies require extensive vetting process for applications
![Page 13: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/13.jpg)
13
Coding Approach - General
► Familiar access and feel through ArcToolbox
► 4 scripts, 4 tool interfaces
► May develop a stand-alone GUI using wxPython
► Options to Analyze & Report or Analyze & Fix for each script
![Page 14: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/14.jpg)
14
Coding Approach – Layer File ValidationChallenge 1
Solution: Validate layer files and fix or remove broken links
Check the data source and symbology value fields arcpy.mapping.ListBrokenDataSources (Tilton 2011) and arcpy.Describe (Watkins 2014)
Find correct dataset to re-link• d
atasetName property of Layer objects
![Page 15: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/15.jpg)
15
Coding Approach – Replica CurrencyChallenge 2
Solution: Test our local geodatabases against the State Office data stack for currency and update as needed
Index both the replica and parent geodatabases• a
rcpy.da.Walk function
Test analogous feature classes’ similarity• M
odified dates
• arcpy.Describe objects
• Feature Compare (Data Management > Data Comparison > Feature Compare)
Update if indicated
![Page 16: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/16.jpg)
16
Coding Approach – Metadata CurrencyChallenge 3
Solution: Check for metadata updates on the State Office data stack and replace local metadata as needed
Update metadata• (
Conversion Tools > Metadata > Import Metadata)
![Page 17: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/17.jpg)
17
Coding Approach – MXD ValidationChallenge 4
Solution: Test MXDs for broken links and repair or replace data as needed
Locate and index MXDs• a
rcpy.da.Walk
Collect layers with broken links• a
rcpy.mapping.ListBrokenDataSources
Re-llink to appropriate data source• R
elatively simple for Final Data; may use look-up table
• Difficult for project data, particularly if renamed
![Page 18: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/18.jpg)
18
Timeframe
Jan-Feb •2 weeks of development time for each of the 4 components, totaling 8 weeks of development time
Mar •Additional 4 weeks for documentation, testing/debugging, GUI development, etc.
Apr •Presentation to Idaho Leadership Committee
![Page 19: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/19.jpg)
19
Results
► Similarity in server structures allow for state-wide application
► Confidence in our local data’s accuracy and currency
► Less time spent fixing data and projects
► Reliability of data stores for end users
![Page 20: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/20.jpg)
20
References
Tilton, Thad. Charming the Snake: An Introduction to Python and the ArcPy site-package. ArcUser Vol. 14 ,No. 1 (Winter 2011), p. 53-55.
Watkins, David. Using Python to Gather Information about Data in SDE. 2014 Esri User Conference Proceedings.
![Page 21: Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land Management By Adam Ridley, GIS Specialist Cottonwood Field Office – Idaho.](https://reader036.fdocuments.in/reader036/viewer/2022062519/5697c0141a28abf838ccd0dc/html5/thumbnails/21.jpg)
21
Questions?Photo Credit: Craig Johnson, Fishery Biologist