Esri Best Practices: QA/QC For Your...
Transcript of Esri Best Practices: QA/QC For Your...
![Page 1: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/1.jpg)
Esri Best Practices:
QA/QC For Your GeodataMichelle Johnson & Chandan Banerjee
![Page 2: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/2.jpg)
Overview
Understand Document Implement
![Page 3: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/3.jpg)
Understanding QA/QC
![Page 4: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/4.jpg)
Understanding QA/QC
• Fit For Use
- support your GIS applications?
- support your Business Systems?
• Decision-making
- Good data good decisions save time & money
- Bad data bad decisions more time & more money
• Reporting
- Accuracy & Accountability
![Page 5: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/5.jpg)
Understanding QA/QC
GIS Technician
• Frees up time
• Less rework
• Focus elsewhere
Manager
• Effective data stewardship
• GIS productivity
• Increased GIS usage
Executive
• Confidently make decisions
• Reduce financial risk
• Optimize performance
Knowledge Workers
• Confidence in GIS
• Increase efficiencies
![Page 6: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/6.jpg)
Understanding QA/QC
Daily
Maintenance
Data Conversion/
Migration
Acquisition of Data
(Free or Purchase)
Annual
Data Revisions
![Page 7: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/7.jpg)
Understanding QA/QC
Quality Assurance– Processes or methods to help
prevent errors from being introduced into the data.
- Examples:
- Data Model, Industry-specific Editing Templates,
Attribute Assistant, data-specific editing tools
Quality Control– Processes or tools to
identify errors that are already in the data.
- Examples:
- GP tools (Select by Attribute, Select by Location),
GP Model, Data Reviewer Batch Jobs
![Page 8: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/8.jpg)
6 Elements of Data Quality
The accuracy of the position of features within a spatial reference
system.Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
The presence and absence of features, their attributes and
relationships.
The quality of the temporal attributes and temporal relationships of
features
Classification correctness related to features and their attributes
Adherence to logical rules of data structure, attribution, and
relationships (documented in data product specifications).
The data adhering to the user requirements for its intended use
(ISO-19157:2013 Geographic Information – Data Quality)
![Page 9: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/9.jpg)
Name the Data
Quality Issue
Positional Accuracy
Thematic Accuracy
CompletenessLogical Consistency
Temporal Quality
Usability
![Page 10: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/10.jpg)
Which Element of Data Quality?
Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
Used to map National Parks
Used to route emergency vehiclesIssue
- Street centerline data contains street
names, but no address ranges
- Impact: Not enough information for geocoding
- Inappropriate scale
- Impact: Cannot be used for intended purpose
![Page 11: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/11.jpg)
Which Element of Data Quality?
Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
Swimming pool captured
as wetland
Issue
- Land use polygon coded as residential
instead of commercial
- Impact: Business was not granted a timely permit
- River captured as a single line instead
of a polygon
- Impact: River is not symbolized as being
navigable, when it should be
![Page 12: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/12.jpg)
Which Element of Data Quality?
Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
Neighborhood with missing
building footprint
Issue
- Missing building footprints
- Impact: Planned construction on
top of existing building
- Water mains containing blank or null
diameter attribute values
- Impact: Water utility crews may not have proper
equipment for the job
![Page 13: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/13.jpg)
Which Element of Data Quality?
Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
Issue
• Misplaced or misaligned features
• Impact: Can cause construction problems
• Inaccurate coordinate values
• Impact: Could navigate to incorrect address
Lake feature has been shifted
![Page 14: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/14.jpg)
Which Element of Data Quality?
Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
Outdated chart showing
open runway
Updated chart should show
closed runway
Issue
- Incorrect time or date for discrete
events
- Hydrant status not updated
- Impact: Could affect firefighters’ response time
- Navigational charts
- Impact: Plane could fly into closed airspace
![Page 15: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/15.jpg)
Which Element of Data Quality?
Positional Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
Highway with road surface
type gravel
Issue
• Reducer connects two pipes with same
diameter
• Impact: Could affect hydraulic modeling results
• Road surface type attributed incorrectly
• Impact: Could affect routing and drive times
![Page 16: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/16.jpg)
Documenting QA/QC
![Page 17: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/17.jpg)
Documenting QA/QC
Purpose & Scope
Roles & Responsibility
Testing Environment
Requirements & Acceptance Criteria
QC Workflow– Processes & Tools
Reporting & Addressing Errors (high-level)
![Page 18: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/18.jpg)
Understanding QA/QC
• GIS Applications, Business Systems
- Required fields, feature connectivity,
topological relationships
• What can be automated?
• What will need to be checked manually/visually?
- Data Reviewer Sampling
- Systematic QC using grids
![Page 19: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/19.jpg)
Industry Examples
![Page 20: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/20.jpg)
Industry: Hydrography
• Rule: Lakes and/or ponds must be individual features
• Tool: Data Reviewer Multipart Polygon Check
• Rule: Rivers must flow downhill
• Tool: Data Reviewer Monotonicity Check
4ft
5ft
6ft
8ft
3ft
10ft
11ft
12ft
![Page 21: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/21.jpg)
Industry: Water Utilities
• Rule: Tee fitting must be connected to 3 water lines
• Tool: Data Reviewer Valency Check
Select Junction by Edge Count (Water Utility
Network Reporting toolbar)
• Rule: Hydrants must be connected to hydrant laterals
• Tool: Data Reviewer Geometry on Geometry Check
Find Disconnected Features in Geometric
Network GP Tool
![Page 22: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/22.jpg)
Industry: Facility Management
• Rule: Building Facility Site Identifier must be unique
• Tool: Data Reviewer Unique ID Check
Frequency GP tool
• Rule: Buildings must be greater than a certain size
• Tool: Data Reviewer Evaluate Polygon Perimeter
and Area Check
Select by Attribute
![Page 23: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/23.jpg)
Industry: Land Use Planning
• Rule: Flood zones should not be slivers
• Tool: Data Reviewer Polygon Sliver Check
Database Topology
• Rule: Zoning district polygons should not be duplicates
• Tool: Data Reviewer Duplicate Geometry Check
Find Identical GP tool
![Page 24: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/24.jpg)
Implementing QA/QC
![Page 25: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/25.jpg)
Implementing QA/QC
• Data model
• Industry-specific editing templates
• Attribute assistant
• Data-specific editing tools
![Page 26: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/26.jpg)
Feature Templates &
Attribute Assistant
![Page 27: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/27.jpg)
Implementing QA/QC
• Automated Data Validation
- Geoprocessing Tools & Models
(Toolbox)
- Python Scripts
- ArcGIS Data Reviewer
• Manual Data Validation
- ArcMap Document
- ArcGIS Data Reviewer
- Checklist
![Page 28: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/28.jpg)
Implementing QA/QC
• Daily or when editing
- Step in a workflow
- Current extent
• Periodically
(weekly or monthly)
- Scheduled, automated
- Full database
• New or Updated
Dataset
- When received
- Full dataset
![Page 29: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/29.jpg)
Automated Data
Validation
![Page 30: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/30.jpg)
Leverage GP model to perform data validation
Single Family Home Apartment Complex
Single Family Home
Business rule: Single family parcels must not contain more than one address
![Page 31: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/31.jpg)
Leverage Python Script to perform data validation
Business rule: Compare Features
![Page 32: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/32.jpg)
Summary
• QA/QC is important & perform regularly
• Create a QA Plan & define QC requirements
• Develop a repeatable, automated process
![Page 33: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/33.jpg)
Resources
• LGIM data model
• Industry-related Editing Templates
- Tools
- Attribute assistant
• Industry-related Data Reviewer Templates
- Sample Reviewer Batch Jobs
http://solutions.arcgis.com/
![Page 34: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/34.jpg)
Please Take Our Survey on the App
Download the Esri Events
app and find your event
Select the session
you attended
Scroll down to find the
feedback section
Complete answers
and select “Submit”
![Page 35: Esri Best Practices: QA/QC For Your Geodataproceedings.esri.com/library/userconf/proc18/tech-workshops/tw_1846-39.pdf · 2018 Esri User Conference Presentation Keywords 2018 Esri](https://reader030.fdocuments.in/reader030/viewer/2022011810/5e0c7b13857d4b6b4429e51b/html5/thumbnails/35.jpg)