SQL, a Master Address Repository&

FME

The Oregon Geospatial Enterprise Office (GEO)

is assembling a statewideMaster Address Repository

● 36 Counties

● 242 Cities

● 2 State Plane Zones

● Nobody knows how many addresses

What's a Master Address Repository?

Why is it different from keeping the addresses in my favorite

application?● A Master Address Repository (MAR) is for

general use

– Provides multiple views of the data

– Supports multiple applications

– Supports new applications as they arrive

How do you know how to do this?Is there a standard?

● Yes, theFGDC United States Thoroughfare, Landmark and Postal Address Data Standard– Content

– Classification

– Data Quality

– Data Exchange

● The standard enables exchanging address information losslessly

● The Address Standard structures support MAR database designs because they target the same issues.

● Oregon is committed to it.

It's a little more complicated than it sounds.

Addresses have lots of moving parts

Geodatabases just don't do that sort of thing.

● Problem

– Relationship classes come with a performance cost

– It's far greater than the cost of a join in a relational database

● Solution

– Store geometry in the geodatabase

– Store the tabular information in a relational database

– Join them in a spatial view

● The relational database is a much bigger part of the MAR than the geodatabase

Constraints

● No specifications for source data. Anything digital.

● No command line SQL

● No subject matter expertise in-house

● Processes have to be usable for GUI-only, ESRI-centric staff

Solution: FME

● Allows shaping flexible processes

– Build in standards

– Build in address “smarts”

– Automate quality processes

– They already owned a copy – it was just sitting there.

The Process of Translation

● Client receives data

● Client catalogs data and ensures that the data is ready for translation

● Using a set of FME workspaces the client iteratively transforms the data into a standard format

● Using FME the client stores and checks the data for quality and consistency

● Output final deliverable packages and deliver to originating source

How to Package Expertise

One challenge was the lack of in house expertise. This meant packaging years of expert knowledge in

– Data management

● Data organization

● Text processing

– Workflow design

– Quality control

FME allowed this to be done in an environment easily understood by the client

Some Examples● Main translation bus

– This workspace is the initial translation from the received data format into the data staging tables

● Updating simple domain values

– This series of workspaces process parsed data stored in the staging table to find updates to the simplest address components

● Updating complex domain values

– This series workspace process the simple components to create the more complex values requires to represent an address

● Quality control process

– This set of workspaces check the integrity of the data

● Fishbones

– A specific quality control measure used to connect address points to the street they sit on

Main Translation Bus

● Input: Received data file

● Output: SQL Server

This process is intended to begin the process of moving the delivered data in to the MAR. The first step is to get it in to a working (“staging”) table.

Simple Domain Inserts

● Input: SQL Server

● Output: SQL Server

This process looks in the simple domain table, StName, for new values then inserts those new values.

Complex Domain Inserts

● Input: SQL Server

● Output: SQL Server

This process locates complex new complex domain elements and then adds these new elements to the appropriate table.

Quality Control: RelatedNotNull● Input: SQL Server

● Output: CSV

No process is perfect and because of that it is critical to check the quality of data prior to package and delivery. This specific check, as defined in the FGDC address standard, ensures that all fields that should have foreign keys are filled correctly.

Quality Control: Fishbones● Input: SQL Server

● Output: ESRI Feature Class

The fishbones workspace creates inventory address fishbones that tie address geometry together with the nearest street of the same name. This allows a client to see when an address or group of addresses has an incorrect street name or when no geometry was ever delivered.

Inventory Fishbones: Curry County, Oregon

Conclusion

FME has provided a powerful and accessible tool for meeting the needs of clients who have limited access to specialized experts. The FME backbone of this project has enabled Oregon's Geospatial Enterprise Office to begin to maintain a comprehensive master address repository entirely in house. FME provided the right tool to allow the GEO to take control of their processes in a manner that fits their budgets and inherent strengths.

Thank you for your time.

Questions?

Conclusion

FME has provided a powerful and accessible tool for meeting the needs of clients who have limited access to specialized experts. The FME backbone of this project has enabled Oregon's Geospatial Enterprise Office to begin to maintain a comprehensive master address repository entirely in house. FME provided the right tool to allow the GEO to take control of their processes in a manner that fits their budgets and inherent strengths.