Post on 18-Jun-2015
description
CONNECT. TRANSFORM. AUTOMATE.
Maximizing Data for Reverse 911 Amanda Graf Senior Project Manager California CAD Solutions, Inc.
Situation
! Local municipality needed to leverage their existing GIS system to enable reverse 911 notification of emergency events to mobile phones
! GIS must interface with existing TENS (Telephone Emergency Notification System)
! County Emergency Dispatch system had a match rate of less than 65% for mobile numbers
! County system could not interface with City GIS
CONNECT. TRANSFORM. AUTOMATE.
Complicating Factors
! Non-repeatable County process meant the updates were excruciatingly tedious and prone to error
! No documentation of the County geocoding process led to no confidence in the data results
! Erratic updates to base data used for geocoding ! Jurisdictional battles between City and County
CONNECT. TRANSFORM. AUTOMATE.
Major Factors Impeding Success
! Multiple Data Vendors with radically different data structures and update methodologies ! AT&T – Monthly updates with a complete listing of
all phone records ! Verizon – Weekly updates with incremental
changes from the prior update delivery ! Inability to get AT&T and Verizon to make
changes to data anomalies (errors) ! Multiple sources of Address information
CONNECT. TRANSFORM. AUTOMATE.
Plan
! Document address data sources and determine hierarchy of processing
! Normalize address notations among all the data sources used
! Process and normalize AT&T data ! Process and normalize Verizon data ! Deliver Geocoded dataset themed by source ! Deliver List of unmatched addresses ! Deliver documentation of entire process
CONNECT. TRANSFORM. AUTOMATE.
Process / Approach
! Granular approach to the problem was the most effective ! Multiple FME routines
! 1 - Process AT&T Data ! 2 – Process Verizon Data ! 3 – Combine datasets into single datastore ! 4 – Geocode the data
! Scripted batch files to automate processing
CONNECT. TRANSFORM. AUTOMATE.
1 – Process AT&T Data
! AT&T Data ! Straight forward CSV file ! “Street Name” included both street name and
street type in a single field ! Liberal use of SubstringExtractors,
AttributeTrimmers, and Testers used to break the information out into separate fields
CONNECT. TRANSFORM. AUTOMATE.
2 – Process Verizon Data
! Verizon Data ! Fixed Length format requiring use of
SubstringExtractors ! Critically important to process the data sequentially
since a single number can be entered more than once in any particular update file
! Determine if Insert, Update, or Delete is the appropriate action for each record
CONNECT. TRANSFORM. AUTOMATE.
Insert, Update or Delete?
CONNECT. TRANSFORM. AUTOMATE.
3 - Merge Datasets
! Massive Data normalization process ! AT&T, Verizon, County Assessor, City Public Works
! Each organization has their own way of designating (and spelling) addresses
! 1st or First? ! AV or AVE or Ave.? ! Mc Clay or McClay? (Use the MC Hammer) ! Green Oak PL should be Green Oak DR ! Misspellings Agencies won’t fix
CONNECT. TRANSFORM. AUTOMATE.
Normalize Data
CONNECT. TRANSFORM. AUTOMATE.
FME Advantages
! Update FME routine with known exceptions and the work only needs to be done once
! Quick and easy to incorporate new exceptions as they are found
! Original source data is unaltered thereby enabling a viable audit trail of information
CONNECT. TRANSFORM. AUTOMATE.
4 - Geocoding
! 7 data sources used in geocoding process (sources noted in order of priority)
! County Assessor Data ! City Situs Address Data ! County Assessor Mobile Home Data ! City Situs Mobile Home Address Data ! Street Centerline (Address Range Matching) ! Lat/Lon Lookup Table ! Known Invalid Addresses
CONNECT. TRANSFORM. AUTOMATE.
Verification Process
! USPS.com ! Matched with LatLon Lookup table.
The lookup table was created by looking up the addresses on www.batchgeocode.com/lookup. All addresses are verified as valid addresses against USPS.com.
! Loop Back through FME Routine 3 & 4 with edits and additional exceptions
CONNECT. TRANSFORM. AUTOMATE.
Unmatched Examples
! 201 FOREIGN EXCHANGE ! 0 AFB ! 1 VOIP CALLER ! T-MOBILE@HOME SERVICE
CONNECT. TRANSFORM. AUTOMATE.
Batch Processing
! FME routines can be run from a batch file ! By using published parameters the FME routines
stay the same even as the source dataset names change each quarter
! Use a template to create a .bat file for processing the data for the current quarter
! Input names of source files (published parameters)
! Run
CONNECT. TRANSFORM. AUTOMATE.
Display Themed Data in Map
CONNECT. TRANSFORM. AUTOMATE.
Pull Reports and Notify Residents
CONNECT. TRANSFORM. AUTOMATE.
Results
FME Saved the Day!! ! RESULTS!!! 99.6% of all records were matched
(100% of all records that had valid addresses were matched)
! Fast, easy integration with the existing City GIS site
! Documented, traceable results of worked performed
CONNECT. TRANSFORM. AUTOMATE.
Thank You!
! Questions?
! For more information: ! Amanda Graf – amanda.graf@calcad.com
! California CAD Solutions, Inc. www.calcad.com
CONNECT. TRANSFORM. AUTOMATE.