Large GIS Data Reprojection With FME

WorkbenchUTM Zone Fanout Solution

Robert L HegartyBentley Systems Inc.

Hello!Currently a Sr Consultant employed by Bentley Systems inc. since December 2007, Mr Hegarty has worked with GIS for the Telecom Industry since 1995, and an FME user since 1998. Prior to coming to Bentley, Mr Hegarty spent 13 years at a major MSO within the OSP Engineering Design department. He was the recipient of the 2006 BE Inspired Award for the Geospatial Communications category. He has been a SCTE member since 2006, and is SCTE certified Broadband Transport Specialist.

The presentation material has been chosen due to it’s large geographic scope and content.

Geospatial Alignment Issue

Cable customer had an unfortunate geospatial alignment issue due to an initial project configuration error. Although the country of Mexico spans 6 UTM zones, the project was initially created entirely in one 6

Degree UTM column. This introduced cylindrical and misaligned coverages when viewed across all zones. At the time of the data analysis (Jan 2015), the customer had over 3.2 million addresses documented within

the system along with the associated outside plant data.

Project Goal The goal of this project was to fan-out and properly align their data using FME Workbench:• Assign single Geographic Coordinate System (GCS)

to remove the current geographical ambiguities within their Oracle Spatial data.

• Eliminate the imminent issue of overlapping data from the adjacent UTM Zones.

How Did We Solve This?

Let’s start with the first set of slides.

The Plan1. Scrub Data: This will benefit customer by ensuring a more accurate re-projection in the shortest possible time frame.2. Acquire new database server: This will benefit customer by significantly increased database performance, reduced lockups, reduced contention issues, reduce incidents caused by low throughput capacity; thus resulting in lower Engineering costs. 3. Create highly accurate Spatial Matrix: This will benefit customer by ensuring the most accurate re-projection possible.4. Test the proposed configuration: This will benefit customer by validating the approach of storing geometry in Oracle utilizing a GCS, while designing in the proper projection for any given location. This also provides plenty of opportunity to update, refine, and improve the project .xml schema.5. Database should be written from old server to new server with FME Workbench workspace re-projection occurring in-between: This will benefit customer by ensuring maximum availability of the data during the time periods leading up to the final ‘Go’. 6. Quality control (data improvement) is a reiterative process: This will benefit customer by ensuring the most accurate re-projection possible.7. Implement plan.

ScrubData AnalyzeData

TestMethod

Transform

VerifyResults

Scope

• 52 / 104 Oracle Spatial Tables Affected

• 26.5 Million Oracle Records Total

• 60 Concurrent Users• 3.5 GB Oracle Data• Top Tables:

TableName RecordCount"BASETEXT" 1,088,997

"ADDRESSLEADERLINE" 2,074,873

"LOTLINE" 2,982,660

"ADDRESS" 3,222,227

Analysis

• Mexico UTM Zones• Addresses by UTM

Zone• Current Geo Locations• Address Quality

Mexico UTM Zones

UTM Zone Begining Latitude (West)

Ending Latitude Example Areas Included

UTM Zona 11N 120°W (-120) 114°W Baja, Tijuana

UTM Zona 12N 114°W 108°W Sonora

UTM Zona 13N 108°W 102°W Guadalajara, Aguascalientes

UTM Zona 14N 102°W 96°W DF, Monterrey , Cuernavaca

UTM Zona 15N 96°W 90°W Tabasco

UTM Zona 16N 90°W 84°W Cancun

Mexico UTM Zones

Mexico UTM Zone DistortionWorst Case Depiction

Addresses in the UTM Zone

City State UTMZoneENSENADA BC 11MEXICALI BC 11ROSARITO BC 11TIJUANA BC 11TIJUANA TJ 11AGUASCALIENTES AG 13ALDAMA CH 13CALVILLO AG 13CD.JUAREZ CH 13CHIHUAHUA CH 13DELICIAS CH 13JESUSMARIA AG 13ACAPULCO GR 14ATLATLAHUCAN MO 14AYALA MO 14CHILPANCINGO GR 14CHIMALHUACAN EM 14CUAUTLA MO 14CUERNAVACA MO 14DISTRITOFEDERAL DF 14EMILIANOZAPATA MO 14ESTADODEMEXICO EM 14HUITZILAC MO 14JIUTEPEC MO 14MEXICO DF 14OAXACA OC 14SANVICENTE EM 14TEMIXCO MO 14TEPEJI HG 14TEPOZTLAN MO 14TULA HG 14VERACRUZ VZ 14XICO EM 14XOCHITEPEC MO 14YAUTEPEC MO 14CAMPECHE CA 15CAMPECHE CC 15CAMPECHE QR 15COATZACOALCOS VZ 15MINATITLAN VZ 15CANCUN QR 16CHETUMAL QR 16COZUMEL QR 16MERIDA MA 16MERIDA YN 16PLAYADELCARMEN QR 16

Where Do They Fall?Mexico City Zone 14 Aguascalientes Zone 13

Where Do They Fall?Mexico City Zone 14 Tijuana Zone 11

Data Quality

Garbage In = ) , * & = - + < / (**)> <> or != or ~= >= <= <>

Data Quality

• Address Quality• Spatially impossible

Coordinates• Mis Matched

Geometries• Corrupted Records• Corrupted Geometries• Referential Integrity

Errors

Address Data Quality

Address Data Quality

More Data QualityMis Matched Geometryi.e. gtype = 2007 vs 2003; select t.ogc_geometry.sdo_gtype from powersupplyboundary t where t.ogc_geometry.sdo_gtype <>2003;

Referential Integrity Errors

TableName Qty

Addressleaderline 7165

Boundaryname 1439

Polepedusrfield 565

Powersupplyblock 26

Roadname 598

Stranddistance 5592

Tapendoflineblock 152

TableName Qty

Strand 17

Coaxcable 15

Nodeboundary 3

Fiberspan 7

Road 9

Lotline 77

Waterline 1

Addressleaderline 8

Powersupplyboundary 3

i.e. select XFM_ID from addressleaderline l where ADDRESSLINK not in(select a.xfm_id from address a);

More Data Quality

CorruptedGeometry

CorruptedDatabaseRecords. . importing table "FIBERSPLICEENCLOSURE" 16979 rows importedIMP-00017: following statement failed with ORACLE error 6550:<snip>IMP-00003: ORACLE error 6550 encounteredORA-06550: line 1, column 2204:PLS-00103: Encountered the symbol "R10N5009" when expecting one of the following:

) , * & = - + < / > at in is mod remainder not rem =><an exponent (**)> <> or != or ~= >= <= <> and or like like2like4 likec between || multiset member submultiset

23TableswithOutofBoundsDataGeometryexistsinthedatabaseatcoordinatesthatareimpossiblyfarawayfromMexico.

Testing Testing the UTM Fanout using the Address Table.

Buffering Addresses For Analysis And POC With UTM Expansion

Address Geo Located

Using FME for Data Preparation

Logging the Out of Bounds Records

Logging the Out of Bounds Records

The Secret Sauce

Spatial Matrix

Spatial Matrix Creation PIP

The Final Workspace For Processing All Populated Geometry Tables

What’s Inside

The Magic Box?

The Magic Box

Summary• Geographical Location Issue Solution

• Data Quality Improvement Processes• Data Scrubbing• Analyze Existing Geographic Locations • Testing – Methodology Verification• Transform Geographic Data - 52 Geometry Tables

• FME Workbench Enabling Technology (Transformers Used):• Clipper• AttributeCreator, AttributeValueMapper• Bufferer, DonutHoleExtractor• AttributeFilter, Sorter• Reprojector• GeometryFilter, PointOnAreaOverlayer

• Final Verification –26M Records Spatially Aligned with 99.999% Data Retention.

Thank you!

Skype: RLHegartyTwitter: @RL_GIS