Post on 24-Feb-2016
description
TIMS – ODOT’s Mapping Portal
Garrett Staats, Solutions Architect, ODOT
Melissa Williams, Senior Project Engineer, Stantec
Ian Kidner, GIMS Systems Admin, ODOT
2013 Ohio GIS ConferenceSeptember 11 - 13, 2013 | Columbus Marriott Northwest | Dublin, Ohio
TIMSTransportation Information Mapping System
A tool providing centralized access to ODOT data,
while
minimizing the level of technical expertise required,
while
also providing sophisticated tools for viewing, analyzing, and communicating with ODOT information,
allowing
for greater collaboration between ODOT, ODOT customers, and ODOT business partners
Geodatabase Schema• Primary criteria: data efficiency
• Create feature classes from event tables• Standardize domains, create new domains• Create indexes
• Customized to ODOT source data vs. using ESRI’s Roads & Highways Data Model
• Allow transfer of data without interrupting web services
Geodatabase Schema• Final data structure:
• 40 feature classes, no feature datasets• PRJCT – ODOT project data• PVMT – Pavement Management System (PMS) data• RD_INV – state and county routes, roadway characteristics,
ownership, rail crossings, scenic byways, maintenance authority, truck routes, etc.
• REFER – cities, counties, ODOT districts, railroads, urban areas, etc.
• STRUCT – bridges, culverts, outfalls• TRAFFIC – segments, stations
• 4 data tables• 1 relationship
Geodatabase Schema• Domains:
• Standardized across feature classes• County names to standard ODOT 3-letter abbreviation• Direction to 1 or 2-letter abbreviation• Yes/No/Unknown/NA to 1 or 2-letter abbreviation• Month to short integer
• Created new coded value domains• Railroad operators
• “Grand Trunk Western Railroad Incorporated” vs. “GTW”
• Culvert channel protection• “CONCRETE RIP RAP” vs. “RP”
Geodatabase Schema• Edited schema using X-Ray for ArcCatalog Add-In
• Provides spreadsheet interface to your geodatabase, with separate worksheets for each dataset and domain, creates xml file
FeatureClassName PVMT_PMS_PCR_LNDatasetType FeatureClassDescription Pavement Management System (PMS) PCRFeatureDataset nullTags nullShapeType PolylineFeatureType SimpleAliasName PVMT_PMS_PCR_LNHasM falseHasZ falseSubtypeFieldName nullDefaultSubtype nullDSID 133
FieldsFieldName Type Length Description AliasName DomainName DefaultValue IsNullable Precision Scale Required DomainFixed
PERP_YR SmallInteger 2The perp year to which a record corresponds (COD: WGIS_PMS_PCR_TBL.???) Perpetuation_Year null null true 0 0 null null
TRANS_JURISDICTION_CD String 1
One letter code indicating primary legal authority: S -State, C-County, T- Township, M - Municipal, F - Federal, P - Private (COD: WGIS_PMS_PCR_TBL.TRANS_JURISDICTION_CD) Jurisdiction
dLRS_TRANS_JURISDICTION null false 0 0 null null
COUNTY_CD String 3County Code (COD: WGIS_PMS_PCR_TBL.COUNTY_ABBREV3_CD) County_Code dCounty null false 0 0 null null
TRANS_ROUTE_CD String 2Two letter code denoting the type of route (IR,US, SR, etc) (COD: WGIS_PMS_PCR_TBL.TRANS_ROUTE_CD) Route_System
dLRS_TRANS_ROUTE null false 0 0 null null
TRANS_ROUTE_ID String 5Assigned five (5) digit route number (COD: WGIS_PMS_PCR_TBL.TRANS_ROUTE_ID) Route_Number null null false 0 0 null null
TRANS_ID_EXTENSION_CD String 1
Extension of Trans_Route_ID for State and local system. Used in conjunction with apha-numeric route numbers: CR 10A (COD: WGIS_PMS_PCR_TBL.TRANS_ID_EXTENSION_CD)
Route_Number_Extension
dLRS_TRANS_ID_EXTENSION null false 0 0 null null
TRANS_ID_DESCRIPTION_CD String 1
One character field which defines Trans_Route_ID: P - Proposed, A - Alternate, etc. * is default (COD: WGIS_PMS_PCR_TBL.TRANS_ROUTE_DESCRIPTION_CD) Route_Description
dLRS_TRANS_ID_DESCRIPTION null false 0 0 null null
TRANS_BI_DIRECTIONAL_CD String 1
Cardinal/non-cardinal identifier (COD: WGIS_PMS_PCR_TBL.TRANS_BI_DIRECTIONAL_CD) Cardinal_Direction
dLRS_TRANS_BI_DIRECTIONAL null false 0 0 null null
NLF_ID String 14 Network Linear Feature ID (COD: WGIS_PMS_PCR_TBL.NLF_ID)Network_Linear_Feature_ID null null false 0 0 null null
Geodatabase SchemaDomainName dBldgTypeDomainType CodedValueFieldType StringMergePolicy DefaultValueSplitPolicy DefaultValueDescription Building TypeOwner null
Coded ValuesCode NameDQ Dairy QueenDQTIC Dairy Queen with Tourist Information CenterDG District GarageDZUNG Double Zung BuildingEDQ Extended Dairy QueenEDQTIC Extended Dairy Queen with Tourist Information CenterFCSS Fabric Covered Salt StorageFES Full Earth ShelterFESTIC Full Earth Shelter with Tourist Information CenterM MansardMTIC Mansard Tourist Information CenterPES Partial Earth ShelterP Primitive Rest AreaPH Primitive Handicap Rest AreaTM Taj Mahal Rest AreaMSB-1 PrivyRT Radio TowerSD Salt DomeWF Wooden Flush Rest AreaZNG Zung BuildingZNGTIC Zung Building with Tourist Information CenterNONE NONE
Geodatabase Schema• Used ArcGIS Diagrammer to create .htm schema report
Develop ETLs• ETL = Extract, Transform, Load• Used Model Builder to create 10 ETLs• Transfer source data from Oracle into SDE geodatabase
• Requires Data Interoperability extension• Needs to be performed on a routine basis, so models converted
to geoprocessing services that can be scripted• Need to create routes first, then create feature classes from
event tables• Need to use versioned editing to minimize interruption to data
Develop ETLs• Basic outline of ETL:
• 1) Create new version in target dataset based on timestamp• 2) Delete all features from target dataset• 3) Copy source data to scratch.gdb• 4) Perform calculations on fields in scratch.gdb
• Lower to upper case• Strip blanks• Parse field (e.g. address)• Convert to coded domain values• Replace values (e.g. 0/1 to N/Y)
• 5) Make route event layer• 6) Append into target dataset• 7) Reconcile and post to default version• 8) Change version to default, delete temporary version
Develop ETLs• Problem:
• Could not view attributes for Oracle spatial data table views• Solution:
• Perform Feature Class to Feature Class to scratch.gdb even if no field calculations required, then define projection before appending to final dataset.
Develop ETLs• Problem:
• Logpoint values appeared incorrectly• ODOT source data - 1.27 (float)• SQL Server SDE target geodatabase - 1.26999998 (field defined as
float with precision/scale of 0/0 in xml schema)• Explanation:
• SDE does not recognize the ArcGIS float data type, so defaults to double type with maximum precision/scale of 38/8.
• Conversion of float to double yielded extra unwanted precision.• Solution:
• Define field with specific precision/scale of 8/3 in xml schema.• Convert float to string in initial Feature Class to Feature Class
step; then append string field to double target field.
Develop ETLs• Problem:
• ChangeVersion tool doesn’t work with table views as input (documented bug NIM079735 in ArcGIS 10.0).
• Solution: • Created custom Python script to create connection file on the fly
to change versions.
Develop ETLs• Problem:
• Error when deleting points in an origin feature class participating in a relationship.
• Solution: • Changed foreign key to be nullable to allow for deletion of
records.
System Architecture
System Design Goals
System Architecture
System Architecture
• ETLs deployed to SDE Server as GP tools• Execute as scheduled python script tasks• Python script emails user for failure or completion• Administrative tasks (compressing, update statistics etc)
have been scripted out as well
System Architecture• Problem
• Geoprocessing tools “timing out” • Explanation
• TCP sessions terminated after prolonged idle time• Solution
• Ensure GP tool is not idle more than 30 minutes
System Architecture• Problem
• Slow performance from initial Oracle geodatabase • Explanation
• ESRI was able to reproduce similar performance • Oracle interface was being overhauled in subsequent releases
of ArcGIS, thus, the issue would not be addressed 10.0• Solution
• Initial testing in SQL Server yielded substantially better performance
System Architecture• Problem
• Several fields from Oracle tables not being read properly in ETLs or ArcCatalog
• Solution• Changed the OLE DB Provider from Oracle Provider to
Microsoft OLE Provider for Oracle
System Architecture• Problem
• Geodatabase field definitions differed from actual data• NLFID fields in the Geodatabase defined as 14 character text
(NLFID is a common 14-character text value used in the department)• ETL was failing due to bad data – NLFID values in one data were
returning 15&16 character values causing the failure• Solution
• Alter Geodatabase schema to reflect the actual data (even if technically incorrect)
System Architecture• Problem
• ETLs with ODPS Crash data were failing or taking a long time to complete
• Memory usage would increase, then the service would crash• Explanation
• The ‘delete feature’ operation fails with large datasets• Would consume close to 3.4gb of RAM then crash
• Solution• The ETL originally deleted features from the versioned table.
(10.0 has no truncate function)• Migrated ETL to 10.1 and use a manual process to truncate and
update data