Time Machines and Attribute Alchemy
-
Upload
safe-software -
Category
Technology
-
view
22 -
download
0
Transcript of Time Machines and Attribute Alchemy
![Page 1: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/1.jpg)
Time Machines and Attribute
Alchemy
![Page 2: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/2.jpg)
Date & Time Attribute Manipulation
![Page 3: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/3.jpg)
1. Dates and times
![Page 4: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/4.jpg)
RCMP E Division Heidi Lee | Robert Shultz Goal: Load GPS records into ArcGIS. Problems: ➔ Inconsistent date formats ➔ Time zones ➔ Daylight saving
Dates and times are complicated.
![Page 5: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/5.jpg)
Formatting?
● YYMMDD, HHMMSS, UTC
● Jun 2016
● ‘on Saturday, Jan 9th 2016, 01:00 am’ & ‘+0530’
● 2016-12-07 12:20:07.785403-05
● 20160313020000.000 (March 13 - Daylight Saving)
● <d v="2016-12-13T00:00:00"/> (Excel)
● YYYY-MM-DD hh:mm:ss[.nnnnnnn] (SQL Server ‘datetime2’ value)
Calculations?
● Date2-Date1 = How many days?
![Page 6: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/6.jpg)
Improved FME Date Functions
![Page 7: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/7.jpg)
Read Transform Parse Process Format Write
![Page 8: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/8.jpg)
Read Transform Parse Process Format Write
![Page 9: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/9.jpg)
Over to FME
![Page 10: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/10.jpg)
Time Zones
FME 2017 UTC Offset e.g. -08:00
FME 2018 IANA Time Zone
e.g. America/Vancouver
![Page 11: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/11.jpg)
Attribute Alchemy
![Page 12: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/12.jpg)
Last Year
Attribute Management Quality Control
![Page 13: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/13.jpg)
Southern Company Jeff DeWitt HOK Inc. David Baldacchino
Goals:
➔ Test for patterns in attribute values
➔ Extract substrings from attribute values
➔ Validate strings
2. Finding patterns NGI Belgium Jan Beyen RCMP E Div. Heidi Lee
![Page 14: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/14.jpg)
Southern Company Problem: Attribute value cleanup
- MONTANA * or Sales/Other (1) HOK Inc. Problem: Extract Sheet numbers from file names
- MyProject - Sheet - A512 - PARTITION TYPES & … - G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
NGI Problem: Validate address strings
- Rue Achille Masset 52A
![Page 15: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/15.jpg)
Sheet number extraction MyProject - Sheet - A512 - PARTITION TYPES & … G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ... ^[\S\s]*?[-]?[ ]?([A-Z0-9]{1,5})[ ]+[-]+[\S\s]*$ Address validation Rue Achille Masset 52A ^((([a-zA-Z]+) )+)([0-9]+)([a-zA-Z]*)$
Wave the wand of regex
![Page 16: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/16.jpg)
Regular Expression Editor Making regex easier since FME 2016
![Page 17: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/17.jpg)
Alternatively: String functions
![Page 18: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/18.jpg)
Regex vs. String Functions Example
Code ABD3705337067 Regular Expression: ([A-Z]{3})([0-9]+) String Functions:
Attribute String Function
alpha @Left(@Value(Code),3)
beta @Substring(@Value(Code),3,-1)
![Page 19: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/19.jpg)
TRC Inc. Peter Veensta
Goals: ➔ Compare current and previous
Excel rows. ➔ Sum attribute values with the
previous row.
3. Time-travelling attributes.
FPInnovations Matt Kurowski
![Page 20: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/20.jpg)
Past, Present & Future Attributes
feature[-1].measure measure
feature[+1].measure
![Page 21: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/21.jpg)
Attribute Aggregation Challenge Rule: If T is 3 or less, aggregate this row with the row above.
![Page 22: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/22.jpg)
Attribute Aggregation Challenge
![Page 23: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/23.jpg)
Summary 1. DateTime transformers and Text Editor
functions help with: ○ Date/time formatting ○ Calculations ○ Time zones
2. Regex and string functions help with
patterns.
3. Work with current and previous attribute values in the AttributeManager.
![Page 24: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/24.jpg)
Story Time with the
AttributeManager
![Page 25: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/25.jpg)
Automation: Schemas & Data Enhancement
for Dry Rot Insurance
Sigbjørn Tillerli Herstad
![Page 26: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/26.jpg)
Goal: To automate and enhance the quality of daily/weekly imports of customer data to a common schema.
![Page 27: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/27.jpg)
Problem: Source Data Mayhem ● Codan Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates
● DNB Forsikring – CSV – 1 file – 5 attributes
● Eika Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates
● Enter Forsikring – CSV – 2 files – 14 attributes
● Frende Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates
● Gjensidige – CSV– 2 file – 20 attributes – missing AddressID, coordinates
● IF Skadeforsikring – XLS – 1 file – 10 attributes – missing owner
● Jernbanepersonalets Forsikring – CSV – 3 files – 5 attributes
![Page 28: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/28.jpg)
Achieving Automation • Define schema to use for import to database (52+ attributes, one
feature type) • AttributeFilter: Separate data streams for each company • FeatureReader: Reads the actual data • AttributeManager: Convert to common schema • Can be run at scheduled intervals when a file arrives
![Page 29: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/29.jpg)
Achieving Quality Custom transformers to improve and split data, reused on multiple files:
• SplittFornavnEtternavn: Separate firstname and lastname into 2 different attributes.
• SplitTelefonOgMobiltlf: Decide if number is a cellular or landline and create 2 different attributes.
• SplitStreetNameNumberLetter: You have one attribute in which contains streetname, housenumber, houseletter. Output is 3 different attributes.
![Page 30: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/30.jpg)
Achieving Quality Use existing services and databases to look up and verify values:
• CheckAIDToOwner: Checks if this is the official owner of that property.
• NorkartGeocoder: API to check the validity of an address, handles misspellings, validates postal number, municipality number, etc. Fresh data every day!
![Page 31: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/31.jpg)
Achieving a Common Schema Translate each customer’s
schema to the common schema:
AttributeManager
One separate
AttributeManager for each company.
![Page 32: Time Machines and Attribute Alchemy](https://reader034.fdocuments.in/reader034/viewer/2022042907/58e7e45c1a28abbc0b8b4a77/html5/thumbnails/32.jpg)
Questions?
Tutorial: fme.ly/datetime
AttributeManager documentation