Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info...
-
Upload
nguyenkhanh -
Category
Documents
-
view
222 -
download
6
Transcript of Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info...
![Page 1: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/1.jpg)
Automating Data Management with FME
![Page 2: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/2.jpg)
Washington State Counties
![Page 3: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/3.jpg)
Data and UsesDerived Products: Address (points) Parcels (boundaries) Streets (centerlines) Owner Information (tabular)
Geocoding Land base
![Page 4: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/4.jpg)
Challenges 38 sources All in different data schemas Source formats are different
Keeping data up to date
![Page 5: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/5.jpg)
Safe Software• Desktop 2016/17• Server 2016/17
Microsoft• Management Studio 2008• Server 2012
ESRI• Desktop 10.5.1• AGS 10.5.1• Portal 10.5.1
![Page 6: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/6.jpg)
Standardize Data
![Page 7: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/7.jpg)
1 - Map Data2 - Scheduler
FME Workbench(s)
Data Mapping
Attr1Attr2Attr3
Derived Prod Info
Attr1Attr2Attr3
3 – Download DataCounty FTP
FME Workbench(s)Map data to StandardFME Workbench(s)
Check data
WAMAS Data
Service
Derived Product
Attr1Attr2Attr3
4 – Data Validation
![Page 8: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/8.jpg)
FME Workbench – Address Points
1 – Call workbenches for each County
![Page 9: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/9.jpg)
FME Workbench – Address Points
1 – Call workbenches for each County
![Page 10: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/10.jpg)
FME Workbench – Address Points
1 – Call workbenches for each County
![Page 11: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/11.jpg)
FME Workbench – Address Points
2 – Get attribute mapping for county schema
![Page 12: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/12.jpg)
FME Workbench – Address Points
2 – Get attribute mapping for county schema
To next workbench@Value(attrValue)@Value(attrValue)@Value(attrValue)@Value(attrValue)
![Page 13: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/13.jpg)
FME Workbench – Address Points
3 – Parse Address and validate
\\servername\\servername
![Page 14: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/14.jpg)
FME Workbench – Address Points
3 – Parse Address and validate
![Page 15: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/15.jpg)
FME Workbench – Address Points
3 – Parse Address and validate
@Value(attrValue)
![Page 16: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/16.jpg)
FME Workbench – Address Points
3 – Parse Address and validate
![Page 17: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/17.jpg)
FME Workbench – Address Points
4 – Validate and write to Prod
https://state-wamas.wa.gov/webdirectory/pagename.asmx/Getstandardizedaddress?address=@Value(mADDR)&address2=@Value(mADDR2)&city=@Value(mCITY)&zip=@Value(mZIP)&zip4=@Value(mPLUS4)&state=@Value(mSTATE)&company=&Consumer=@Value(mCONSUMER) HTTP/1.1
![Page 18: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/18.jpg)
FME Workbench – HTML Reports
![Page 19: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/19.jpg)
FME Workbench – HTML ReportsTransformers:HTMLReportGeneratorSorterHTMLLayouterWriter:HTMLWriter
![Page 20: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/20.jpg)
FME Workbench – HTML Reports
![Page 21: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/21.jpg)
FME Workbench – HTML Reports
![Page 22: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/22.jpg)
FME Server – Schedule
![Page 23: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/23.jpg)
Self Serve Portal
![Page 24: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/24.jpg)
![Page 25: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/25.jpg)
![Page 26: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/26.jpg)
Parcel and Owner Information
![Page 27: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/27.jpg)
Street Information
![Page 28: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/28.jpg)
Address Point Information
![Page 29: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/29.jpg)
Address Point Information (City)
![Page 30: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/30.jpg)
Data Counts and UsageAddress Points – 2,167,403Parcels – 3,114,394Streets – 731,172Owner Information – 2,101,161PSAP Streets (911) – 566,910
Geocodes – 2.6mBatch Service – 1.5m
Apple – 14.5kAndroid – 10k
![Page 31: Automating Data Management with FME Workbench(s) Data Mapping Attr1 Attr2 Attr3 Derived Prod Info Attr1 Attr2 Attr3. 3 – Download Data. County FTP. FME Workbench(s) Map data to Standard.](https://reader034.fdocuments.in/reader034/viewer/2022051800/5acb84db7f8b9aad468b9d7f/html5/thumbnails/31.jpg)
Issues• FME 2017 – FMEServerJobSubmitter @Value(xxxx)• FTP data moving to OpenData
Next Steps• Create a generic workbench to accept each derived product type• Email notifications
Take away• Schedule Workbench Multiple Workbenches• Store attributes/configuration in database• Create Universal Custom Transformers (re-usable)• Log processes and generate reports