Ken Bragg: Batch data processing in FME
-
Upload
gimnv -
Category
Technology
-
view
92 -
download
10
Transcript of Ken Bragg: Batch data processing in FME
Today’s Potions
1. Wildcards
2. Batch Deploy
3. Parent/child Workspaces
4. Parent/child Server Workspaces
Dataset Wildcards
Extended glob syntax:
Symbol Matches
? Any single character
* Any sequence of zero or more characters
[chars] Any single character in chars.
[a-d] Any character between a and d inclusive
{a,b,...} Any of the sub-patterns a, b
/**/ 0 or more subdirectories
Wildcard Bulk Data Processing
Enticements ü Simple to set up
ü Can transform across file boundaries
- Needs memory & time
Wildcard Bulk Data Processing Pitfalls
x Recovery from data errors difficult
x Feature Type vs File vs Format Issues
x No granular log x No ability to
parallelize
Batch Deploy Enticements
ü Simple to set up ü Runs quickly ü Can script via
command line ü Run on demand
Batch Deploy Pitfalls
x Recovery from data errors difficult
x No granular log x Destination dataset
naming can be tricky
Parent/Child Workspace Ingredients
• Parent Workspace: – PathReader – WorkspaceRunner
• Child Workspace:
– FeatureWriter
Parent/Child Workspace Enticements
ü Separate transformation from workflow
ü Generate audit logs ü All authored within
Workbench
Parent/Child Workspace
Pitfalls
x Not all writers can be used concurrently
x Slow to run each child workspace separately
x Recovery from data errors not easy if concurrent runs used
Parent/Child Server
Workspace Ingredients
• Parent Workspace: – PathReader – FMEServerJobSubmitter – FMEServerJobWaiter
• Child Workspace:
– FeatureWriter
Parent/Child Server
Workspace Enticements
ü Separate transformation from workflow
ü Generate audit logs ü All authored within
Workbench ü Make full use of
parallelism = FAST
Parent/Child Server
Workspace Pitfalls
x Not all writers can be used concurrently
x Data needs to be accessible to Server Engines - Consider using Server Data Resources
x Craft your reload/audit plan
Summary ● Many ways to handle bulk data moves
● Choose your potion wisely - each has pluses and minuses
● FME Server is the most robust automation choice