Creating Commercial Data Products with FME
11
Creating Commercial Data Products with FME
-
Upload
safe-software -
Category
Technology
-
view
24 -
download
2
Transcript of Creating Commercial Data Products with FME
Creating Commercial Data Products with FME
Aboutsetld is a next generation “geospatial+” data and service provider
Goals:– Provide cost-effective data products– Help transform data into actionable information– Bring data management and science capability to orgs of any size (GIS / BI as-a-Service)
Qualifications: – Proven industry experience – esri Startups Program / Emerging Business Partner– AWS Activate Partner– Microsoft BizSpark Member (w/ Azure) – Safe Software Solution Provider (1 of 16 in W.H.) – FME Certified Professional (1 of 30 in U.S.)
Offerings:– Foundational and custom data products – Consulting (from data flows to
dissemination innovation)– GIS / BI as-a-Service
Overview
3
• Creating Commercial Data Products?
• Challenges
• Why Choose FME?
• Our Foundation
• Innovation with FME
• Sustainable & Valuable Solutions
• What’s Next?
• Questions
Creating Commercial Data Products?
“I don’t need to do that. How will this talk help me?”
What we hear about FME adoption across industries
– It’s everywhere – FME is easy, fast, powerful and people love it– Business folks are flowing information to maps, charts, reports and alerts like never before
IT, data governance and sustainability planning are often an afterthought
– But, but......business folks are innovating! Get out of the way!– My process gives me and my team what we need. Who cares
what it looks like or how it’s done?
Let’s think harder…how will you share your work and the spirit of innovation with the rest of your organization?
4
Multiple Tools AvailableManual laborScriptingPick a programming languageOpen source GISArcGISOther ETLsFME
Multiple Actions NeededStartingThinking / UnderstandingDecidingScopingStructuringIgnoringParsingReconciliationReformattingCalculationsEnrichmentAggregationUnits checks and validationsMerging and appendingTransferal of attributionSpatial self-tagging and attributionProjections and transformsQA / QCFinishing
Multiple SourcesU.S. federal governmentState governmentsForeign government and agenciesAcademics, colleges and universitiesNon-governmental organizations (NGO)Corporate websites, reports and downloadsAcquired orthophotos, LIDAR, drone imageryEconometrics, IMF, World BankUnited Nations (UN)John’s external HDConservationistsNon-spatialUSB StickPresentationsSensorsRPLSNews
Multiple FormatsShapefiles SpreadsheetsASCIIAccess MDBsWeb Scraping and HarvestsFile Geodatabase (FGDB)Personal Geodatabase (PGDB)SDE feature classes / exportsCAD / BIMGeographic Data Format (GDF)Raster…many flavorsJSONKML
Challenges
We lever standardized internal policy, workflows and FME to process, enrich, QC and deliver!
Why Choose FME?
Effective data integration is more important than ever
– More sources, more data, more formats and greater expectations for fast, actionable information
– FME’s nearly self-documenting visual workbench, support for 400+ formats and QC tools are fantastic
– Basing our business on unsupported open source tools would be difficult
Safe Software track record
– Solving big problems in bunches for many years– The team innovates daily and takes feedback– Excellent technical support community
I’ve been a user since 2005 and have grown with FME
– Began with scheduled spatial data loads – Then, enriched data to drive important maps daily– Moved to other roles where our team completely rebuilt all
geospatial and BI capability at a major oil and gas company – And now……stepping further back in the process to lend a hand
6
Our Foundation
7
setld GoM Data Product
Structure and convention– Structure what you can; may be easier than you realize– Everything in its place, named appropriately– Revise if product evolution dictates (don’t be lazy ++)
Scheduled automation and live monitoring– Gather and prep inputs– Process and enrich– QC reporting / dashboarding
Useful Dissemination– Meaningful atts and cartographic design– Often enhance information to suit tools like AGOL WAB
– What are we doing?
– Input prep and inputs
– Main FME jobs w/ QC outputs
– Staged FTP deliverables (from SDE)– Maps, .lyrs for publish, guides
– Internal product support / dev
Innovation with FME
8
FME RolesQuick prototypingQuick QCOne-off data fixesData maintenanceProduction jobs w/ QC dog legsRevise as we learn moreFile management / migrationHolistic systems QC jobs
Tips & TricksHarvest JSON, etc. from web Use FME to make other .FMWsOutput a .BAT from FME, use inlineStep out to Python, and backFeature writer > .CSV > Feature Reader (sort on read)Use 64-bit for more than a few million rows
Data Enrichment ExamplesNet productionActivity order and summationFormat betterment Classify assetsDeduce statuses Circular referencesCalcs - 3 / 6 / 12 / 24 / 60 monthly prod. avgsRender directional events at TVDGenerate and maintain metadata
Workforce Assist ExamplesTask ListsPrioritization and progressMarketing summaries
QA / QC Made FunAutomatic summary emailsSlack notificationsDashboarding
Sustainable & Valuable Solutions
9
In-progress developments include:
– U.S. Well Header– U.S. Well Production Data– U.S. Pipeline Set– Mexico Data Product– West Africa Data Product
After 1 year of proving and improving this strategy –we’ll continue processing the majority of our data with FME
We hope to see more big data and (perhaps) AI tools built into FME in the coming years!
10
What’s Next?
