Seattle bI Meetup BI & ETL @ Big Fish

34
SEATTLE BI MEETUP BI & ETL @ BIG FISH April 2 nd , 2014 Emre Motan

description

Seattle bI Meetup BI & ETL @ Big Fish. April 2 nd , 2014 Emre Motan. About the Speaker. Emre Motan BI Engineer, Big Fish Been in Seattle 1.5 years, previously in Chicago Involved in BI community Chicago SQL BI PASS chapter UW BI Certificate Program BI Over Beers TDWI - PowerPoint PPT Presentation

Transcript of Seattle bI Meetup BI & ETL @ Big Fish

Page 1: Seattle  bI Meetup BI & ETL @ Big Fish

SEATTLE BI MEETUPBI & ETL @ BIG FISH

April 2nd, 2014Emre Motan

Page 2: Seattle  bI Meetup BI & ETL @ Big Fish

About the Speaker Emre Motan BI Engineer, Big Fish Been in Seattle 1.5 years, previously in

Chicago Involved in BI community

Chicago SQL BI PASS chapter UW BI Certificate Program BI Over Beers TDWI

Random: Basketball, co-rec sports, greyhounds

Page 3: Seattle  bI Meetup BI & ETL @ Big Fish

About the Seattle BI Meetup Started in 2012, I took over after period

of inactivity Meetups will be monthly Primary goal is to educate

Topics will be wide-ranging but more technical

Networking is encouraged Speakers will be people who use

technologies

Page 4: Seattle  bI Meetup BI & ETL @ Big Fish

About BI Meetup (cont.) Looking to develop relationships with

willing speakers, venues, and sponsors Desire is to have meetings in different

venues each month As part of hosting, it would be nice to

“sponsor” with food & drink

Page 5: Seattle  bI Meetup BI & ETL @ Big Fish

About Big Fish World’s largest producer

of casual games Core business used to

be PC-based “Hidden Object” games

Now pushing deep into Mobile space with titles such as Big Fish Casino and Fairway Solitaire

Page 6: Seattle  bI Meetup BI & ETL @ Big Fish

Today’s Story BI/DW Implementation ETL Framework ETL Development BIML

BIDS Helper Mist BIML IDE w/ Hadron BIML Compiler

Summary

Page 7: Seattle  bI Meetup BI & ETL @ Big Fish

BI INFRASTRUCTURE

April 2nd, 2014Emre Motan

Page 8: Seattle  bI Meetup BI & ETL @ Big Fish

BI Infrastructure

Page 9: Seattle  bI Meetup BI & ETL @ Big Fish

What do we want from our ETL? Minimize manual coding errors Minimize time spent on boilerplate Support agile software development

practices Use software development best

practices Pair / collaborate Source control / diff changes Auto-generate / automate

Reduce effort spent on operations and support

Page 10: Seattle  bI Meetup BI & ETL @ Big Fish

ETL FRAMEWORK

April 2nd, 2014Emre Motan

Page 11: Seattle  bI Meetup BI & ETL @ Big Fish

ETL Framework Overview Cycles (subject areas, master package)

Jobs (restart-able units of work, package)

Steps (logic units, Data Flow / Execute SQL / …)

Page 12: Seattle  bI Meetup BI & ETL @ Big Fish

12

Job AStep 1Step 2Step 3

Job BStep 4Step 5Step 6

Job CStep 7Step 8Step 9

Job DStep 10Step 11Step 12

Job EStep 13Step 14Step 15

CycleLog Start

Log Success

Do Work Log Fail

Page 13: Seattle  bI Meetup BI & ETL @ Big Fish

13

1st RunJob AStep 1Step 2Step 3

Job BStep 4Step 5Step 6

Job CStep 7Step 8Step 9

Job DStep 10Step 11Step 12

Job EStep 13Step 14Step 15

SuccessFailureDidn’t Run

Page 14: Seattle  bI Meetup BI & ETL @ Big Fish

14

2nd RunJob AStep 1Step 2Step 3

Job BStep 4Step 5Step 6

Job CStep 7Step 8Step 9

Job DStep 10Step 11Step 12

Job EStep 13Step 14Step 15

Completed Successfullylast run; Skip Next Time

Rerun on Failure

Choose to Retry because of a logicaldependency

Choose to SkipRun because it didnot run last time

Page 15: Seattle  bI Meetup BI & ETL @ Big Fish

Example Job Flow

Data WarehouseStaging ODS Reporting

LayerStg_sales

Stg_customers

Stg_products

Stg_payment_methods

ods_sales

ods_customers

ods_products

ods_payment_methods

fact_sales

dim_customers

dim_products

dim_payment_methods

Source Systems

Ecommerce

ELT Server

Extract/Load

Page 16: Seattle  bI Meetup BI & ETL @ Big Fish

ETL Framework Summary Framework is injected at compile time

Uses SQL Server 2008 R2 via stored procedures Logical units automatically logged Event handlers automatically added Metadata based alterations for flow control (skip/restart)

Metadata based balances and validation scripts to detect warning/error conditions

Variables and values stored for use in ETL Metadata based run-time Alterations for flow control

(e.g. skip job, skip step) Custom tools to administer ETL infrastructure and

metadata

Page 17: Seattle  bI Meetup BI & ETL @ Big Fish

BIML ECOSYSTEM

April 2nd, 2014Emre Motan

Page 18: Seattle  bI Meetup BI & ETL @ Big Fish

Language and Compiler BIML (BI Markup Language)

Lightweight XML dialect Represents SQL Server BI Stack objects

(SSIS, SSAS, SQL Server) Works like ASP.NET / PHP (combines

declarative & imperative language) Hadron

Compiles BIML to SQL Server BI Stack artifacts

Called via MSBuild, Mist, …

Page 19: Seattle  bI Meetup BI & ETL @ Big Fish

Tools BIDS Helper

Free, open-source extension to BIDS Code in BIML, then generate SSIS Subset of functionality

Mist IDE Graphical & Text Based Editors Transformers Extensions

Page 20: Seattle  bI Meetup BI & ETL @ Big Fish

ETL DEVELOPMENT

April 2nd, 2014Emre Motan

Page 21: Seattle  bI Meetup BI & ETL @ Big Fish

Big Fish BI Engineering We integrate a wide variety of data

sources We don’t develop in BIDS/SSIS We code in BIML, compile in Mist or via

Hadron directly

Page 22: Seattle  bI Meetup BI & ETL @ Big Fish

BI Engineering ETL Development Flow

Develop BIML locally, committed to SVN Generate most of the code besides

business logic Run code validations before / during

compile Compile BIML during development or on

deployment to ELT boxes Produce SSIS packages Handle pushing to target environment

Kick off Cycles using Job Scheduler or DtExec

Page 23: Seattle  bI Meetup BI & ETL @ Big Fish

DEMOS

April 2nd, 2014Emre Motan

Page 24: Seattle  bI Meetup BI & ETL @ Big Fish

Demo: Simple SSIS Package Demo of BIMLScript1.biml

Show BIDS environment Show BIML Generate SSIS Run SSIS

Page 25: Seattle  bI Meetup BI & ETL @ Big Fish

Demo: Programmatic BIML Demo of BIMLScript2.biml

Introduce .NET addition to BIML script Describe what we’re doing with getting

tables from DB Describe how we’ll loop over each table,

and then each column of table, to generate insert commands

Generate SSIS Run SSIS

Page 26: Seattle  bI Meetup BI & ETL @ Big Fish

Demo: Mist Visual Designer Show audience visual designer of one

job Select elements to see visual designer We don’t use visual designer very often

since most code is auto-generated now and we have established patterns

Page 27: Seattle  bI Meetup BI & ETL @ Big Fish

Demo: Mist Project Show Mist environment with sample

cycle Show cycle file with one job Show job file Show metadata for sample table (source,

ODS) Show Extensions

Page 28: Seattle  bI Meetup BI & ETL @ Big Fish

Demo: Auto-generating ETL One substantial accelerator of our work is auto-

generating ETL for new extracts, loads, and processing

BIML representation of table Columns, business keys, primary keys, data types Annotations like ETL pattern required (full load,

incremental new, incremental new/updated) Only need to code transformation logic, all

boilerplate is auto-generated BimlScript to autogenerate boilerplate ETL code

Page 29: Seattle  bI Meetup BI & ETL @ Big Fish

Demo: NZ SSIS Console Cycle/Job/Step Status Alterations Variables Deploy / Execute Cycles

Page 30: Seattle  bI Meetup BI & ETL @ Big Fish

SUMMARY

April 2nd, 2014Emre Motan

Page 31: Seattle  bI Meetup BI & ETL @ Big Fish

Why did Big Fish choose BIML? Non-standard technology needs

(extensibility) Ease of developing and maintaining ETL to

leave more time for high business value work Plenty of people with SSIS experience in

Seattle Cost effective Organization already supported SQL Server

and SSIS Happy developers

Page 32: Seattle  bI Meetup BI & ETL @ Big Fish

Q & A

April 2nd, 2014Emre Motan

Page 33: Seattle  bI Meetup BI & ETL @ Big Fish

THANK YOU

April 2nd, 2014Emre Motan

Page 34: Seattle  bI Meetup BI & ETL @ Big Fish

Resources BIDSHelper.codeplex.com

Varigence.com

BimlScript.com

Biml Tutorials by Andy Leonard