Sponsored & Brought to you by

A Lap around Azure Data Factory

Martin Abbott

http://www.twitter.com/martinabbott

https://au.linkedin.com/in/mjabbott

A Lap around Azure Data Factory

Martin Abbott@martinabbott

About me10+ years experienceIntegration, messaging and cloud personOrganiser of Perth Microsoft Cloud User GroupMember of GlobalAzure Bootcamp admin teamBizTalk developer and architectIdentity management maven IoT enthusiastSoon to be Australian Citizen

AgendaOverviewData movementData transformationDevelopmentMonitoringDemonstrationGeneral information

Overview of an Azure Data Factory

Overview of an Azure Data Factory• Cloud based data integration• Orchestration and transformation• Automation• Large volumes of data• Part of Cortana Analytics Suite Information Management• Fully managed service, scalable, reliable

Anatomy of an Azure Data FactoryAn Azure Data Factory is made up of:

Linked services• Represents either

• a data store• File system• On-premises SQL Server• Azure storage• Azure DocumentDB• Azure Data Lake Store• etc.

• a compute resource• HDInsight (own or on demand)• Azure Machine Learning Endpoint• Azure Batch• Azure SQL Database• Azure Data Lake Analytics

Data sets• Named references to data• Used for both input and output• Identifies structure• Files, tables, folders, documents• Internal or external• Use SliceStart and SliceEnd

system variables to create distinct slices on output data sets, e.g., unique folder based on date

Activities• Define actions to perform on data• Zero or more input data sets• One or more output data sets• Unit of orchestration of a pipeline• Activities for

• data movement• data transformation• data analysis

• Use WindowStart and WindowEnd system variables to select relevant data using a tumbling window

Pipelines• Logical grouping of activities• Provides a unit of work that

performs a task• Can set active period to run

in the past to back fill data slices• Back filling can be

performed in parallel

Scheduling• Data sets have an availability

"availability": { "frequency": "Hour", "interval": 1 }

• Activities have a schedule (tumbling window)"scheduler": { "frequency": "Hour", "interval": 1 }

• Pipelines have an active period"start": "2015-01-01T08:00:00Z" "end": "2015-01-01T11:00:00Z“ OR“end” = “start” + 48 hours if not specified OR“end”: “9999-09-09” for indefinite

Data Lineage / Dependencies• How does Azure Data Factory know how to link

Pipelines?• Uses Input and Output data sets • On the Diagram in portal, can toggle data lineage on

and off• external required (and externalData policy optional)

for data sets created outside Azure Data Factory • How does Azure Data Factory know how to link

data sets that have different schedules?• Uses startTime, endTime and dependency model

Functions• Rich set of functions to• Specify data selection queries• Specify input data set dependencies

• [startTime, endTime] – data set slice• [f(startTime, endTime), g(startTime, endTime)] – dependency

period• Use system variables as parameters

• Functions for text formatting and date/time selection• Text.Format('{0:yyyy}',WindowStart)• Date.AddDays(SliceStart, -7 -

Date.DayOfWeek(SliceStart))

Data movement

Data movementSOURCE SINKAzure Blob Azure BlobAzure Table Azure TableAzure SQL Database Azure SQL DatabaseAzure SQL Data Warehouse Azure SQL Data WarehouseAzure DocumentDB Azure DocumentDBAzure Data Lake Store Azure Data Lake StoreSQL Server on-premises / Azure IaaS SQL Server on-premises / Azure IaaSFile System on-premises / Azure IaaS File System on-premises / Azure IaaSOracle Database on-premises / Azure IaaSMySQL Database on-premises / Azure IaaSDB2 Database on-premises / Azure IaaSTeradata Database on-premises / Azure IaaSSybase Database on-premises / Azure IaaSPostgreSQL Database on-premises / Azure IaaS

Data movement• Uses the Copy activity and Data Movement Service or Data Management Gateway (for on-premises

or Azure IaaS)

• Globally available service for data movement (except Australia)• executes at sink location, unless source is on-premises (or IaaS) then uses Data Management Gateway

• Exactly one input and exactly one output• Support for securely moving between on-premises and the cloud• Automatic type conversions from source to sink data types• File based copy supports binary, text and Avro formats, and allows for conversion between formats• Data Management Gateway supports multiple data sources but only a single Azure Data Factory

Source

Data Movement ServiceWA

NSerialisation-Deserialisation

Compression

ColumnMapping

…WAN

Sink

Source

Data Management GatewayLA

N/ WAN

Serialisation-Deserialisation

Compression

ColumnMapping

…Sink

LAN/ WAN

Data analysis and transformation

Data analysis and transformationTRANSFORMATION ACTIVITY COMPUTE ENVIRONMENT

Hive HDInsight [Hadoop]

Pig HDInsight [Hadoop]

MapReduce HDInsight [Hadoop]

Hadoop Streaming HDInsight [Hadoop]

Machine Learning activities: Batch Execution and Update Resource

Azure VM

Stored Procedure Azure SQL Database

Data Lake Analytics U-SQL Azure Data Lake Analytics

DotNet HDInsight [Hadoop] or Azure Batch

Data analysis and transformation• Two types of compute environment

• On-demand: Data Factory fully manages environment, currently HDInsight only• Set timeToLive to set allowed idle time once job finishes• Set osType for windows or linux• Set clusterSize to determine number of nodes• Provisioning an HDInsight cluster on-demand can take some time

• Bring your own: Register own computing environment for use as a linked service• HDInsight Linked Service

• clusterUri, username, password and location• Azure Batch Linked Service

• accountName, accessKey and poolName• Machine Learning Linked Service

• mlEndpoint and apiKey• Data Lake Analytics Linked Service

• accountName, dataLakeAnalyticsUri and authorization• Azure SQL Database Linked Service

• connectionString

Development

Development• JSON for all artefacts• Ease of management by source control• Can be developed using:

• Data Factory Editor• In Azure Portal• Create and deploy artefacts

• PowerShell• Cmdlets for each main function in PS ARM

• Visual Studio• Azure Data Factory Templates• .NET SDK

Visual Studio• Rich set of templates including

• Sample applications• Data analysis and transformation using Hive

and Pig• Data movement between typical

environments• Can include sample data• Can create Azure Data Factory, storage

and compute resources• Can Publish to Azure Data Factory• No toolbox, mostly hand crafting JSON

Tips and Tricks with Visual Studio Templates• Something usually fails

• Issues with sample data• Run once to create Data Factory and storage accounts• Usually first run will also create a folder containing Sample Data but NO JSON

artifacts• May need to manually edit PowerShell scripts or perform manual upload• Once corrected, deselect Sample Data and run again creating new solution• Ensure Publish to Data Factory is deselected and JSON artifacts are created

• Issues with Data Factory deployment• Go to portal and check what failed• May need to manually create item but deleting published item and recreating

with JSON from project• When deploying, may need to unselect item that is failing

• You cannot delete from the project• Need to Exclude From Project• Once excluded can delete from disk

Deployment• Add Config files to your Visual

Studio project• Deployment files contain, for

instance, connection strings to resources that are replaced at Publish time

• Add deployment files for each environment you are deploying to, e.g., Dev, UAT, Prod

• When publishing to Azure Data Factory choose appropriate Config file to ensure correct settings are applied

• Publish only artefacts required

Monitoring

Monitoring• Data slices may fail• Drill in to errors, diagnose, fix and rerun• Failed data slices can be rerun and all

dependencies are managed by Azure Data Factory

• Upstream slices that are Ready stay available

• Downstream slices that are dependent stay Pending

• Enable diagnostics to produce logs, disabled by default

• Add Alerts for Failed or Successful Runs to receive email notification

Demonstration

General information

Pricing – Low frequency ( <= 1 / day )USAGE PRICE

Cloud First 5 activities/month Free6 – 100 activities/month $0.60 per activity>100 activities/month $0.48 per activity

On-Premises First 5 activities/month Free6 – 100 activities/month $1.50 per activity>100 activities/month $1.20 per activity

* Pricing in USD correct at 4 December 2015

Pricing – High frequency ( > 1 / day )USAGE PRICE

Cloud <= 100 activities/month $0.80 per activity>100 activities/month $0.64 per activity

On-Premises <= 100 activities/month $2.50 per activity>100 activities/month $2.00 per activity

* Pricing in USD correct at 4 December 2015

Pricing – Data movementCloud $0.25 per hour

On-Premises $0.10 per hour

Pricing – Inactive pipeline$0.80/month

* Pricing in USD correct at 4 December 2015

Summary• Use Azure Data Factory if:

• Dealing with Big Data• Source or destination is in the cloud• Cut down environment cost• Cut down administration cost• Azure is on one side of the movement / transformation

• Consider hybrid scenarios with other data management tools, for example SQL Server Integration Services

More Information• Documentation portal

• https://azure.microsoft.com/en-us/documentation/services/data-factory/

• Learning map• https://azure.microsoft.com/en-us/documentation/articles/

data-factory-learning-map/

• Samples on github• https://github.com/Azure/Azure-DataFactory

Thank you!