Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure...
Transcript of Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure...
![Page 1: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/1.jpg)
![Page 2: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/2.jpg)
aka.ms/DATA10 #MSIgnite
Why modernize?
![Page 3: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/3.jpg)
ZettabytesGigabytes
1990’s 2020’s
Combine both worlds
To meet business needs
Hybrid
Azure Data Platform
Social
Graph IoT
Image
LOB
CRM
ERP
SQL Server
LOB
CRMERP
The evolving world of data
![Page 4: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/4.jpg)
CognitivePrescriptive
PredictiveDiagnostic
Descriptive
The evolving world of analytics
![Page 5: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/5.jpg)
Azure ExpressRoute Azure network
security groups
Azure Functions Visual StudioOperations
Management Suite
Azure Active Directory Azure key
management service
NSG
Cognitive servicesBot service
Azure Analysis Services Power BI
Azure Search Azure Data Catalog
Azure SQL DB Azure Cosmos DB
SQL
Azure ML Azure
Databricks
ML Server
Azure
HDInsight
Azure
Databricks
Azure Stream
Analytics
Azure
HDInsight
Azure
Databricks
Azure SQL data warehouse
Azure Blob Storage Azure Data Lake
Store
1001
Azure IoT Hub Azure event hubs
Kafka on Azure HDInsight
Azure Data
Factory
Azure Import/Export
service
Azure SDKAzure CLI
>_
The Azure Big Data Landscape
![Page 6: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/6.jpg)
Governed Self-ServiceNo Data SilosHybrid
Analyze All DataWorkload Optimized Compute
Elastic Architectures
Azure Modern Data Warehouse benefits
![Page 7: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/7.jpg)
aka.ms/DATA10 #MSIgnite
Solution scenarios
![Page 8: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/8.jpg)
Real-time analytics
“We’re trying to get
insights from our
devices in real-time”
Advanced analytics
“We’re trying to
predict when our
customers churn”
SQL
Modern data warehousing
“We want to integrate
all our data—including
Big Data—with our
data warehouse”
Big Data and advanced analytics
Solution scenarios
![Page 9: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/9.jpg)
Real-time analytics
“We’re trying to get
insights from our
devices in real-time”
Advanced analytics
“We’re trying to
predict when our
customers churn”
SQL
Modern data warehousing
“We want to integrate
all our data—including
Big Data—with our
data warehouse”
The modern data warehouse extends the scope of the data warehouse to
serve Big Data that’s prepared with techniques beyond relational ETL
Azure Modern data warehousing
![Page 10: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/10.jpg)
Azure Data Lake Storage
Store
Power BI
Visualize
Azure SQLData Warehouse
Model & Serve
Azure Data Factory
Azure Databricks
Ingest & Prep
Best end-to-end ecosystem to turn your data into actionable insights
Unparalleled performance
Azure Modern Data Warehouse Processes
![Page 11: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/11.jpg)
Loading and preparing data for analysis with a data warehouse
Data warehousing pattern in Azure
Business and custom
apps (structured)
Logs, files, and media
(unstructured)
Data preparationData storage
Data Lake Store Azure Storage
Serving
Azure SQL DW
AAS
Cosmos DB
Operational data
Cosmos DB SQL DB
HDInsight Data Factory
Azure Databricks
Data
factory
Azure Import/Export
Service
APIs, CLI, and
GUI tools
Azure Data
Box
Data ingestion
Power BI
Dashboards
Applications
Visualize
![Page 12: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/12.jpg)
aka.ms/DATA10 #MSIgnite
Example solution architecture
![Page 13: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/13.jpg)
There are no right or wrong solutions,
only optimal solutions
Lead with certain solutions and customize
based on customer scenarios
Customer voice and product and service
maturity govern lead solutions
Consider price and performance, ease of
use, and ecosystem acceptance as factors
Everything is fluid - a lead solution today
might be non-optimal tomorrow, based on
the factors above and new releases
Things to note
![Page 14: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/14.jpg)
The storage that persists
the transferred data that
will be consumed by
subsequent processing
Data ingestion and storage
![Page 15: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/15.jpg)
Data ingestionLoad flat files
into data lake
on a schedule
Data storage
Azure Storage/
Data Lake Store
Azure Data
Factory
Power BI
Dashboards
Logs, files, and media
(unstructured)
Applications
Business and custom
apps (structured)
Ingesting data into Azure Data Lake with Azure Data Factory
Data warehousing pattern in Azure
![Page 16: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/16.jpg)
Data ingestionLoad flat files
into data lake
on a schedule
Data storage
Azure Storage/
Data Lake Store
Azure Data
Factory
Power BI
Dashboards
Logs, files, and media
(unstructured)
Applications
Business and custom
apps (structured)
Ingesting data into Azure Data Lake with Azure Data Factory
Data warehousing pattern in Azure
Transactional
storage
Applications
manage their
transactional
data directly
SQL DB
![Page 17: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/17.jpg)
Is data cleansing, structuring,
curation, and aggregation in
data warehousing.
The data is batch processed
in preparation for loading
into a data warehouse
Data preparation
![Page 18: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/18.jpg)
Data ingestionLoad flat files
into data lake
on a schedule
Data storage
Azure Storage/
Data Lake Store
Azure Data
Factory
Power BI
Dashboards
Logs, files, and media
(unstructured)
Applications
Business and custom
apps (structured)
Ingesting data into Azure Data Lake with Azure Data Factory
Data warehousing pattern in Azure
Transactional
storage
Applications
manage their
transactional
data directly
SQL DBExtract and
transform
relational dataData prep
Azure Data
Factory
![Page 19: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/19.jpg)
Data ingestionLoad flat files
into data lake
on a schedule
Data storage
Azure Storage/
Data Lake Store
Azure Data
Factory
Power BI
Dashboards
Logs, files, and media
(unstructured)
Applications
Business and custom
apps (structured)
Ingesting data into Azure Data Lake with Azure Data Factory
Data warehousing pattern in Azure
Transactional
storage
Applications
manage their
transactional
data directly
SQL DBExtract and
transform
relational dataData prep
Azure Data
Factory
Data preparation
Read data
from files
using DBFSAzure Databricks
![Page 20: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/20.jpg)
Processed data served by a
data warehouse to analytic
clients and reporting tools
The data warehouse provides
increased query flexibility and
reduced query latency in
comparison to batch data
processing options
Data serving
![Page 21: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/21.jpg)
Data ingestionLoad flat files
into data lake
on a schedule
Data storage
Azure Storage/
Data Lake Store
Azure Data
Factory
Power BI
Dashboards
Logs, files, and media
(unstructured)
Applications
Business and custom
apps (structured)
Ingesting data into Azure Data Lake with Azure Data Factory
Data warehousing pattern in Azure
Transactional
storage
Applications
manage their
transactional
data directly
SQL DBExtract and
transform
relational dataData prep
Azure Data
Factory
Data preparation
Read data
from files
using DBFSAzure Databricks
Load
into SQL
DW
tables
Serving
Load processed
data into tables
optimized for
analytics
Azure SQL DW
Visualize
![Page 22: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/22.jpg)
![Page 23: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/23.jpg)
Monitoring
Provides insights into
the status and health of
the data warehouse
solution
Automation
Enables all components
of the modern data
warehouse solution to
be controlled, deployed,
and monitored
programmatically
Security
Enables the modern
data warehouse to
control access in order
to protect sensitive data
and maintain desired
compliance
Big Data and advanced analytics
Modern Data Warehouse considerations
![Page 24: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/24.jpg)
![Page 25: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/25.jpg)
What is Azure Data Factory?
![Page 26: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/26.jpg)
A cloud-based data integration service that allows
you to orchestrate and automate
data movement and data transformation.
AZURE DATA FACTORY
![Page 27: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/27.jpg)
AZURE DATA FACTORY는
코딩이 필요 없는 하이브리드 데이터 통합 서비스 솔루션- 다양한 데이터 소스에서 데이터를 추출하고- 원하는 분석 엔진 또는 비즈니스 인텔리전스 도구에 게시- 데이터 파이프라인을 모니터링 및 관- 데이터가 클라우드와 온-프레미스 중 어디에 있든,
엔터프라이즈급 보안으로 작업- 80개 이상의 데이터 원본 커넥터를 사용.
- 그래픽 사용자 인터페이스를 사용하여 데이터 파이프라인을빌드하고, 모니터링하고, 관리
![Page 28: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/28.jpg)
Azure Data Factory orchestrates and operationalizes data pipeline workflow
Modern DW for BI
Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure
storage
Polybase
Azure SQL Data
Warehouse
Data
factory
Data
factory
Azure
Databricks
(Spark)
Analytical
dashboards
(PowerBI)
Model &
Serve
Prep &
Train
StoreIngest Intelligence
Azure Analysis ServicesOn Prem, Cloud
Apps & Data
![Page 29: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/29.jpg)
MODEL & SERVE
Azure Analysis ServicesAzure SQL Data Warehouse
Power BI
Modernize your enterprise data warehouse at scale
A Z U R E D A T A F A C T O R Y
On-premises data
Oracle, SQL, Teradata,
fileshares, SAP
Cloud data
Azure, AWS, GCP
SaaS data
Salesforce, Workday,
Dynamics
INGEST STORE PREP & TRAIN
Azure Data Factory Azure Blob Storage
Azure Databricks
Polybase
Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure SQL Database and Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs.
Orchestrate with Azure Data Factory
![Page 30: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/30.jpg)
Author, orchestrate and monitor with Azure Data Factory
Hybrid and Multi-Cloud Data Integration
Azure Data FactoryPaaS Data Integration
DATA SCIENCE AND MACHINE
LEARNING MODELS
ANALYTICAL DASHBOARDS
USING POWER BI
DATA DRIVEN APPLICATIONS
On-Prem SaaS Apps Public Cloud
![Page 31: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/31.jpg)
![Page 32: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/32.jpg)
Access all your data – 90+ connectors & growing
* Supported file formats: CSV, Parquet, AVRO, ORC, JSON
Azure (12) Database (24) File Storage (5) NoSQL (3) Services and Apps (28) Generic (3)
Blob Storage Amazon Redshift Netezza Amazon S3 Cassandra Amazon MWS Oracle Service Cloud HTTP
Cosmos DB (SQL API) DB2 Oracle File System CouchbaseCommon Data Service for
AppsPaypal OData
Data Lake Storage Gen1
Drill Phoenix FTP MongoDB Concur QuickBooks ODBC
Data Lake Storage Gen2
Google BigQuery PostgreSQL HDFS Dynamics 365 Salesforce
DB for MySQL Greenplum Presto SFTP Dynamics CRMSalesforce Marketing
Cloud
DB for PostgreSQL HBase SAP BW GE HistorianSalesforce Service
Cloud
File Storage Hive SAP HANA Google AdWords SAP C4C
SQL DB Impala Spark HubSpot SAP ECC
SQL DB Managed Instance
Informix SQL Server Jira ServiceNow
SQL DW MariaDB Sybase Magento Shopify
Search Index Microsoft Access Teradata Marketo Square
Table Storage MySQL Vertica Office 365 Web table
Oracle Eloqua Xero
Oracle Responsys Zoho
![Page 33: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/33.jpg)
CFControl
Flow
IRIntegration
Runtime
@ Parameters
Pipeline
Activities
Triggers
Dataset
Azure Databricks
Data Lake Store
Linked Service
AZURE DATA FACTORY COMPONENTS
![Page 34: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/34.jpg)
COMPONENT DEPENDENCIES
![Page 35: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/35.jpg)
Azure Data Factory Updated Flexible Application Model
Pipeline
Activity
Triggers
Pipeline Runs
Activity Runs
Trigger Runs
Linked Service
Dataset
Data Movement
Data Transformation
Dispatch
Integration Runtime
Activity
Activity
![Page 36: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/36.jpg)
Control Flow Introduced in Azure Data Factory
Coordinate pipeline activities into finite execution steps to enable looping,
conditionals and chaining while separating data transformations into
individual data flows
Activity 1 Activity 2
Activity 3
“On Error”
Activity 1
Success, params
Error, params
My Pipeline 1
…
My Pipeline 2For Each…
Activity 4
Success, params
Trigger
EventWall ClockOn Demand
Activity 1
Activity 2
…
![Page 37: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/37.jpg)
![Page 38: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/38.jpg)
![Page 39: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/39.jpg)
![Page 40: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/40.jpg)
ADF Certifications
HIPAA/HITECH
ISO/IEC 27001
ISO/IEC 27018
CSA STAR
![Page 41: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/41.jpg)
Ingesting data
![Page 42: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/42.jpg)
Data ingestion
Load flat files
into data lake
on a schedule
Data storage
Transactional storage
Applications
manage their
transactional
data directly
Data preparation
Read data from files
using DBFS
Extract and
transform
relational data
Load into SQL
DW tables
Data prep.
Serving
Load processed data
into tables optimized
for analytics
Azure Storage/
Data Lake Store
Azure SQL DW
Azure Databricks
Azure Data
Factory
SQL DBAzure Data
Factory
Logs, files, and media
(unstructured)
Business and custom
apps (structured)
Power BI
Dashboards
Applications
Visualize
Connect & Collect
ADF Copy Activity
Copy Activity
Publish
Data transformation in Azure
Transforming data with Azure Data Factory
![Page 43: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/43.jpg)
Reads data from a source data store.
Performs serialization/deserialization, compression/decompression, column
mapping, and so on. It performs these operations based on the configuration
of the input dataset, output dataset, and Copy activity.
Writes data to the sink/destination data store
COPY ACTIVITY PROCESS
![Page 44: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/44.jpg)
SQL Server
Self-hosted
Integration Runtime
IR
Azure Integration
Runtime
IR
INTEGRATION RUNTIME
![Page 45: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/45.jpg)
Supported file formats:
Text
JSON
Avro
ORC
Parquet
Copy activity can compress
and decompress files with
The following codecs:
Gzip
Deflate
Bzip2
ZipDeflate
COPY FILES WITH THE COPY ACTIVITY
![Page 46: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/46.jpg)
Monitoring data ingestion
![Page 47: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/47.jpg)
Pipeline runsActivity runs
Monitoring
![Page 48: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/48.jpg)
Summary
Azure Data Factory (ADF) is a cloud-based
data integration service that allows you to
orchestrate and automate data movement
and data transformation.
Ingesting data can be performed by the ADF
Copy Activity
The ADF Copy Activity can be used to
connect and collect data for ingestion, and to
publish data to BI tools and applications.
Different Integration Runtimes are required
for different ingestion scenarios
File copy are very efficient using the ADF
Copy Activity
You can monitor the performance of the ADF
Copy Activity both visually and
programmatically
![Page 49: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/49.jpg)
![Page 50: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/50.jpg)
aka.ms/mymsignitethetour
aka.ms/DATA30Repo
aka.ms/DATA30
RESOURCES
![Page 51: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/51.jpg)
What is Azure Data Factory?
![Page 52: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/52.jpg)
A cloud-based data integration service that
allows you to orchestrate and automate
data movement and data transformation.
AZURE DATA FACTORY
![Page 53: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/53.jpg)
MonitorPublishTransform
& EnrichConnect & Collect
AZURE DATA FACTORY PROCESS
![Page 54: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/54.jpg)
CFControl
Flow
IRIntegration
Runtime
@ Parameters
Pipeline
Activities
Triggers
Dataset
Azure Databricks
Data Lake Store
Linked Service
AZURE DATA FACTORY COMPONENTS
![Page 55: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/55.jpg)
COMPONENT DEPENDENCIES
![Page 56: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/56.jpg)
Transforming data with the ADF Mapping Data Flow
![Page 57: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/57.jpg)
Transform
& Enrich
Data ingestion
Load flat files
into data lake
on a schedule
Data storage
Transactional storage
Applications
manage their
transactional
data directly
Data preparation
Read data from files
using DBFS
Extract and
transform
relational data
Load into SQL
DW tables
Data prep.
Serving
Load processed data
into tables optimized
for analytics
Azure Storage/
Data Lake Store
Azure SQL DW
Azure Databricks
Azure Data
Factory
SQL DBAzure Data
Factory
Logs, files, and media
(unstructured)
Business and custom
apps (structured)
Power BI
Dashboards
Applications
Visualize
DATA TRANSFORMATION IN AZURE
Transforming data with Azure Data Factory
![Page 58: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/58.jpg)
Mapping Data Flow
SSIS PackagesCompute resources
METHODS FOR TRANSFORMING IN AZURE DATA FACTORY
![Page 59: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/59.jpg)
Mapping Data Flow
Compute resources
SSIS Packages
Code free data transformation at scale
METHODS FOR TRANSFORMING DATA IN AZURE DATA FACTORY
![Page 60: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/60.jpg)
Mapping Data Flow
Perform data cleansing, transformation,
aggregations, etc.
Enables you to build resilient data flows in a code
free environment
Enable you to focus on building business logic and
data transformation
Underlying infrastructure is provisioned
automatically with cloud scale via Spark execution
Code free data transformation at scale
BENEFITS OF MAPPING DATA FLOW
![Page 61: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/61.jpg)
Code free data transformation at scale
USING THE MAPPING DATA FLOW
![Page 62: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/62.jpg)
Code free data transformation at scale
STARTING THE MAPPING DATA FLOW
![Page 63: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/63.jpg)
TRANSFORMATION OPTIONS IN THE MAPPING DATA FLOW
![Page 64: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/64.jpg)
Triggering and monitoring
![Page 65: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/65.jpg)
Code free data transformation at scale
TRIGGERING THE MAPPING DATA FLOW
![Page 66: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/66.jpg)
Summary
Azure Data Factory (ADF) is a cloud-based
data integration service that allows you to
orchestrate and automate data movement
and data transformation.
Transforming data can be performed in ADF
by orchestrating a compute resource, calling
an SSIS package or using the Mapping Data
Flow feature
The Mapping Data Flow feature enables code
free data transformation at scale
Enable you to focus on building business
logic and data transformation
It is added to an ADF Pipeline, and can be
scheduled or triggered
You can monitor the Mapping Data Flow
both visually and programmatically
![Page 67: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/67.jpg)
![Page 68: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/68.jpg)
What is Azure SQL Data Warehouse?
![Page 69: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/69.jpg)
Workload
Management
Separate
Storage/Compute
Pause/Resume
Big DataElastic ScalePaaS
AZURE SQL DATA WAREHOUSE
![Page 70: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/70.jpg)
SQL Data Warehouse Architecture
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110Control Node
![Page 71: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/71.jpg)
QueryLoadProvision
AZURE SQL DATA WAREHOUSE PROCESSES
![Page 72: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/72.jpg)
Loading design goals
![Page 73: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/73.jpg)
Data ingestion
Load flat files
into data lake
on a schedule
Data storage
Transactional storage
Applications
manage their
transactional
data directly
Data preparation
Read data from files
using DBFS
Extract and
transform
relational data
Load into SQL
DW tables
Data prep.
Serving
Load processed data
into tables optimized
for analytics
Azure Storage/
Data Lake Store
Azure SQL DW
Azure Databricks
Azure Data
Factory
SQL DBAzure Data
Factory
Power BI
Dashboards
Logs, files, and media
(unstructured)
Applications
Business and custom
apps (structured)
Visualize
Loading data into Azure SQL Data Warehouse
Data warehousing loading in Azure
![Page 74: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/74.jpg)
File based
PolyBase
Heterogenous
SSIS
File based
BCP
Loading Methods
![Page 75: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/75.jpg)
File based
PolyBase
Heterogenous
SSIS
File based
BCP
For large amounts of data, there is only one choice
Loading Methods
![Page 76: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/76.jpg)
Variety of file formats
PolyBase supports a
variety of file formats
including RC, ORC
and Gzip files.
Azure Data Factory support
Azure Data Factory also
supports PolyBase loads
and can achieve similar
performance to running
PolyBase manually
Leverages MPP architecture
PolyBase is designed to
leverage the MPP
(Massively Parallel
Processing) architecture of
SQL Data Warehouse and
will therefore load and
export data magnitudes
faster than any other tool.
The best practice for loading large amount of data
PolyBase benefits
![Page 77: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/77.jpg)
External Tables
External File Format
External Data Source
Components of PolyBase
![Page 78: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/78.jpg)
Loading best practices
![Page 79: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/79.jpg)
Manage your files
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110Control Node
![Page 80: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/80.jpg)
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110Control Node
Reduce concurrent access
![Page 81: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/81.jpg)
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110Control Node
Create a dedicated load user account
![Page 82: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/82.jpg)
Manage singleton updates
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110
Compute Node
Compute Node
01101010101010101011
01010111010101010110
01101010101010101011
01010111010101010110Control Node
![Page 83: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/83.jpg)
Production Tables
Staging Tables
Azure SQL DW
Load into
SQL DW
tablesAzure Storage/
Data Lake Store
View it as a two-stage process
Optimize your loads
![Page 84: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/84.jpg)
Improve the query performance for users
Create statistics after loading
Azure SQL DW
Production Tables
![Page 85: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/85.jpg)
Maximizing Performance
![Page 86: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/86.jpg)
Maximizing Query Performance
Replicated Tables
Round Robin Tables
Hash Distributed Tables
![Page 87: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/87.jpg)
Maximizing Query Performance
Round-robin Tables
Is the default option for newly created
tables
Evenly distributes the data across the
available compute nodes in a random
manner, giving an even distribution of data
across all nodes
Loading into Round-robin tables is fast
Queries on Round-robin tables may require
more data movement as data is
“reshuffled” to organize the data for the
query
Great to use for loading staging tables
![Page 88: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/88.jpg)
Maximizing Query Performance
Hash Distributed Tables
Distributes rows based on the value in the
distribution column, using a deterministic hash
function to assign each row to one distribution.
Is designed to achieve high performance for
queries that run against large fact tables in a
star schema.
Choosing a good distribution column is
important to ensure the hash distribution
performs well
As a starting point, use on tables that are
greater than 2GB in size and has frequent
inserts, updates and deleted
But don’t choose a volatile column for the hash
distributed column
![Page 89: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/89.jpg)
Maximizing Query Performance
Replicated Tables A full copy of a table is placed on every single
compute node to minimize data movement
Works well for dimension tables in a star
schema that are less than 2GB in size and are
used regularly in queries with simple
predicates
Should not be used on dimension tables that
are updated on a regular basis
![Page 90: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/90.jpg)
Create statistics after loading
Azure SQL DW
Production Tables
![Page 91: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/91.jpg)
SummaryAzure Data Factory (ADF) is a cloud-based
data integration service that allows you to
orchestrate and automate data movement
and data transformation.
Enable you to focus on building business
logic and data transformation
It is added to an ADF Pipeline, and can be
scheduled or triggered
You can monitor the Mapping Data Flow
both visually and programmatically
Load data efficiently
Multiple methods of loading
![Page 92: Why modernize? - info.microsoft.com€¦ · Visualize Azure SQL Data Warehouse Model & Serve Azure Data Factory Azure Databricks Ingest & Prep Best end-to-end ecosystem to turn your](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f243d0f32920e634506c8d8/html5/thumbnails/92.jpg)