Performance Tuning SSIS
description
Transcript of Performance Tuning SSIS
![Page 2: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/2.jpg)
About the Ugly Guy SpeakingSQL Server MVPFounder of Pragmatic WorksCo-Founder of BIDN.com, SQLServerCentral.com and SQLShare.comWritten more than a dozen books on SQL Server
![Page 3: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/3.jpg)
GeoSpatial Data:Semi structured
Legacy data: binary files
Application database
Integration is a seamless, manageable operationSource, prepare, & load data in single, auditable processScale to handle heavy and complex data requirements
SQL Server Integration Services
GeoSpatialComponents
Customsource
Standardsources
Data-cleansingcomponents
Merges
Data miningcomponents
Warehouse
Reports
Mobiledata
Integration Services in Action
Cube
![Page 4: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/4.jpg)
Advanced Session
![Page 5: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/5.jpg)
Today’s Problems with Integration
Integration todayIncreasing data volumesIncreasingly diverse sources
Requirements reached the Tipping PointLow-impact source extractionEfficient transformationBulk loading techniques
![Page 6: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/6.jpg)
Tuning DecisionsChoose the right tool for the jobDon’t be afraid to use T-SQLWill parallelism work?
![Page 7: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/7.jpg)
Source OptimizationFlat files – When available, use Fast ParseOLE DB sources – Change network packet sizeUse T-SQL whenever possible in the OLE DB Source
JoiningNULL handlingWhere clauses
![Page 8: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/8.jpg)
Impact of Compression on ETL
NONE ROW PAGE05
101520253035
0123456
BULK INSERT into a Heap with and without Data Compression
Time to BULK INSERT 50M rows (min)Table Size after Load (GB)
Compression Type
Tim
e (m
inut
es)
Tabl
e Si
ze a
fter L
oad
(GB)
* Not official Microsoft results.
![Page 9: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/9.jpg)
Tuning the Source
Demo
Connection manager tuningFlat file tuningOLE DB Source tuning
![Page 10: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/10.jpg)
Transform Componentsx x x
The Pipeline presents the buffer to each downstream component
x x xx x xx x xx x xx x x
![Page 11: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/11.jpg)
SSIS Data Flow Architecture
Synchronous vs. Non Synchronous
![Page 12: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/12.jpg)
Case Study: Patterns
105 seconds 83 seconds
![Page 13: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/13.jpg)
Source Data Extraction
Extracting data from the source is expensiveEfficient extraction is key to improving ETL performanceInvolves bulk loading data into staging areas or warehouse
Time consuming & resource intensiveTriggers (synchronous IO penalty)Timestamp columns (Schema changes)Complex queries (delayed IO penalty)Custom (ISV, mirror, snapshot, …)
Incremental data load is key to efficient extractionNeed to know what changed at source since a point in time
Expensive lookups to determine changed columnsProviding information up front about which columns changedwill improve efficiency
![Page 14: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/14.jpg)
SQL Server 2008: Change Data Capture (CDC)
Information about what changed at the sourceChanges captured from the log asynchronouslyEnabled per tableCDC APIs provide access to change data
Change Tables
OLTP
Data Warehouse
![Page 15: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/15.jpg)
Change Data Capture
Demo
Traditional CDC with SSISIntegrating CDC in 2008
![Page 16: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/16.jpg)
Lookup Component
Three modes of operationFull Cache: for small lookup datasetsNo Cache: for volatile lookup datasetsPartial Cache: for large lookup datasets
Tradeoff memory vs. performanceUse Cascaded LookupsMerge Join may be alternative
![Page 17: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/17.jpg)
SQL Server 2008: Lookup Transform
Hydrate cache files for large data setsCan reuse cacheCan load cache during day and use in nightly ETL
![Page 18: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/18.jpg)
DemoCascading lookup optimizationsCache file lookup
![Page 19: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/19.jpg)
Data Destinations
Use “Fast Load” or SQL Server DestinationTable Lock on insert operationsTrace flags for improvementOld principles still apply
![Page 20: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/20.jpg)
Destination Tuning
Demo
![Page 21: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/21.jpg)
Building a Work Queue System
Create a work queue table.
Create a loop to shift over the work queue constantly checking out work
Spawn x times with a batch file
![Page 22: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/22.jpg)
Demo Results
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
1 Process finishes in 64 seconds
Elap
sed
Tim
e
![Page 23: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/23.jpg)
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
2 Processes finish in 36 seconds
Elap
sed
Tim
e
Demo Results
![Page 24: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/24.jpg)
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
4 Processes finish in 28 seconds
Elap
sed
Tim
e
Demo Results
![Page 25: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/25.jpg)
1 2 3 4 5 6 7 800:00.0
00:08.6
00:17.3
00:25.9
00:34.6
00:43.2
00:51.8
01:00.5
01:09.1
8 Processes finish in 27 seconds
Elap
sed
Tim
e
Demo Results
![Page 26: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/26.jpg)
Parallel Load
Demo
![Page 27: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/27.jpg)
Managing Resources
Logging events to watch pipeline internalsPipelineExecutionPlan, PipelineExecutionTree, BufferSizeTuning
System Monitor to track I/O issuesBuffers In Use tracks how many buffers are presently being usedBuffers Spooled tracks how many 10 mb buffers have been spooled to disk
![Page 28: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/28.jpg)
Measuring PerformancePerfmon
![Page 29: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/29.jpg)
Location
Consider the following configuration…
Where should SSIS run? (Licensing issues aside)
SQL Server 1 SQL Server 2
SSIS Server
![Page 30: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/30.jpg)
WSRM
Windows System Resource Manager (WSRM) can throttle CPU and memory
Creates a soft throttleCan be scheduled so SSIS gets priority on weekends and nightsOnly activates policy if resources begin to become constrained (about 70%)WSRM is free with Windows Server 2003 Enterprise Edition and included in Windows Server 2008
![Page 31: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/31.jpg)
WSRMCreating a soft schedule cap
Demo
![Page 32: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/32.jpg)
Summary
PlanningDon’t underestimate the power of the whiteboard!
Use the right tool for the right jobLeverage the power of the engine
Patterns and PracticesUnderstand best practicesBut don’t be afraid to experiment
![Page 33: Performance Tuning SSIS](https://reader036.fdocuments.in/reader036/viewer/2022062305/56815ab3550346895dc865ea/html5/thumbnails/33.jpg)
The End Already?
Questions
http://www.bidn.com/people/brianknight
@BrianKnight
http://www.youtube.com/pragmaticworks