5. 5. Data Flow BasicsData Flow Basics

Presented by: Pramod Singla

Pramod_singla@yahoo.co.in

.

ContentContent Recap and Q&A Data Flow Task Pipeline Architecture Data Sources

◦ Demo: ADO.NET Source ◦ Demo: Excel Source ◦ Demo: Flat File Source ◦ Demo: OLE DB Source ◦ Demo: XML Source ◦ Demo: Raw File Destination ◦ Demo: Raw File Source

Data Destinations ◦ Demo: OLE DB Destination◦ Demo: DataReader Destination◦ Demo: Excel Destination◦ Demo: Flat File Destination ◦ Demo: SQL Server Destination Analysis Services Destinations ◦ Demo: Dimension Processing ◦ Demo: Partition Processing ◦ Demo: Data Mining Model Training

Summary

Recap and Q&ARecap and Q&AVariables

◦ Demo: Introducing the Variables Window◦ Demo: Working with Variables◦ Demo :Using variables in Execute SQL task

Expressions◦ Demo: Creating Time stamped log file◦ Demo: Operators and More Variables

Expression Usage◦ Demo: Using a Variable in an Expression◦ Demo: Using an Expression with a Connection Manager◦ Demo: Using Expressions with Precedence Constraints

Data Flow Task Data Flow Task Encapsulates the data flow

engine that Extract, transform, and load data

A Data Flow task can include multiple data flows

When order is important, use multiple Data Flow tasks, each task containing one data flow.

Diagram includes the inputs, outputs, ,error outputs external columns.

Pipeline Architecture Pipeline Architecture The run-time engine implementsthe control flow and package management infrastructure

The data flow engine is a specialized, high performance engine that is exclusively dedicated to extracting, transforming, and loading data.

The data flow engine provides the in-memory buffers

Arrow in dataflow are called Service path

ADO.NET SourceADO.NET Source The ADO NET source consumes data from a .NET provider and makes the

data available to the data flows

SSIS 2012 onwards, you can use the ADO NET source to connect to Microsoft Windows Azure SQL Database

The ADO.net source provides two different data access modes for extracting data:◦ A table or view.◦ The results of an SQL statement.

Excel SourceExcel Source The Excel source extracts data

from worksheets or ranges in Microsoft Excel workbooks

The Excel source provides four different data access modes for extracting data:◦ A table or view.◦ A table or view specified in a

variable.◦ The results of an SQL statement. ◦ The results of an SQL statement

stored in a variable.

Flat File SourceFlat File Source The Flat File source reads data from a

text file. The text file can be in delimited, fixed width, or mixed format.

Delimited format uses column and row delimiters to define columns and rows.

Fixed width format uses width to define columns and rows. This format also includes a character for padding fields to their maximum width.

Ragged right format uses width to define all columns, except for the last column, which is delimited by the row delimiter.

OLE DB SourceOLE DB Source The OLE DB source extracts

data from a variety of OLE DB-compliant relational databases

The OLE DB source provides four different data access modes for extracting data:◦ A table or view.◦ A table or view specified in a

variable.◦ The results of an SQL statement. The

query can be a parameterized query.◦ The results of an SQL statement

stored in a variable.

XML SourceXML Source The XML Source does not

require a Connection Manager, but instead references a XML data source either as a◦ direct reference to a file◦ a variable containing a file reference

or ◦ a variable containing XML data.

Raw File DestinationRaw File Destination The Raw File destination writes

raw data to a file. Because the format of the data is native to the destination

write data more quickly than other destinations

 You use the Raw File source to retrieve raw data that was previously written by the destination

This source does not use a connection manager.

Raw File SourceRaw File Source The Raw File source reads raw

data from a file

Reads data more quickly than other sources

Retrieve raw data that was previously written by the Raw File destination

This source does not use a connection manager.

OLEDB DestinationOLEDB Destination The OLE DB destination loads data into a variety of OLE DB-compliant databases

The OLE DB destination provides five different data access modes for loading data:

◦ A table or view. You can specify an existing table or view, or you create a new table.◦ A table or view using fast-load options. You can specify an existing table or create a new table.◦ A table or view specified in a variable.◦ A table or view specified in a variable using fast-load options.◦ The results of an SQL statement.

Does not support parameters. To execute a parameterized INSERT statement, use OLE DB Command transformation

If OLE DB destination uses fast-load data access mode, specify following options:◦ Keep identity values from the imported data file or use unique values assigned by SQL Server.◦ Retain a null value during the bulk load operation.◦ Check constraints on the target table or view during the bulk import operation.◦ Acquire a table-level lock for the duration of the bulk load operation.◦ Specify the number of rows in the batch and the commit size.

DataReader DestinationDataReader Destination Exposes the data in a data flow by using the ADO.NET DataReader interface

The data can then be consumed by other application

Excel DestinationExcel Destination The Excel destination loads data into

worksheets or ranges in Microsoft Excel workbooks

Can create new sheet based on input

In Excel, a worksheet or range is the equivalent of a table or view.

The lists of available tables in the Excel Source and Destination editors display only existing worksheets

Flat File DestinationFlat File Destination The Flat File destination writes data to a text file

This destination uses a Flat File connection manager to access the text file

Specify whether to overwrite a data in a destination file that has the same name.

SQL Server DestinationSQL Server Destination Connects to a local SQL Server database and bulk loads data

Cannot access a SQL Server database on a remote server

Users who execute packages that include the SQL Server destination require the "Create global objects" permission.

 Assigning "Create Global Objects" user right to that account: Click Start, point to Administrative Tools, and then click Local Security Policy.   Expand Local Policies, and then click User Rights Assignment.   In the right pane, double-click Create global objects In the Local Security Policy Setting dialog box, click Add.   In the Select Users or Group dialog box, click the user accounts that you want to add,

click Add, and then click OK.  Click OK. 

If you are running the package from either SQL Server Management Studio (SSMS) or Business Intelligence Development Studio (BIDS) or DTExecUI.exe, launch those tools under the elevated Administrator account

Dimension ProcessingDimension Processing The Dimension Processing

destination loads and processes an SQL Server Analysis Services dimension

The Dimension Processing destination includes the following features:

◦ Options to perform incremental, full, or update processing.

◦ Error configuration, to specify whether dimension processing ignores errors or stops after a specified number of errors.

◦ Mapping of input columns to columns in dimension tables.

Partition ProcessingPartition Processing The Partition Processing destination loads and processes an SQL Server Analysis

Services partition.

The Partition Processing destination includes the following features:◦ Options to perform incremental, full, or update processing.◦ Error configuration, to specify whether processing ignores errors or stops after a specified number of errors.◦ Mapping of input columns to partition columns.

SummarySummary Data Flow Task Pipeline Architecture Data Sources

◦ Demo: ADO.NET Source ◦ Demo: Excel Source ◦ Demo: Flat File Source ◦ Demo: OLE DB Source ◦ Demo: XML Source ◦ Demo: Raw File Destination ◦ Demo: Raw File Source

Data Destinations ◦ Demo: OLE DB Destination◦ Demo: DataReader Destination◦ Demo: Excel Destination◦ Demo: Flat File Destination ◦ Demo: SQL Server Destination Analysis Services Destinations ◦ Demo: Dimension Processing ◦ Demo: Partition Processing

DEMO5_DataFlowBasics.dtsx 5_DataFlowBasics_OnlinDemo.dtsx

Resources & QuestionsResources & QuestionsContact me :

◦ Pramod_singla@yahoo.co.in◦ http://pramodsingla.wordpress.com/

Microsoft Resources:◦ http://pluralsight.com/training/courses/TableOfContents?courseName=ssis-basic&highlight=stacia-misner_08-ssis-basic-debugging!stacia-misner_06-ssis-basic-expressions*2,3#08-ssis-basic-debugging◦ http://technet.microsoft.com/en-us/library/

microsoft.sqlserver.dts.pipeline.wrapper(v=sql.105).aspx◦ http://technet.microsoft.com/en-us/library/bb522498(v=sql.105).aspx◦ http://www.mssqltips.com/sqlservertip/2557/ssis-architecture-and-internals-interview-questions/◦ http://msdn.microsoft.com/en-us/library/ms141122.aspx◦ http://www.bimonkey.com/support/ssis-component-samples-list/◦ http://technet.microsoft.com/en-us/library/ms137622.aspx◦ http://support.microsoft.com/kb/2009672◦ https://www.simple-talk.com/sql/reporting-services/using-sql-server-integration-services-to-

bulk-load-data/