A Complex SSIS Package
-
Upload
nitil-dwivedi -
Category
Documents
-
view
851 -
download
1
Transcript of A Complex SSIS Package
![Page 1: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/1.jpg)
Package values passed to parameters.dtsx
Control
Flow
Data
Flow
![Page 2: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/2.jpg)
Event Handler (Raise events once Package starts executing)
Package Explorer
![Page 3: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/3.jpg)
Connection Used—
1. OLEDB Connection
2. Flat File Connection
Control Flow Task Used—
1. Execute SQL Task
2. Data Flow Task
3. Script Task
Data Flow Task/Transformation Used—
1. Flat File Source/Destination
2. OLEDB Destination
3. Multicast Transformation
4. Conditional Split Transformation
5. Data Conversion Transformation
6. Derived Column Transformation
7. Row Count Transformation
Event Handler Used—
1. OnError
2. OnPrevalidate
3. OnPostValiadte
1-Truncates Employee_Details and Employee_Strenth Tables in SQL Server--->
It uses ‘Execute SQL Task’ to Truncate Tables [Employee_Details] and [Employee_Strenth] in ‘Nitil
Database’ using ‘Connects to SQL Server--LocalHost.Nitil’ Connection
Truncate table [Employee_Details];
Truncate table [Employee_Strenth];
![Page 4: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/4.jpg)
2-Picks up Every ( 4 Files) Flat File---Foreach Loop Container---->
It uses ‘Foreach Loop Contaner’ to pick up data (flat files’ data) from path
‘C:\Users\Nitil\Desktop\package practice\ForEach’ using variable ‘Path of file’. Connection string is
mapped with variable ‘Path of file’ to store the connection and Expressions are used here to evaluate
variable’s value dynamically(changes the connection every time for all 4 files) so it could pick up all the
files from path ‘C:\Users\Nitil\Desktop\package practice\ForEach’
3-Copy Data from Flat File and load it to sql Server and Flat Files---Data Flow Task--->
Once all files are picked up it will perform E (Extract), T(Transform) and L(Load) the Data
a- Copy Data from Flat File---Flat File Source--->Extracts data from flat file Path using connection
‘Extract Flat files--Flat File Connection Manager 1’
![Page 5: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/5.jpg)
Note: ‘fx’ mentioned in connection that means this connection is using expressions
b. Copy Error Output Of Flat File to "FlatFile_Source_Error Table"---OLEDB Destination--->
If any error comes due to irrelevant data (for Example data type is not matched with flat file source and SQL destination Table) in flat file that will load in table ‘[Nitil].[dbo].[FlatFile_Source_Error]’ and we can analyze the error by querying the data from table
![Page 6: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/6.jpg)
c. Move data to Multiple location---Multicast---->
‘Malticast’ Transformation is used to move data to multiple locations
For example in this package, I am moving data to 3 locations after transformations.
d- Performs Aggregation on Data—Aggregate--->
‘Aggregate Transformation’ is used here to perform aggregation on data
![Page 7: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/7.jpg)
For Example In this package, I am going to find ‘Total employees’ for each Job, gender and marital
status.
e- Loads Aggregated Data to "Employee_Strenth" Table---OLEDB Destination--->
Once aggregation is done data will load to table [Nitil].[dbo].[Employee_Strenth] Below output is mentioned once you query the data from ‘[Employee_Strenth]’
![Page 8: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/8.jpg)
f- Move Data to different location on the basis of Condition-----Conditional Split--->
Conditional Split Transformation moves data to different locations on the basis of condition.
For Example In this package, I have mentioned conditions for desired output and data get spitted in
Female and Male output
![Page 9: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/9.jpg)
g- Loads Female Ouput--Flat File Destination--->
It loads data for Female Employees in folder ‘C:\Users\Nitil\Desktop\package
practice\ForEach\Output’ using connection “Load "Female"Output--Flat File Connection
Manager”
h- Loads Male Output--Flat File Destination--->
![Page 10: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/10.jpg)
It loads data for Male Employees in folder ‘C:\Users\Nitil\Desktop\package
practice\ForEach\Output’ using connection “Load "Male"Output--Flat File Connection
Manager”
I-Converts Data type of Column "LoginID"---Data Conversion--->
Data Conversion Transformation is used to change the data type a column
In this package, I have changed data type of ‘LoginID’ Column to UNICODE STRING using
Data Conversion Transformation.
j-Extracts Name from 'LoginID' and changes modified date to recent Date--Derived
Column---->
Derived column Transformation is used to create a new column from existion data.
In this package, I have extracted ‘Name of Employee’ from ‘LoginID’ of employee using
SubString function ‘SUBSTRING(LoginID,17,LEN(LoginID)) () ‘and apart of that created a
new column ‘ Updated Date’ using function ‘(DT_DBTIMESTAMP2,3)GETDATE() ‘which
gives recent date when data is updated
![Page 11: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/11.jpg)
K-Get No of Rows--Row Count--->
Row Count Transformation counts no of rows present in dataset using variables
In this package,’ GetNoOfRows’ variable is used to store no of rows present in dataset
L- Load Data to "employee_Details" Table---OLE DB Destination--->
‘OLEDB Destination‘ Task loads data to SQL Server Table
![Page 12: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/12.jpg)
In this package, Data gets loaded to ‘[Nitil].[dbo].[Employee_Details]’ table. You can see in below image what data would get queried once you run the SELECT query.
![Page 13: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/13.jpg)
3- Extract Name as resultset from Employee Details Table for Specific National ID---Execute SQL Task---
This is a ‘execute SQL Task ‘ which executes below query by getting value of Input Parameter(NID) and
gives result as ‘Name’ of Employee
select name from [Employee_Details] where [ NationalIDNumber] = ?
![Page 14: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/14.jpg)
4. Display the Name of Employee--Script Task
It is script task which read ‘Name‘ of Employee and display it.
I mentioned VB Script below
Public Sub Main() Dim Name As String = Dts.Variables("Name").Value MsgBox("Name of Employee for National ID 998320692 is : " + Name) Dts.TaskResult = ScriptResults.Success End Sub
Apart of these I have Used 3 types of ‘event Handlers’ using ‘Execute SQL Task
Once package runs events are raised
1. OnError: this event Captures errors in Packages
In this package, I have created a table ‘PKG_Log’ with columns [ID],[PackageName],[StartTime],[EndTime],[ErrorMSG] so whenever package runs below query
will update the ‘PKG_Log’ table and give execution details.
![Page 15: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/15.jpg)
Update PKG_LOg set ErrorMSG=?
EndTime=GETDATE()
where ID=?
Here ‘PKGID’ has fixed value as ‘101’ and mapped using execute SQL Task
![Page 16: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/16.jpg)
2-Onprevaliadete—It Updates start Time of Package
T his event handler is for the OnPreValidate event. This event is raised by an executable
when its validation starts.
Once execution starts, It updates ‘PKG_LOG’ table using below query in execute SQL
Task. I already mapped ‘PKG ID ‘in execute SQl Task
Update PKG_LOg set StartTime=GetDate() where ID=?
![Page 17: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/17.jpg)
3- OnPostValidate--- This event is raised by an executable when its validation is finished.
Once execution ends, It updates ‘PKG_LOG’ table using below query using execute SQL
Task. I already mapped ‘PKG ID ‘in execute SQl Task
Update PKG_LOg set ENDTime=GetDate() where ID=?
Now, this Sample package ends here. Hope It will help to get good idea about SSIS Tasks/Package.
![Page 18: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/18.jpg)
Below you can find a small example of Script Task in SSIS
Sample : Using Script Task
1- I am going to set up a simple SSIS project. First I create 2 variables: MyName & YourName. I have scoped these at the packages level
2- After that I have dragged and dropped SCRIPT Task from “ SSIS Toolbox” to “Control Flow Pan “ which will be using a little bit of custom code
3- Now opening up the script task I have clicked on the button next to the readonlyvariables and readwritevariables areas. For t his sample, I am
setting up the MyName variable to be read only and the YourName variable to re ad-write.
![Page 19: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/19.jpg)
Note :Read Only Variable is used to Read value using variable and Read Write Variable is used read and then write the value u sing
variable(Assigning or Changing value of variable)
4- Now click on Edit Script and make a call Using Visual Basic script, like below, to access your variables. Here I am going to display the values in some message boxes. Reading from the MyName variable and overwriting the YourName variable.
Public Sub Main()
Dim MyName As String = Dts.Variables("MyName").Value
MsgBox(MyName)
Dts.Variables("YourName").Value = "Tom"
Dim YourName As String = Dts.Variables("YourName").Value
MsgBox(YourName)
Dts.TaskResult = ScriptResults.Success
End Sub
5- After writing the script, click ok on script task editorand execute the task by clicking on Green mark
![Page 20: A Complex SSIS Package](https://reader033.fdocuments.in/reader033/viewer/2022042611/58ad3ebd1a28ab8b598b513b/html5/thumbnails/20.jpg)
We will see output like below
In this output we can see MyName variable reads the name as “ Nitil” and yourName variable change name from “ Ritesh” to “ Tom”