CD Css is Sam

SQL Server Technical Article

Writer: Sandra Ward

Published: 11 2008

Applies To: SQL Server 2008

Summary: These are the accompanying notes describing the Change Data Capture for Specified Interval Package Sample available on Codeplex. The sample demonstrates the use of CDC technology in support of SSIS incremental load packages.

Change Data Capture for Specified Interval Package Sample

Copyright

The information contained in this document represents the current view of Microsoft Corporation on the issues

discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it

should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the

accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED

OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under

copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or

transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or

for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights

covering subject matter in this document. Except as expressly provided in any written license agreement

from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks,

copyrights, or other intellectual property.

2008 Microsoft Corporation. All rights reserved.

Microsoft is a registered trademark of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective

owners.

Table of ContentsGetting Started............................................................................................1

Sample Structure.........................................................................................1

The Sample Environment Test Harness......................................................................2

Package Architecture for Incremental Load................................................................2

SetupCDCSample Package............................................................................3

Initializing the Environment and Enabling Change Data Capture................................3

Execute SQL Task - Create Tables and Enable Change Data Capture...................3

Creating Capture Instances for the Source Tables................................................5

Generating Wrapper TVFs for the Query Functions...............................................6

Generating the Sample Workload...............................................................................8

Script Tasks – Apply Inserts and Updates to CDC Enabled Tables.........................8

Script Task – Mark Workload Completion..............................................................9

Initializing the Target Tables.....................................................................................10

Execute SQL Task – Create Database Snapshot..................................................10

Data Flow Tasks to Perform the Initial Load........................................................10

Setting Up the Periodic Requests for Change Data...................................................11

Execute SQL Task – Verify Capture Process is Started........................................12

Execute SQL Task – Determine Datetime Base for Initial Extraction Interval.......13

Script Task – Log Capture Not Started Message..................................................14

Extracting and Processing Change Data...................................................................16

For Loop Container - Cycle Master at 10 Second Intervals..................................17

Validating Incremental Load and Reporting Completion Status................................19

Execute SQL Task – Check for Mismatch in Replicas...........................................19

VB Script Task – Output Run Completion Status..................................................20

SetupCDCSample Package Variables........................................................................22

MasterCDC Package....................................................................................23

The Change Data Capture Database Validity Interval...............................................23

The Change Data Capture Validity Interval for a Capture Instance...........................24

A Closer Look at an SSIS Wrapper Function..............................................................25

Master Package Configurations and Variables..........................................................26

Package Configurations for Initializing Package Variables...................................26

Master Package Variables...................................................................................27

Master Package Tasks...............................................................................................28

Execute SQL Task – Check for Data.....................................................................28

Execute Package Tasks – Extract Change Data for CDC Enabled Tables.............30

Script Task – Log Extract Error............................................................................30

Script Task – Delay..............................................................................................31

Script Task – Log Extraction Complete................................................................32

Child Packages for the Change Data Capture for Specified Interval Package Sample......................................................................................................33

Child Package Variables............................................................................................35

Child Package Tasks.................................................................................................35

Script Task - Generate SQL Data Query...............................................................35

Data Flow Task – Process Change Data...............................................................37

Error Logging in the Child Packages.........................................................................40

Wrappers for CDC TVFs...............................................................................41

Instantiated Multi-Statement TVF Wrappers.............................................................41

A Closer Look at an SSIS Wrapper Function..............................................................43

Use of datetime Values in the Wrapper Signature...............................................43

Returning Column Information in the Wrapper Function.....................................45

Extracting Information from the CDC Update Mask.............................................46

Customizing Wrapper Functions for the First Extraction Interval..............................48

Running the Change Data Capture for Specified Interval Package Sample in BI Studio........................................................................................................50

Conclusion.................................................................................................52


Getting Started This document goes through the Change Data Capture for Specified Interval Package Sample in detail, describing how the Change Data Capture feature in SQL Server 2008 can be used to support ETL from an SSIS package. Code for the Change Data Capture for Specified Interval Package Sample is available through CODEPLEX. The README supplied with the sample provides detailed information on installing the sample files locally, along with the requirements needed to run the sample. This accompanying document is available on MSDN.

The Change Data Capture for Specified Interval Package Sample makes use of the databases AdventureWorks2008 and AdventureWorksDW2008, both of which are available for download on Codeplex. Follow the instructions at the download site to insure they are properly loaded in your environment.

NOTE: After loading AdventureWorks2008, execute sp_helpusers to determine whether the database user ‘dbo’ has an associated login. If the LoginName for the returned ‘dbo’ entry is NULL, the database user ‘dbo’ has been orphaned. Run the following command to associate the database user ‘dbo’ with the login ‘sa.’

exec sp_changedbowner 'sa'

The database user ‘dbo’ must be associated with a valid login in order for the Change Data Capture capture process to successfully harvest changes from the log and deposit them in the database change tables.

There are two package variables in the SetupCDCSample package that are key to successfully running the sample: SQLServerInstallPath and BasePath. SQLServerInstallPath, which defaults to c:\program files\Microsoft SQL Server\ identifies the install path for SQL Server on the local machine. If this is different in your environment, modify the package variable appropriately. BasePath, which defaults to @[SQLServerInstallPath] + “100\Samples\Integration Services\Package Samples\Change Data Capture for Specified Interval Package Sample\Change Data Capture Sample\”, identifies the standard install path of the sample relative to the SQL Server install path. If you have installed the sample elsewhere, set this path appropriately.

Sample StructureThe Change Data Capture feature of SQL Server 2008 allows the DML activity against database tables to be captured in change tables. The change data is made available to applications through table valued functions that query the change tables. This sample demonstrates the use of CDC technology from within SSIS incremental load packages to obtain source table changes to be applied to a Data Mart.

1


The Sample Environment Test HarnessA setup package is provided that both initializes the sample environment and provides a test harness for driving the data extraction process. The package begins by initializing the source tables to an initial state. It then configures the database for Change Data Capture, and creates a capture instance for each table to be tracked. The package then launches several tasks to generate DML activity against the source tables. At the same time that DML is being applied to the source tables, a database snapshot is taken by the Setup package. The snapshot is then used to provide data for an initial load of the target tables within the Data Mart. Metadata maintained in the snapshot is used to identify the starting point for the initial extraction for the incremental load.

Once the initial load completes, the setup package enters a loop that launches the master package to harvest changes for 10 second intervals. It is the action of the master package and its associated child packages to obtain change data for a specified interval and apply the changes to the target environment that represents the principle focus of the sample. The loop logic continues to monitor the progress of the DML tasks through global variables. When the loop logic determines that the DML tasks have all completed and that the window of the next extraction interval has moved beyond the period of time when the workload was applied, the loop logic terminates. On completion, SQL CHECKSUM is used to compare the source files to the replicas. Table differences, if present, are noted in an event log entry that is made to record the status of the completed run.

Package Architecture for Incremental Load The core of the sample consists of four packages: one master package and three child packages. The master package obtains the extraction interval for the incremental load from the setup package, and then verifies that the interval lies within the current Change Data Capture validity interval for the database. If the low end-point of the extraction interval is earlier in time than the minimum commit time for change table entries, the master package logs an error to the application event log and terminates. This is an indication that change table cleanup has been too aggressive, and the needed change data is no longer available in the change tables. If the high end-point of the extraction interval is later in time that the maximum commit time associated with a change table entry, the capture process has not completed the population of the change tables for the interval. In this case, the master package delays giving the capture process the opportunity to catch up before checking again. The delay loop will continue to execute until the capture process has caught up, or an iteration limit is reached and the package terminates depositing an error message in the event log. If both end-points of the extraction interval are in range, the master package launches the child packages to retrieve the change data and update the Data Mart. The master package waits for notification from all packages before writing an informational message to the application event log at the conclusion of the extraction cycle. The log entry identifies the start and end times of the extraction interval as well as other packages variables.

2


SetupCDCSample PackageThe package SetupCDCSample.dtsx sets up the test environment for the sample. It uses AdventureWorks2008 as its source database, and AdventureWorksDW2008 as the target. The database snapshot AdventureWorks2008_dbss is created for AdventureWorks2008 to provide a consistent view of the source tables for the initial load. The diagram below shows the control flow for the setup package.

Figure 1: SETUPCDCSAMPLE Package

Initializing the Environment and Enabling Change Data Capture

Execute SQL Task - Create Tables and Enable Change Data Capture

The SetupCDC Sample package begins with a single SSIS Execute SQL Task. Its purpose is to run the T-SQL script CDCSetupTables.sql to setup the environment for Change Data Capture.

3


The script begins by generally cleaning up the sample environment, removing any objects created in previous runs. This makes it straightforward to rerun the sample and still allow the created objects to endure at the end of the run.

The script then enables Change Data Capture for the database AdventureWorks2008. The Change Data Capture feature of SQL Server 2008 that allows the DML activity against database tables to be captured in change tables must initially be enabled at the database level by a member of the fixed server sysadmin role.

You can use the following T-SQL query to determine whether Change Data Capture is already enabled for a database:

SELECT is_cdc_enabled from sys.databases

WHERE name = 'AdventureWorks2008'

AND is_cdc_enabled = 1

WIthin the sample, this query is used to first determine whether Change Data Capture has already been enabled for the database. If it is, the following stored procedure is run to disable it.

exec sys.sp_cdc_disable_db

Disabling Change Data Capture at the database level will cleanup all of the Change Data Capture metadata for the database including the metadata associated with capture instances that have have previously been created for tracked tables. This allows each execution of the sample to execute from a clean test environment.

The following stored procedure is then executed to enable Change Data Capture for AdventureWorks2008:

exec sys.sp_cdc_enable_db

The script then removes preexisting tables and functions from the schema CDCSample in AdventureWorks2008. The schema CDCSample is used for the new tables that will be created to serve as the source tables for the SSIS packages used to extract change data.

The following three tables are created in the CDCSample schema of AdventureWorks2008, each mirroring an existing AdventiureWorks2008 table:

CDCSample.Customer

4


CDCSample.CreditCard

CDCSample.WorkOrder

Three multi-statement table valued functions are generated and instantiated at this time: CDCSample.fn_net_changes_Customer, CDCSample.fn_net_changes_CreditCard and CDCSample.fn_net_changes_WorkOrder. Three additional custom table valued functions are also created to handle the initial extraction interval when synchronizing to the initial load. Both sets of functions serve as wrapper functions for the CDC generated functions used to query for change data. They will be discussed in detail when the SSIS packages for extracting change data are examined.

A portion of the table data in the original AdventureWorks2008 tables is then used to initialize the source tables in the CDCSample schema. The remaining data will be used later to generate a dynamic workload when tracking is enabled.

With the source tables initialized, the script now creates three destination tables in the database AdventureWorksDW2008, one for each source table. For the purposes of the sample, the column structure of the destination tables is identical to that of the source tables. The function of the sample is to simply apply the changes associated with the source to the destination to allow the destination to reflect changes to the source in a timely fashion.

Creating Capture Instances for the Source TablesOnce the source tables are initialized, the setup script creates capture instances for each of the source tables. While the database itself must be enabled for Change Data Capture by a member of the sysadmin server role, capture instances for individual tables can be created by members of the db_owner database role. The stored procedure sys.sp_cdc_enable_table is used to create a capture instance for a source table. The following calls are made within the setup script to create capture instances for the three source tables.

exec sys.sp_cdc_enable_table

'CDCSample', 'Customer', 'Customer',

@supports_net_changes = 1, @role_name = null


'CDCSample', 'CreditCard', ' CreditCard ',



5


'CDCSample', 'WorkOrder', 'WorkOrder',


The first two parameters to the stored procedure are the schema and table name of the source table to be tracked. The third parameter is the name chosen for the associated capture instance. Any name can be chosen, but within a given database, the capture instance name must be unique. Since it is used to identify the change data associated with a given source table, it usually makes sense to name the capture instance in a manner that provides cues to its associated source table. If not specified, it will default to the schema name followed by the table name, separated by an underscore. In the sample, the tablename has been used as the capture instance name.

The parameter @supports_net_changes is used to indicate that functions to query for net changes should be generated for the capture instance. The source table must have a primary key or a defined unique index if this parameter is set to 1. Here, the term ‘net changes’ is used in a very specific way. The function returning net changes will only return a single row for each changed row for a given query window representing the final state of the row at the end of the interval. The operation returned with the row will be the one needed to correctly apply the row to the destination. In contrast, when a query for all changes is requested, a row is returned in the result set for each committed change to a row of the table.

The parameter @role_name = null is used to indicate that no gating role is begin used to restrict access to change data. In order to have access to change data, the requestor must have select access to the captured columns of the source table. In addition, if the caller is not sysadmin or db_owner and a gating role has been defined, the caller must also be a member of the gating role. By default, a gating role must be defined. If, however, the @role_name parameter is explicitly set to null when the capture instance is created, no gating role is used and select access alone is sufficient to gain access to the change data.

Generating Wrapper TVFs for the Query FunctionsAt the time Change Data Capture is enabled for a source table, the single statement Table Valued Functions (TVFs) needed to query the change table for change data are automatically generated. Functions to query for all changes are always generated, while those used to extract net changes are generated only if the @support_net_changes parameter is set to 1. The principle drawback of the generated TVFs with respect to SSIS developers is that the query interval used to identify the range for which change data is needed is Log Sequence Number (LSN) based rather than time based. Change Data Capture does, however, provide a stored procedure that will script wrapper functions for the generated TVFs that use time based rather than LSN based parameters as interval boundaries. The sample setup script makes use of this capability and instantiates wrapper stored procedures for each of the defined net changes functions after the capture instances are created.

6


The following simple stored procedure is created to generate the wrappers. The call to sp_cdc_generate_wrapper_function when no parameters are specified will return a result set that includes scripts to generate wrappers for all of the capture instances that the caller is authorized to access. The instantiation loop then selects from among those wrappers only those that wrap the net changes queries.

create procedure [CDCSample].[generate_wrappers]

as

begin

declare @wrapper_functions table

(function_name sysname, create_stmt nvarchar(max))

insert into @wrapper_functions

exec sys.sp_cdc_generate_wrapper_function

declare @stmt nvarchar(max)

declare #hfunctions cursor local fast_forward for

select create_stmt from @wrapper_functions

where function_name like 'fn_net_changes%'

open #hfunctions

fetch #hfunctions into @stmt

while (@@fetch_status <> -1)

begin

exec sp_executesql @stmt

fetch @hfunctions into @stmt

end

close @hfunctions

deallocate #hfunctions

end

The above procedure is then called after the capture instances have been created.

7


exec CDCSample.generate_wrappers

Generating the Sample Workload

Script Tasks – Apply Inserts and Updates to CDC Enabled Tables

Once the environment has been enabled for Change Data Capture three Execute SQL Tasks are launched in parallel to generate DML activity against the source tables. After each task completes, a second task is launched to generate additional activity. In total, there are six tasks that generate load: one applying inserts and one applying updates, for each of three source tables. The update load only targets table rows populated as part of the initial load. A seeded random number generator is used so that the results are reproducible. The insert load is generated from the portion of the table rows not included within the initial load. By periodically enforcing a 10 second delay between batches of inserts and updates, the load is spread across several minutes. This enforced delay makes it easier to demonstrate techniques for systematically moving the query window when data needs to be extracted periodically. Where the period for an actual ETL system is most typically 24 hours, the period used in this sample is 20 seconds. The principle used to walk the change tracking timeline, however, is the same irrespective of the size of the extraction interval.

Structurally, all of the Execute SQL Tasks that are used to generate workload are identical. They take no parameters and do not generate results sets. Each runs a SQL script to apply changes to CDC source tables. The table below shows the common SQL Statement attributes of the workload tasks.

SQL Statement Attribute Value

ConnectionType OLE DB

Connection AdventureWorks2008

SQLSourceType File connection

Below the script files for generating workload are paired with their corresponding tasks.

Execute SQL Task for Workload

SQLStatementSource

Expressions- ConnectionString

Insert Customer Table @[ScriptPath] + ’CDCCustomerInsert.sql’

Modify Customer Table @[ScriptPath] + ’CDCCustomerModify.sql’

8


Insert CreditCard Table @[ScriptPath] + ’CDCCreditCardInsert.sql’

Modify CreditCard Table @[ScriptPath] + ’CDCCreditCardModify.sql’

Insert WorkOrder Table @[ScriptPath] + ’CDC WorkOrderInsert.sql’

Modify WorkOrder Table @[ScriptPath] + ’CDC WorkOrderModify.sql’

Script Task – Mark Workload CompletionThe Script Task, Mark Workload Completion, runs after all the DML workload has been applied to the source tables. It is used to set package variables flagging the completion of the workload tasks and noting the completion time. The loop that periodically launches the master package to harvest change data monitors these package variables and terminates sample execution after the entire workload has been applied to the target environment.

The EntryPoint property which defines the first method that executes when the script task runs is MarkWorkloadCompletion. The ReadWriteVariables property allows the following package variables to be set by the script:

User::WorkloadCompleted,

User::WorkloadEndTime

The VB script code to support setting these package variables is shown below.

Public Sub MarkWorkloadCompletion()

Dim varWorkloadEndTime As Variable =

Dts.Variables("User::WorkloadEndTime")

Dim varWorkloadCompleted As Variable =

Dts.Variables("User::WorkloadCompleted")

Dts.VariableDispenser.LockForWrite("User::WorkloadEndTime")

Dts.VariableDispenser.LockForWrite("User::WorkloadCompleted")

varWorkloadEndTime.Value() = Now

varWorkloadCompleted.Value() = 1

9


Dts.Variables.Unlock()

Dts.TaskResult = ScriptResults.Success

End Sub

Initializing the Target TablesOne of the principle issues to address when applying change data to a target environment is the synchronization of the stream of change data to the initial load. Simply put, the initial load of the target reflects a snapshot of the source at some point in time. The challenge is to determine the last change represented within the snapshot in order to have a basis for determining the first incremental change that needs to be applied.

For Change Data Capture, the recommended strategy for synchronization makes use of a database snapshot created after the source tables are enabled. Use of the snapshot as the source for the initial load insures the cross table consistency of the tracked tables. More importantly, however, the metadata maintained for the snapshot allows you to precisely determine the LSN to use when you apply the first incremental load. The sample treats the first extraction interval differently than the others, using an LSN value for the low end-point and a datetime value as the high end-point. This is needed to insure that no changes are missed or repeated as a result of the synchronization process. A custom wrapper function is created to deal with this special interval.

Execute SQL Task – Create Database SnapshotAs described above, a database snapshot is created to be used as the source for the initial load of the target tables. The snapshot is created concurrently with the execution of the workload tasks to demonstrate that the content of the target tables subsequent to the initial load is not explicitly predetermined. Snapshot metadata will be used to determine the appropriate starting point for the incremental loads that follow.

The package variable User::CreateSnapshot contains the statement used to create the database snapshot. The content of the variable is derived from an expression that references the package variable User::BasePath to construct the path where the snapshot file is to be located.

"CREATE DATABASE @AdventureWorks2008_dbss ON

(NAME = AdventureWorks2008_Data, FILENAME = '" +

@[BasePath] + "AdventureWorks2008_data.ss')

AS SNAPSHOT OF AdventureWorks2008;"

10


The SQL statement has no input parameters and no result set is returned.

Data Flow Tasks to Perform the Initial LoadOnce the snapshot is created it can be queried concurrently to obtain a consistent initial load for the target tables. The sample makes use of data flow tasks to perform the initial load, but any technology that uses the snapshot as the source for the data can be used. Use of the snapshot as the source guarantees the cross table consistency of the target tables that are loaded.

The three Data Flow Tasks used to perform the initial load of the target tables are all structured identically. What differs among the three tasks are the names of the source and destination tables that provide the end points of the data flow. Each consists of an OLE DB source referencing the database snapshot, and an OLE DB destination that references AdventureWorksDW2008, the target database. All of the columns of the source table are output to the data flow. In general, column names associated with the source are mapped to identical names in the destination. Columns defined as computed columns, however, are ignored.

Below is the association between individual Data Flow Tasks and the source and destination tables.

OLE DB Source Attribute Value

OLE DB connection manager AdventureWorks2008_dbss

Data access mode Table or view

OLE DB Destination Attribute Value

OLE DB connection manager AdventureWorksDW2008

Data access mode Table or view

The table below shows the common attributes of the OLE DB source and destination Data Flow Components:

Data Flow Task Source/Destination Table

Load Target Customer Table From Snapshot CDCSample.Customer

Load CreditCard Table From Snapshot CDCSample.CreditCard

Load WorkOrder Table From Snapshot CDCSample.WorkOrder

11


Setting Up the Periodic Requests for Change Data

Once the initial load of the target tables has completed the sample verifies that the capture process responsible for harvesting changes from the transaction log and depositing them in the change tables is active. If there are no entries in cdc.lsn_time_mapping, this is an indication that the capture process did not auto-start and that SQL Agent is not running. If an active capture process is not detected, the sample will log an error in the event log and terminate. If the presence of an active capture process is verified, metadata from the database snapshot is used to determine the LSN that will anchor the initial incremental load. A base time for computing the high end-point for the initial load is also determined at this time.

Execute SQL Task – Verify Capture Process is Started

If there are no entries in cdc.lsn_time_mapping, this is an indication that the capture process did not auto-start when Change Data Capture was enabled. This typically occurs because SQL Agent is not running. This condition is detected here so that an event log message can be posted to indicate the failure to auto-start and the run can be terminated. The package variable User::CaptureStarted is used to report this condition.

The following SQL Statement is defined as Direct input:

declare @start_time datetime, @capturestarted bit

select @start_time = min(tran_end_time)

from cdc.lsn_time_mapping

if @start_time is null

begin

select @capturestarted = 0

end

else

begin

select @capturestarted = 1

end

select @capturestarted as CaptureStarted

The SQL statement has no input parameters.

12


The result set is defined as a single row with the column CaptureStarted:

Result Name Variable Name

CaptureStarted User::CaptureStarted

If there are no entries in the cdc.lsn_time_mapping table, the script task Log Capture Not Started Message is called to log an informational message indicating that the capture process did not auto start, which requires SQL Agent to be running.

Execute SQL Task – Determine Datetime Base for Initial Extraction Interval

After the initial load of the target tables the sample uses an Execute SQL task to determine the end-points for the first extraction interval. Subsequent extraction intervals will be based off the initial interval, at ten second intervals. For the initial extraction interval, the lower bound is expressed as an LSN value to insure that no incremental changes are lost between the snapshot and the first incremental load. This LSN value is determined by using the function sys.fn_cdc_dbsnapshotLSN to retrieve the last commit LSN from the database snapshot. This value is saved as the package variable LastLSN.

Once the lower boundary expressed as an LSN value is determined, the sample uses information in the cdc.lsn_time_mapping table to assign an approximate datetime value to this LSN. It is this approximate datetime value that is used to compute the high end-point for the initial extraction interval. The script has a built in delay to wait until the capture process has processed all LSN values through the determined LastLSN value. After this is insured, the minimum tran_end_time of the entries in cdc.lsn_time_mapping with start_lsn value greater than LastLSN is then determined. This value is saved in the package variable ExtractEndTime. Later in the loop that cycles the master package every 10 seconds, this value will be used as the base from which the high-end point of the first extraction interval is determined.


declare @command nvarchar(max), @database_name nvarchar(1000)

,@lastLSN binary(10), @max_lsn binary(10), @start_time datetime

,@lastLSNstr nvarchar(42)

exec sys.sp_cdc_dbsnapshotLSN 'AdventureWorks2008_dbss'

,@lastLSN output, @lastLSNstr output

13


select @max_lsn = sys.fn_cdc_get_max_lsn()

while (@lastLSN >= @max_lsn)

begin

waitfor delay '00:00:10'

select @max_lsn = sys.fn_cdc_get_max_lsn()

end

select @start_time = min(tran_end_time)


where start_lsn > @lastLSN

select @start_time = convert(nvarchar(40),@start_time,20)

select @start_time as ExtractStartTime, @start_time as

ExtractEndTime, @lastLSNstr as LastLSN


The result set is defined as a single row with the columns ExtractEndTime and LastLSN.


ExtractEndTime User:: ExtractEndTime

LastLSN User:: LastLSN

Script Task – Log Capture Not Started MessageThe Script Task Log Capture Not Started Message, is used to log an information message to the Windows event log to indicate that the capture process did not auto-start.

The precedence constraint that defines the workflow between the Execute SQL Task Determine Extraction Start Time and this task is the following:

Completion and

@CaptureStarted == false evaluates to TRUE

14


The EntryPoint property which defines the first method that executes when the script task runs is LogCaptureNotStartedMessage. The ReadOnlyVariables property setting makes the following package variables available to the script:

System::PackageName,

System::StartTime,

User::CaptureStarted

The VB script code to support writing to the event log is shown below.

Public Sub LogCaptureProcessNotStartedMessage()

Dim varPackageName As Variable =

Dts.Variables("System::PackageName")

Dim varStartTime As Variable = Dts.Variables("System::StartTime")

Dim varCaptureStarted As Variable =

Dts.Variables("User::CaptureStarted")

Dim sLog As String

Dim sEventMessage As String

Dim sMachine As String

Dim sSource As String

Dts.VariableDispenser.LockForRead("System::PackageName")

Dts.VariableDispenser.LockForRead("System::StartTime")

Dts.VariableDispenser.LockForRead("User::CaptureStarted")

sLog = "Application"

sSource = varPackageName.Value().ToString

sEventMessage = "The CDC Capture Process was not started." _

& "Make certain that SQL Agent is running." _

& Chr(10) _

& "=============================================" & Chr(10) _

& "The Package: " + varPackageName.Value().ToString _

& Chr(10) _

15


& "Started: " & varStartTime.Value().ToString _

& Chr(10) _

& "Current Time:" & System.DateTime.Now _

& Chr(10) _

& "=============================================" _

& Chr(10) _

& "Capture Started: " & varCaptureStarted.Value().ToString

sMachine = "."

Dim ELog As New EventLog(sLog, sMachine, sSource)

ELog.WriteEntry(sEventMessage, EventLogEntryType.Information)



End Sub

The event log message posted will be similar to that shown in Figure 2:

16


Figure 2: Event Log Message Signaling Capture Process Not Started

Extracting and Processing Change DataAfter the boundaries of the first incremental load have been determined, the setup package enters a loop which invokes the master package to harvest change data at 10 second intervals until all the changes from the generated workload have been applied to the target.

For Loop Container - Cycle Master at 10 Second Intervals

The Loop Container logic itself is designed to execute the container tasks until all of the changes in the generated workload have been extracted and applied to the target and then terminate. Two conditions need to be satisfied in order for the loop to terminate. First the workload generation tasks need to have all completed. This is determined by checking the package variable WorkloadCompleted, which is set by the Mark Workload Completion task that runs after all of the load generation tasks run. Second, all of the

17


changes associated with the workload must have been applied to the target tables. This is determined by checking the proposed start time for the next extraction interval. If this value is greater than the time when the Workload Completion Task ran, then we are assured that the extraction intervals already processed fully cover the time when changes were applied to the source tables and the run can terminate.

The loop container also maintains a package variable that is used as an interval counter. It initializes the variable to 0 and then increments it by one during each iteration. Since the interval counter is a package variable, it can be passed to the master package launched from within the loop.

For Loop Properties

Property Name Property Value

InitExpression @IntervalID = 0

EvalExpression !((@WorkloadCompleted == 1) && (@ExtractStartTime > @WorkloadEndTime))

AssignExpression @IntervalID = @IntervalID + 1

Within the for loop container are two tasks - a Script Task that computes the new end-points for the extraction interval and an Execute Package Task that invokes the master package driving the extraction cycle.

Script Task - Set Extract IntervalThe Script Task Set Extract Interval is used to determine the end-points for the next query window. It uses the high end-point of the previous window as the low end-point of the current window. It then adds 10 seconds to the new low end-point to obtain the new high end-point. The computed end-points are returned as a single row result set and deposited in package variables User::ExtractStartTime and User::ExtractEndTime.

The EntryPoint property which defines the first method that executes when the script task runs is SetExtractInterval. The ReadWriteVariables property setting makes the following package variables available to the script:

User::ExtractStartTime

User::ExtractEndTime

18


The VB script code to support computing the query interval end-points is show below.

Public Sub SetExtractInterval()

Dim varStartTime As Variable =

Dts.Variables("User::ExtractStartTime")

Dim varEndTime As Variable =

Dts.Variables("User::ExtractEndTime")

Dts.VariableDispenser.LockForWrite("User::ExtractStartTime")

Dts.VariableDispenser.LockForWrite("User::ExtractEndTime")

varStartTime.Value() = varEndTime.Value()

varEndTime.Value() =

DateAdd(DateInterval.Second, 10, varStartTime.Value())



End Sub

Execute Package Task – Run Master to Extract DataThe Execute Package Task, Run Master to Extract Data, functions as the test harness for the ETL extraction cycle. Its’ sole function is to launch the master package that will in turn launch the individual extraction packages for the individual tables.

Validating Incremental Load and Reporting Completion Status

Once the generated workload has been applied to the target environment, the state of the replicas should match that of the source tables. The SQL Script Task Check for Mismatch in Replicas uses SQL CHECKSUM to verify the contents of the source and target tables match, setting package variables to record the status of the run. Finally the VB Script task Output Run Completion Status is launched to output the run status to the event log.

19


Execute SQL Task – Check for Mismatch in ReplicasThe Execute SQL Task Check for Mismatch in Replicas is used to determine whether the replicas created match the source tables. CHECKSUM is used to compare the table contents and package variables are set for each of the tracked tables to indicate whether differences were detected.


declare @CustomerMismatch int,

@CreditCardMismatch int,

@WorkOrderMismatch int,

@Checksum bigint,

@ChecksumDW bigint

select @CustomerMismatch = 0,

@CreditCardMismatch = 0,

@WorkOrderMismatch = 0,

@Checksum = 0,

@ChecksumDW = 0

select @Checksum = CHECKSUM(*)

from AdventureWorks2008.CDCSample.Customer

select @ChecksumDW = CHECKSUM(*)

from AdventureWorksDW2008.CDCSample.Customer

if (@Checksum <> @ChecksumDW)

begin

set @CustomerMismatch = 1

20


end


from AdventureWorks2008.CDCSample.CreditCard


from AdventureWorksDW2008.CDCSample.CreditCard


begin

set @CreditCardMismatch = 1

end


from AdventureWorks2008.CDCSample.WorkOrder


from AdventureWorksDW2008.CDCSample.WorkOrder


begin

set @WorkOrderMismatch = 1

end

select @CustomerMismatch as CustomerMismatch,

@CreditCardMismatch as CreditCardMismatch,

@WorkOrderMismatch as WorkOrderMismatch


21


The result set is defined as a single row with the columns CustomerMismatch, CreditCardMismatch, and WorkOrder Mismatch.


CustomerMismatch User::CustomerMismatch

CreditCardMismatch User::CreditCardMismatch

WorkOrderMismatch User::WorkOrderMismatch

VB Script Task – Output Run Completion StatusThe VB Script Task Output Run Completion Status writes a status entry to the Event Log indicating the result of the comparisons made between the source tables and the target tables that were updated using Change Data Capture.

The EntryPoint property which defines the first method that executes when the script task runs is OutputRunCompletionStatus.. The ReadOnlyVariables property setting makes the following package variables available to the script:

System::PackageName

System::StartTime

User::CreditCardMismatch

User::CustomerMismatch

User::WorkOrderMismatch

The VB script code to log completion status is shown below. Note that a comparison failure for any of the tables will cause the task to complete with a failure status.

Public Sub OutputRunCompletionStatus()

22




Dim varStartTime As Variable =

Dts.Variables("System::StartTime")

Dim varCustomerMismatch As Variable =

Dts.Variables("User::CustomerMismatch")

Dim varCreditCardMismatch As Variable =

Dts.Variables("User::CreditCardMismatch")

Dim varWorkOrderMismatch As Variable =

Dts.Variables("User::WorkOrderMismatch")

Dim sLog As String




Dim sCustomer As String

Dim sCreditCard As String

Dim sWorkOrder As String

sCustomer = "Customer replica is identical."

sCreditCard = "CreditCard replica is identical."

sWorkOrder = "WorkOrder replica is identical."



Dts.VariableDispenser.LockForRead("User::CustomerMismatch")

Dts.VariableDispenser.LockForRead("User::CreditCardMismatch")

23


Dts.VariableDispenser.LockForRead("User::WorkOrderMismatch")


If varCustomerMismatch.Value = 1 Then

sCustomer = "Customer replica does not match source."

Dts.TaskResult = ScriptResults.Failure

End If

If varCreditCardMismatch.Value = 1 Then

sCreditCard = "CreditCard replica does not match source."


End If

If varWorkOrderMismatch.Value = 1 Then

sWorkOrder = "WorkOrder replica does not match source."


End If



sEventMessage = "CDC SSIS Sample Completion Status" _

& Chr(10) _

& "=============================================" & Chr(10) _


& Chr(10) _


& Chr(10) _


& Chr(10) _

24


& "=============================================" & Chr(10) _

& "Customer table: " & sCustomer _

& Chr(10) _

& "=============================================" & Chr(10) _

& "CreditCard table: " & sCreditCard _

& Chr(10) _

& "=============================================" & Chr(10) _

& "WorkOrder table: " & sWorkOrder _

& Chr(10) _

& "============================================="

sMachine = "."




End Sub

SetupCDCSample Package VariablesThe following package variables are defined for the setup package:

Name Data Type Description and Default Value

BasePath String Install path for Change Data Capture for Specified Interval Package Sample packages

Default: @[SQLServerInstallPath] + "100\\Samples\\Integration Services\\Package Samples\\Change Data Capture for Specified Interval Package Sample\\

25


Change Data Capture Sample\\"

ScriptPath String Install path for Change Data Capture for Specified Interval Package Sample scripts

Default: @[BasePath] + "Scripts\\"

SQLServerInstallPath String Install path for SQL Server

Default: c:\program files\Microsoft SQL Server\

CaptureStarted Boolean Flag indicating capture process has started

Default: False

CreateSnapshot String SQL statement to create database snapshot

Default: "CREATE DATABASE AdventureWorks2008_dbss ON

( NAME = AdventureWorks2008_Data, FILENAME = '" + @[BasePath] + "AdventureWorks2008_data.ss' )

AS SNAPSHOT OF AdventureWorks2008;"

ExtractEndTime datetime End time of next extraction

Default: 5/6/2007 8:54 AM

ExtractStartTime datetime Start time of next extraction

Default: 5/6/2007 8:54 AM

IntervalID Int32 Extraction Interval identifier

Default: 0

LastLSN String LSN anchor for first extraction

Default: 0x00000000000000000000

WorkloadCompleted Int32 0 if Customer replica is identical at the end of the run: 1 if validation detected a mismatch

Default: 0

WorkloadEndTime datetime Time of workload completion

Default: 2/21/2008 12:31 PM

CustomerMismatch Int32 0 if Customer replica is identical at the end of the run: 1 if validation detected a mismatch

Default: 0

CreditCardMismatch Int32 0 if CreditCard replica is identical at the end of the run: 1 is validation detected a mismatch

Default: 0

WorkOrderMismatch Int32 0 if WorkOrder replica is identical at the end of the run: 1 is validation detected a mismatch

Default: 0

26


MasterCDC PackageThe package MasterCDC is responsible for launching all of the individual packages used to extract change data for the SQL Server source tables and apply the changes to the Data Mart. The principle task of the master package is to verify that the extraction interval passed from the parent lies with the current Change Data Capture database validity interval.

The Change Data Capture Database Validity Interval

The Change Data Capture validity interval for a database is simply the time interval for which change data is currently available for its capture instances. In principle, it begins when the first capture instance is created for a database table, and extends forward in time to the present. In practice, the validity interval is a moving window just as the extraction interval is a moving window. Change data deposited in tables would grow unmanageably if it wasn’t periodically and systematically pruned. By default, only three days of data are retained in the change tables. Hence, the period of time covered by the validity interval is typically 72 hours.

While the cleanup process works to move the low end-point of the validity interval to the right on the Change Data Capture timeline, the capture process does the same for the high end-point. Since the capture process extracts change data from the transaction log, there is a built in latency between the time that a change is committed to a source table and the time that the change appears within its associated change table. While this latency is typically small, it is nevertheless important to remember that change data is not available until the capture process has processed the related log entries.

For each transaction that results in one or more entries appearing within database change tables, an entry is logged to the cdc.lsn_time_mapping table. Each entry in the mapping table contains both a commit Log Sequence Number or commit LSN, and a transaction commit time (columns start_lsn and tran_end_time respectively.) Change Data Capture uses the table cdc.lsn_time_mapping to identify the current bounds of the database validity interval, with the smallest and largest commit times represented in the entries of cdc.lsn_time_mapping table denoting the low and high end-points of the validity interval for the database.

The Change Data Capture Validity Interval for a Capture Instance

While it is often the case that the database validity interval and an individual capture instance’s validity interval will coincide, this is not always true. The validity interval of the capture instance begins when the capture process recognizes the capture instance and begins logging associated changes to its change table. As a result, if capture instances are created at different times, each will initially have a different low end-point. The start_lsn column of the result set returned by sys.sp_cdc_help_change_data_capture shows the current low end-point for each defined capture instance. When the cleanup

27


process cleans up change table entries, it adjusts the start_lsn values for all capture instances to reflect the new low water mark for available change data. Only those capture instances with start_lsn values currently less than the new low water mark are adjusted. Over time, if no new capture instances are created, the validity intervals for all individual instances will coincide with the database validity interval.

The validity interval is important to consumers of change data because the extraction interval for a request must be fully covered by the current Change Data Capture validity interval for the capture instance. If the low end-point of the extraction interval is to the left of the low end-point of the validity interval, there could be missing change data due to aggressive cleanup. If the high end-point of the extraction interval is to the right of the high end-point of the validity interval, the capture process has not yet processed through the time period represented by the extraction interval and change data could also be missing. This relationship is illustrated in the diagram below.

Figure 3: Change Data Capture Validity Intervals

A Closer Look at an SSIS Wrapper FunctionIt is important to note that the CDC query functions will fail if the request interval is not fully covered by the validity interval. In this sample, validity checks are systematically performed up front in the master package, so that the recoverable case, where the capture process simply needs to catch up, can be dealt with automatically.

28


The first task of the master package is to verify the extraction interval that it is passed. If the low end–point of the extraction interval is outside the validity interval, the package logs an error to the event log and exits with error status. If the high end-point of the extraction interval identifies a time that is ahead of current processing by the CDC capture process, the master package will delay to allow the capture process to catch up. Once the extraction interval is in range, the master package launches the child packages. Each child package picks up the datetime values defining the extraction interval from its parent through Package Configurations. When all client packages have completed, the master package logs an event message indicating completion.

The control flow diagram for the master package is shown below.

Figure 4: Master Package for Incremental Data Extraction

29


Master Package Configurations and Variables

Package Configurations for Initializing Package Variables

The master package uses Package Configurations to initialize several runtime parameters. The table below shows the parent variables that are used to initialize local package variables of the master package.

Configuration Name

Configuration Type

Configuration String Target Object Target Property

Configure ExtractStartTime

Parent Package Variable

User::ExtractStartTime ExtractStartTime Value

Configure ExtractEndTime


User::ExtractEndTime ExtractEndTime Value

Configure IntervalID


User::IntervalID IntervalID Value

Configure Last LSN Parent Package Variable

User::LastLSN LastLSN Value

Configure Base Path


User::BasePath BasePath Value

Master Package VariablesThe following package variables are defined for the master package:

Name Data Type Scope Description

BasePath String MasterCDC Path where Change Data Capture for Specified Interval Package Sample packages are installed

DataReady Int32 MasterCDC Code indicating query status

0 = Need to wait for capture process

1 = Start time predates validity interval

2 = Ready for data query ( interval > 1)

3 = Ready for first data query

30


5 = Timeout ceiling reached waiting for capture process

Delay Int32 MasterCDC milliseconds to delay before rechecking if capture process has caught up

ExtractStartTime datetime MasterCDC Start time of extraction interval ( Interval > 1)

ExtractEndTime datetime MasterCDC End time of extraction interval

IntervalID Int32 MasterCDC Interval ID

LastLSN String MasterCDC LSN anchor used for first extraction interval

TimeoutCeiling Int32 MasterCDC Number of cycles to delay waiting for capture process to catch up prior to terminating with error

TimeoutCount Int32 MasterCDC Number of delays that have already been invoked

Master Package Tasks

Execute SQL Task – Check for DataThe Execute SQL Task, Check for Data, is used by the master package to determine how to proceed based upon the relationship of the requested extraction interval to the current Change Data Capture database validity interval and the current timeout count. It assigns one of 5 possible values to the package variable DataReady to indicate which of five possible conditions holds. It requires an OLE DB connection to the source database.


declare @DataReady int, @TimeoutCount int

if not exists

( select tran_end_

31



where tran_end_time > ? )

select @DataReady = 0

else if ? = 0


else if not exists

( select tran_end_time


where tran_end_time <= ? )


else select @DataReady = 2

select @TimeoutCount = ?

if (@DataReady = 0

select @TimeoutCount = @TimeoutCount + 1

else

select @TimeoutCount = 0

if (@TimeoutCount > ?) select @DataReady = 5

select @DataReady as DataReady, @TimeoutCount as TimeoutCount

The SQL statement has five input parameters:

Variable Name Direction Data Type Parameter Name

32


User::ExtractEndTime Input DATE 0

User::IntervalID Input SHORT 1

User::ExtractStartTime Input DATE 2

User::TimeoutCount Input SHORT 3

User::TimeoutCeiling Input SHORT 4

The result set is defined as a single row with the columns DataReady and TimeoutCount:


DateReady User::DataReady

TimeoutCount User::TimeoutCount

The SQL query first determines whether there are any entries in the cdc.lsn_time_mapping table that are later than the requested end of the extraction interval. If there are none, this means that the capture process has not yet processed all the changes in the request interval and DataReady is set to 0. If the capture process is caught up, the Interval ID is next checked. If it is set to 0, this is the first interval to be processed which requires special treatment. DataReady in this case is assigned a value of 3 to indicate this is the first interval. If this is not the first interval, a check is then made to verify that the starting point for the extraction interval is not smaller than all existing entries in the cdc.lsn_time_mapping table. DataReady in this case is set to 1, indicating the extraction interval is invalid and cannot be automatically corrected. Finally, if the starting point of the interval is not outside the validity interval, DataReady is set to 2 indicating that change data can be extracted from change tables for the interval. Once DataReady has been determined, the query determines if the master package is in a delay loop waiting for the capture process to catch up. If DataReady is 0, the timeout counter is incremented and then checked to see if the configured number of wait intervals has been exhausted. If the counter exceeds the ceiling currently set to 20,

33


@DataReady is set to 5 to indicate the wait for the capture process has timed out. Note that whenever a non-zero value is determined for @DataReady, the timeout counter is reset to 0.

After execution of the Execute SQL Task Check for Data by the package, there are three possible paths that can be taken: If DataReady is 0 the required changes have not yet been propagated to the change tables. In this case, the Script Task Delay executes to wait for a period of time to allow the capture process to catch up. If DataReady is 1, the low end point of the extraction interval is outside the CDC Change Data Capture validity interval. In this case, an error is logged by the VB Script Task Log Extract Error and the package terminates, since there is no automatic recovery if there is a possibility for data loss. Similarly, if DataReady is 5 the allowable number of delays has been exhausted and the Script Task Log Extract Error is called to log the timeout error. Finally, if DataReady is 2 or 3, all changes through the indicated ExtractEndTime have been deposited in change tables and it is possible to allow the data extraction packages to gather change data. In this case, the three data extraction packages are launched.

Execute Package Tasks – Extract Change Data for CDC Enabled Tables

The Execute Package Tasks Extract Cutomer Data, Extract CreditCard Data and extract WorkOrder Data launch the individual extraction packages for the individual tables.

Script Task – Log Extract ErrorThe Script Task, Log Extract Error is used to log an error to the Windows event log.

The precedence constraint that defines the workflow between the Execute SQL Task Check Data and this task is the following:

Success Completion Code and

@DataReady == 1 || @DataReady == 5 evaluates to TRUE

The EntryPoint property which defines the first method that executes when the script task runs is LogExtractError. The ReadOnlyVariables property setting makes the following package variables available to the script:

User::DataReady

System::ExecutionInstanceGUID,

User::ExtractStartTime,34



System::StartTime

The VB script code to support writing to the event log is shown below.

Public Sub LogExtractError()



Dim varStartTime As Variable = Dts.Variables("System::StartTime")

Dim varInstanceID As Variable =

Dts.Variables("System::ExecutionInstanceGUID")

Dim varExtractStartTime As Variable =

Dts.Variables("User::ExtractStartTime")

Dim varDataReady As Variable = Dts.Variables("User::DataReady")

Dim sLog As String




Dim iDataReady As Integer



35


Dts.VariableDispenser.LockForRead("System::ExecutionInstanceGUID")

Dts.VariableDispenser.LockForRead("User::ExtractStartTime")

Dts.VariableDispenser.LockForRead("User::DataReady")



iDataReady = varDataReady.Value()

If iDataReady = 1 Then

sEventMessage = "Start Time Error"

Else

sEventMessage = "Timeout Error"

End If

sEventMessage = sEventMessage _

& Chr(10) _

& "=============================================" & Chr(10) _


& Chr(10) _


& Chr(10) _


& Chr(10) _

& "=============================================" _

& Chr(10) _

& "Extract Start Time: " _

& varExtractStartTime.Value().ToString _

36


& Chr(10) _

& "Execution GUID: " & varInstanceID.Value().ToString

sMachine = "."


ELog.WriteEntry(sEventMessage, EventLogEntryType.Error)



End Sub

Script Task – DelayThe Script Task Delay is used to delay for a given interval of time.

The precedence constraint that defines the workflow between the Execute SQL Task Check Data and this task is the following:

Success Completion Code and

@DataReady == 0 && @TimeoutCount <=@TimeoutCeiling evaluates to TRUE

The EntryPoint property which defines the first method that executes when the script task runs is Delay. The ReadOnlyVariables property setting makes the following package variables available to the script:

User::Delay

The VB script code to support delaying for a period of time is shown below.

Public Sub Delay()

37


Dim varDelay As Variable = Dts.Variables("User::Delay")

Dim iDelay As Integer

Dts.VariableDispenser.LockForRead("User::Delay")

iDelay = varDelay.Value()

Threading.Thread.Sleep(iDelay)



End Sub

Script Task – Log Extraction CompleteThe Script Task, Log Extraction Complete is used to log a completion message to the Windows event log.

The precedence constraint that defines the workflow between the three Execute Package Tasks is Completion.

The EntryPoint property that defines the first method that executes when the script task runs is LogExtractionCompletion. The ReadOnlyVariables property setting makes the following package variables available to the script:

User::DataReady,

System::ExecutionInstanceGUID,

User::ExtractEndTime,

User::ExtractStartTime,

User::IntervalID,

38



System::StartTime

The VB script code to support writing the completion notification to the event log is shown below.

Public Sub LogExtractionCompletion()

Dim varPackageName As Variable = Dts.Variables("PackageName")

Dim varStartTime As Variable = Dts.Variables("StartTime")

Dim varInstanceID As Variable =

Dts.Variables("ExecutionInstanceGUID")

Dim varExtractStartTime As Variable =

Dts.Variables("ExtractStartTime")

Dim varExtractEndTime As Variable =

Dts.Variables("ExtractEndTime")

Dim varIntervalID As Variable = Dts.Variables("IntervalID")

Dim varDataReady As Variable = Dts.Variables("DataReady")

Dim sLog As String




Dts.VariableDispenser.LockForRead("PackageName")

Dts.VariableDispenser.LockForRead("StartTime")

Dts.VariableDispenser.LockForRead("ExecutionInstanceGUID")

Dts.VariableDispenser.LockForRead("ExtractStartTime")

39


Dts.VariableDispenser.LockForRead("ExtractEndTime")

Dts.VariableDispenser.LockForRead("IntervalID")

Dts.VariableDispenser.LockForRead("DataReady")



sEventMessage = "Extract Complete" _

& Chr(10) _

& "=============================================" & Chr(10) _


& Chr(10) _


& Chr(10) _


& Chr(10) _

& "=============================================" _

& Chr(10) _

& "Extract Start Time: " & varExtractStartTime.Value().ToString _

& Chr(10) _

& "Extract End Time: " & varExtractEndTime.Value().ToString _

& Chr(10) _

& "Interval ID: " & varIntervalID.Value().ToString _

& Chr(10) _

& "Data Ready: " & varDataReady.Value().ToString _

& Chr(10) _

& "Execution GUID: " & varInstanceID.Value().ToString

40


sMachine = "."





End Sub

Child Packages for the Change Data Capture for Specified Interval Package Sample

The sample child packages all use instantiated wrapper functions to query for change data for all extraction intervals after the first interval. For the first interval, they use a customized version of the generated wrapper function that allows the low end-point of the extraction interval to be expressed as an LSN value as opposed to a datetime value. The datetime boundaries of the extraction interval, as well as the LSN boundary used for the first interval, are obtained from parent package variables when the packages are launched. The packages also obtain a flag indicating whether this is the initial extraction interval so that the first interval can be handled as a special case.

The basic control flow for the packages is straightforward. A Script Task is called first to construct the SQL query that will be used to query for change data. Control is then passed to the Data Flow Task to request and process the change data. The data flow task uses an OLE DB source component to perform the query, directing the returned result set to a conditional split transformation. The conditional split uses the operation returned in each result set row to direct the rows to appropriate transformations: Deletes and updates are sent to OLE DB command transformations, while inserts are directed to an OLE DB destination.

The control and data flow for these child packages is shown in the figure below.

41


Figure 5: Child Packages Using Multi-Statement TVFs for Data Access

The Configurations used by the child extract packages use package variables from the launching master package to initialize their own package variables.

Configuration Name

Configuration Type

Configuration String Target Object

Target Property

Configure Data Ready


User::DataReady DataReady Value

Configure End Time


User::ExtractStartTime StartTime Value

Configure Start Time


User::ExtractEndTime StartTime Value

Configure Last LSN


User::LastLSN LastLSN Value

Child Package VariablesThe following package variables are defined for the child packages:

Name Data Type Scope Description42


DataReady Int32 CDCWorkOrderExtract Code indicating query status

0 = Need to wait for capture process

1 = Start time predates validity interval

2 = Ready for data query ( interval > 1)

3 = Ready for first data query

5 = Timeout ceiling reached waiting for capture process

ExtractStartTime datetime CDCWorkOrderExtract Start time of extraction interval ( Interval > 1)

ExtractEndTime datetime CDCWorkOrderExtract End time of extraction interval

LastLSN String CDCWorkOrderExtract LSN anchor used for first extraction interval

SQLDataQuery String CDCWorkOrderExtract Query to use to obtain change data

Child Package Tasks

Script Task - Generate SQL Data QueryThe Execute SQL Task Generate SQL Query is used to set up the query for change data. It allows us to work around the inability to pass parameters directly to table valued functions in an OLE DB source. The parameters are passed to the VB script code which is used to compose the query string and return the desired select statement in the package variable SQLDataQuery.

One additional issue bears mentioning. Our general strategy for systematically processing a stream of change data is to use the high end-point of the previous interval to determine the low end-point of the subsequent interval, and to compute a new high end-point based upon the needs of the application environment. This strategy works well for all intervals except the initial interval, when there is no previous interval.

In general, the destination and source will not be synchronized when a decision is made to use CDC technology to apply incremental loads and the first task will be to identify an anchor for the destination. The anchor is defined as an LSN lying within the Change Data Capture validity interval of a capture instance such that (1) All changes with start LSN values up to and including that anchor are already reflected in the destination and (2) All changes with start LSN values greater than the anchor have yet to be applied. Once the anchor is determined, it is used to explicitly set the LSN boundary for the initial extraction.

43


For this sample, a database snapshot provides data for the initial load, allowing an appropriate anchor LSN for the first extraction to be determined directly from snapshot metadata. While use of a database snapshot in preparing the initial load is not a requirement, it greatly simplifies the synchronization process.

The Script Task Generate SQL Data Query is used to generate the query to be used to extract change data. It uses the passed package variable DataReady to determine whether the query is for the first interval or for a subsequent interval. If it is the first interval, it constructs a call to the function fn_net_changes_WorkOrder_First which takes an LSN value as its first parameter. Otherwise, it constructs a call to the function fn_net_changes_WorkOrder which uses two datetime values as the interval end-points.

The EntryPoint property that defines the first method that executes when the script task runs is GenerateSQLQuery. The ReadOnlyVariables property setting makes the following package variables available to the script:

User::DataReady,

User::EndTime,

User::LastLSN,

User::StartTime

The ReadWriteVariables property setting allows the following package variableto be set by the script:

User::SQLDataQuery

The VB script code to support constructing the data query is shown below.

Public Sub GenerateDataQuery()

Dim varStartTime As Variable = Dts.Variables("User::StartTime")

Dim varEndTime As Variable = Dts.Variables("User::EndTime")

Dim varDataReady As Variable = Dts.Variables("User::DataReady")

Dim varSQLDataQuery As Variable =

Dts.Variables("User::SQLDataQuery")

44


Dim varLastLSN As Variable = Dts.Variables("User::LastLSN")

Dim iDataReady As Integer

Dim sStartTime As String

Dim sEndTime As String

Dim sLastLSN As String

Dim dStartTime As DateTime

Dim dEndTime As DateTime

Dts.VariableDispenser.LockForRead("User::StartTime")

Dts.VariableDispenser.LockForRead("User::EndTime")

Dts.VariableDispenser.LockForRead("User::DataReady")

Dts.VariableDispenser.LockForWrite("User::SQLDataQuery")

Dts.VariableDispenser.LockForWrite("User::LastLSN")

iDataReady = varDataReady.Value()

dStartTime = varStartTime.Value()

sStartTime = dStartTime.ToString

("G", DateTimeFormatInfo.InvariantInfo)

dEndTime = varEndTime.Value()

sEndTime = dEndTime.ToString

("G", DateTimeFormatInfo.InvariantInfo)

sLastLSN = varLastLSN.Value().ToString

If iDataReady = 2 Then

varSQLDataQuery.Value =

"select * from dbo.fn_net_changes_WorkOrder('" _

45


& sStartTime & "','" & sEndTime & "', 'all')"

Else

varSQLDataQuery.Value =

"select * from dbo.fn_net_changes_WorkOrder_First('" _

& sLastLSN & "','" & sEndTime & "', 'all')"

End If



End Sub

Data Flow Task – Process Change DataThe Data Flow Task Process Change Data calls a table valued function to query the source for change data and then applies the change data returned in the result set to the destination.

OLE DB SourceThe query used by these child processes is obtained from a package variable and makes use of an instantiated wrapper function. (Wrapper functions are discussed in detail at the end of this document.)

Data Access Mode SQL Command from variable

Variable Name User::SQLDataQuery

The result set returned by the query includes the metadata column CDC_OPERATION that identifies the operation to be used when applying the change to the destination. A conditional split transformation is used to direct the rows to one of three possible components based upon the following defined conditions:

Order Output Name Condition

1 Inserts __CDC_OPERATION == "I"

2 Updates __CDC_OPERATION == "UN"46


3 Deletes __CDC_OPERATION == "D"

OLE DB Command Data Mart DeletesThe Delete flow is directed to the OLE DB Command Data Mart Deletes. The following command is used to apply the changes within this flow to the destination:

delete from CDCSample.WorkOrder where WorkOrderID = ?

The command requires a column from the result set in order to successfully apply the delete. This is the primary key columns for the table. The delete command above is for the child package processing changes to the WorkOrder table, which has a single primary key column WorkOrderID.

The mapping of result set columns to command parameters can be seen in the following diagram:

Figure 6: Result Set Mapping for Deletes

OLE DB Command Data Mart UpdatesThe Update flow is directed to the OLE DB Command Data Mart Updates. The following command is used to apply the changes within this flow to the destination:

47


update CDCSample.WorkOrder setProductID = ?,OrderQty = ?,ScrappedQty = ?,StartDate = ?,EndDate = ?,DueDate = ?,ScrapReasonID = ?,ModifiedDate = ?

where WorkOrderID = ?

This command requires all of the source columns from the result set in order to successfully apply the update.

The mapping of result set columns to command parameters can be seen in the following diagram:

Figure 7: Result Set Mapping for Updates

OLE DB Destination Data Mart InsertsThe Insert flow is directed to the OLE DB Destination. The table name alone identifies the target for the insert. The following diagram shows how the result set columns are mapped to the table columns:

48


Figure 8: Result Set Mapping for Inserts

Error Logging in the Child PackagesEach of the child packages makes use of the SSIS Log Provider for Windows Event Log to allow logging for the OnError event. Logging is configured using the Configure SSIS Logs dialog box. This box appears in the designer when right clicking on a selected package and choosing Logging. This dialog box also allows the current logging settings to be examined. For the child packages, this dialog box shows the SSIS Log Provider for Windows Event Log as the Provider type on the Providers and Logs tab. Under the Details tab, the OnError condition is set.

Even when range validation is done up front, it is always possible that conditions will change between the time the check is performed and the time the TVF executes. Applications must always be prepared to deal with the possibility of range errors. The CDC TVFs check to insure that the extraction interval defined is fully covered by the validity interval for the capture instance and errors if this requirement is not met.

In the sample, if range errors are encountered by any of the child packages, OLE DB will log the error to the event log producing an entry similar to the following.

49


Figure 9: Range Error Posted from OLEDB

The error returned for range errors is error 313, “An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ …”. The figure above shows this error.

Wrappers for CDC TVFs

Instantiated Multi-Statement TVF WrappersWrapper functions for consuming CDC change data serve several important purposes. First, and most importantly, they allow an SSIS Data Flow task to be used in a straightforward manner to query for change data. The inline TVFs that are generated to access CDC change tables do not allow the column structure of the returned result set to be determined by an OLE DB provider. This limitation, however, can be dealt with in a straightforward fashion by wrapping the inline TVFs with a multi-statement TVF that

50


explicitly identifies the columns returned by the CDC function. The scripted wrapper functions do precisely this.

While the need to be able to determine column information about the returned result set represents the principle reason for making use of a wrapper function for the generated CDC TVF, the wrapper serves several other very useful purposes.

The generated CDC query functions used to gather change data use Log Sequence Numbers (LSNs) to mark the boundary of the query window. While LSNs are invaluable in insuring that data can be retrieved from change tables in a systematic manner that guarantees no lost or repeated data, they have virtually no meaning to the application layer that wants to consume change data. Particularly for Data Mart applications, the request for change data is most typically bounded by datetime values. The CDC feature has built-in functionality to deal with systematically mapping between datetime values and LSN values. The wrapper function is an ideal place to address these mapping issues, allowing the application to define its request interval as a datetime pair, with the wrapper function performing the necessary translation between these values and the LSN values needed to query the generated CDC TVFs.

In addition to shielding the SSIS application from dealing with LSN values directly, the wrapper function can perform column filtering on the data returned from the CDC TVF more efficiently than column filtering done when querying the wrapper function. The result set returned by the CDC TVF is expected to serve a variety of clients, and typically the captured columns included within a capture instance are not client specific. When the requesting package needs only a subset of the captured columns, this column filtering is best done within the wrapper code.

Finally, the wrapper code allows information returned from the generated TVF to be recoded in a form that is more easily consumed by the calling application. In its simplest form, recoding can be used to convert the integer based operation codes into more meaningful single character codes that provide built in cues to their meaning, ie mapping 2 to ‘I’ for insert, 1 to ‘D’ for delete, and 4 to ‘UN’ for update new values. The more interesting case, however, is the recoding of column update information extracted from the CDC update mask.

When a CDC function returning net changes is called using the row filter option ‘all with mask’, an update mask is generated that identifies all of the column values that changed due to the update. If more than one update occurred during the interval, the mask represents the aggregate changes for all updates. This ability to easily determine the columns that will change, prior to applying an update, can be extremely useful during ETL dimension processing. Because this information is typically only needed for a small subset of columns, the wrapper is an appropriate place to extract this information from the mask and return it as simple flag columns to the SSIS application layer.

51


We will now look at a simple example of a wrapper function for the table WorkOrders in AdventureWorks2008.

A Closer Look at an SSIS Wrapper Function

Use of datetime Values in the Wrapper SignatureWhat we first note when examining a wrapper function is that its signature contains two datetime values: a start time and an end time. Typically, it is most natural for an SSIS application to use a time interval to delimit a query range, since this allows the CDC change data to be easily related to other Data Mart data. While the CDC TVFs do not allow datetime values to be used directly as query end-points, mapping functions are provided that make it possible to systematically translate datetime values into corresponding LSNs. The wrapper functions provide an ideal location in which to imbed this mapping logic.

In the generated wrapper functions scripted by the procedure sys.sp_cdc_generate_wrapper_function, the following convention is used to guarantee that no data is repeated or skipped. The caller of the wrapper agrees to pass the end time of the previous data query as the start time of the next query, without modification. Within the wrapper, the calls to the mapping functions are always constructed in exactly the same way. This makes it possible to insure that given the same time value passed in the previous call, the mapping function will be able to regenerate the same LSN value in the subsequent call. This is key to guaranteeing that the resulting LSN based intervals do not have breaks or overlap.

Finally, the function sys.fn_cdc_increment_lsn is used to increment the previous end lsn by one to obtain the next start LSN value. Note that it is guaranteed that there are no LSNs that lie between an LSN and that LSN value incremented by 1. Note also that the LSNs passed to the TVFs define a closed interval, so that all change entries having LSN values within the defined interval, including the boundary values, will be included in the returned result.

Below we have extracted the portion of the wrapper function that addresses the mapping of datetime values to LSN values to define the extraction interval. Note that passing a null value for the start time value signals the wrapper function to use the current low end point of the capture instance validity interval as the interval start time. Similarly, a null end time value will cause the high end point of the capture instance to be used as the high end point for the extraction interval. This serves a couple of useful purposes. When you are designing the package to process the result set returned by the wrapper function, you can set the default string that defines the call to use null parameters. This allows you to get needed column information for the result set without worrying too much about providing a valid extraction interval. The call to the wrapper function using null values will only fail if the cdc.lsn_time_mapping table has no entries.

52


Some of the additional checking performed by the wrapper function deserves explanation. We know that the CDC LSN based query functions will return a non-zero value in @@error if the LSN range does not fall within the validity interval of the capture instance. We would like the same to be true when the LSN based query functions are called from within a generated wrapper function. In order to insure this, we need to check the datetime range explicitly, prior to mapping it to an LSN value. If the datetime range value is not within bounds, it is mapped to NULL forcing the range error when the underlying TVF is called.

Also note the following check:

If @from_lsn is not null and @to_lsn is not null and

(@from_lsn = sys.fn_cdc_increment_lsn( @to_lsn ))

return

This is a legitimate condition that can occur when there are no entries in cdc.lsn_time_mapping in the interval between the start and end time. This causes the function sys.fn_cdc_map_time_to_lsn(‘largest less than or equal’, @start_time) and sys.fn_cdc_map_time_to_lsn(‘largest less than or equal’, @end_time) to return the same LSN value. Incrementing the start LSN by one causes the condition to evaluate to true. In this case, it is appropriate to return with an empty result set.

CREATE function [dbo].[fn_net_changes_WorkOrder] (

@start_time datetime = null,

@end_time datetime = null,

@row_filter_option nvarchar(30) = N'all'

)

…

begin

declare @from_lsn binary(10), @to_lsn binary(10)

if (@start_time is null)

53


select @from_lsn =

[sys].[fn_cdc_get_min_lsn]('WorkOrder')

else

begin

if ([sys].[fn_cdc_map_lsn_to_time

([sys].[fn_cdc_get_ min_lsn]('WorkOrder')

> @start_time

or ([sys].[fn_cdc_map_lsn_to_time]

([sys].[fn_cdc_get_max_lsn]()) < @start_time)

select @from_lsn = null

else

select @from_lsn = [sys].[fn_cdc_increment_lsn]

([sys].[fn_cdc_map_time_to_lsn]

('largest less than or equal',@start_time))

if (@end_time is null)

select @to_lsn = sys.fn_cdc_get_max_lsn()

else

begin

if [sys].[fn_cdc_map_lsn_to_time]

([sys].[fn_cdc_get_max_lsn]() < @endt_time)

select @to_lsn = null

else

select @to_lsn =

[sys].[fn_cdc_map_time_to_lsn]

('largest less than or equal',@end_time)

end

54


if @from_lsn is not null and @to_lsn is not null and

(@from_lsn = [sys].[fn_cdc_increment_lsn](@to_lsn))

return

…

end

Returning Column Information in the Wrapper Function

The wrapper function is structured as a multi-statement TVF. The result set returned by the function is defined explicitly, and allows the OLE DB provider to make column information available to the calling program. The need to include this information explicitly would make this one of the more tedious aspects of preparing wrapper functions manually. Note that the result from querying the CDC TVF is inserted into the defined table before exiting the function. The defined table includes all of the table columns plus a final column __CDC_OPERATION. This column is a more user friendly recoding of the __$Operation column returned in the CDC TVF.

create function [dbo].[fn_net_changes_WorkOrder] (




)

returns @resultset table (

[WorkOrderID] int

,[ProductID] int

,[OrderQty] int

,[ScrappedQty] smallint

,[StartDate] datetime

55


,[EndDate] datetime

,[DueDate] datetime

,[ScrapReasonID] smallint

,[ModifiedDate] datetime

,[__CDC_OPERATION] varchar(2)

) as

begin

…

insert into @resultset

select

[WorkOrderID]

,[ProductID]

,[OrderQty]

,[ScrappedQty]

,[StartDate]

,[EndDate]

,[DueDate]

,[ScrapReasonID]

,[ModifiedDate]

,case [__$operation]

when 1 then 'D'

when 2 then 'I'

when 3 then ‘UO’

when 4 then 'UN'

when 5 then ‘M’

else null

56


end as [__CDC_OPERATION]

from

[cdc].[fn_cdc_get_net_changes_WorkOrder]

(@from_lsn, @to_lsn, @row_filter_option')

return

end

go

Extracting Information from the CDC Update MaskCDC functionality includes the ability to identify all column values that changed for each identified update operation. When querying for all changes, this information is always returned. When querying for net changes, the update mask is only returned as a non-null value when the filter option ‘all with mask’ is selected. While this information can be extremely useful, its representation in mask form is awkward for SSIS to consume directly. CDC does, however, provide SQL functions to assist in extracting data from the update mask, making wrapper functions an ideal place for performing this extraction. While information from the update mask is not used within the current sample, it is nevertheless useful to show how logic to extract data from the update mask can be embedded within a simple wrapper function.

While the CDC supplied mask provides information on all captured columns, it is likely that the application will only be interested in this information for a handful of columns. In the example below, the wrapper for the WorkOrders table is modified to include a final flag column that indicates on update whether the column OrderQty has changed.

For this sample we made use of the default wrapper functions so were able to generate the wrappers for all defined, accessible capture instances with a single call. It is possible, however, to tailor the call to the scripting stored procedure to further customize the generated wrapper. In this case, we want to explicitly identify columns that need an update flag. We do this by coding the optional parameter @update_flag_list as a comma separated list of columns for which update information is needed.

The call below generates a wrapper with an additional output column that for update operations, signals whether or not the column OrderQty was modified.

57


exec sys.sp_cdc_generate_wrapper_function

@capture_instance = 'WorkOrder',

@update_flag_list = 'OrderQty'

When examining the code below, note that the returned table @WorkOrder now includes an additional column OrderQty_uflag, after __CDC_OPERATION. This column will hold the additional update flag. Next, note the call to the function sys.fn_cdc_get_column_ordinal to obtain the column ordinal for the column of interest. Once the column ordinal is known, the function sys.fn_cdc_is_bit_set can be applied in the select statement for each returned update to determine whether the bit corresponding to the column is set in the returned mask. Finally, note that when the TVF dbo.fn_net_changes_Workload is invoked, the row filter option must be set to 'all with mask' to signal, in the net changes query, that the mask should be computed.

create function [dbo].[fn_net_changes_WorkOrder] (



@row_filter_option nvarchar(30) = N'all')

returns @resultset table (

[WorkOrderID] int

,[ProductID] int

,[OrderQty] int

,[ScrappedQty] smallint

,[StartDate] datetime

,[EndDate] datetime

,[DueDate] datetime

,[ScrapReasonID] smallint

,[ModifiedDate] datetime

,[__CDC_OPERATION] varchar(2)

,[OrderQty_uflag] bit

) as

58


begin

declare @from_lsn binary(10), @to_lsn binary(10)

declare @ordinal_1 int

select @ordinal_1 = [sys].[fn_cdc_get_column_ordinal]

('WorkOrder', 'OrderQty')

if (@start_time is null)

select @from_lsn = [sys].[fn_cdc_get_min_lsn]('WorkOrder')

else

begin

if ([sys].[fn_cdc_map_lsn_to_time]

([sys].[fn_cdc_get_min_lsn]

('WorkOrder')) > @start_time) or

([sys].[fn_cdc_map_lsn_to_time]

([sys].[fn_cdc_get_max_lsn]()) < @start_time)


else

select @from_lsn = [sys].[fn_cdc_increment_lsn]

([sys].[fn_cdc_map_time_to_lsn]

('largest less than or equal',@start_time))

end


select @to_lsn = [sys].[fn_cdc_get_max_lsn]()

else

59


begin

if [sys].[fn_cdc_map_lsn_to_time]

([sys].[fn_cdc_get_max_lsn]()) < @end_time


else

select @to_lsn = [sys].[fn_cdc_map_time_to_lsn]


end



return


select

[WorkOrderID]

,[ProductID]

,[OrderQty]

,[ScrappedQty]

,[StartDate]

,[EndDate]

,[DueDate]

,[ScrapReasonID]

,[ModifiedDate]

,case [__$operation]

when 1 then 'D'

when 2 then 'I'

60


when 3 then 'UO'

when 4 then 'UN'

when 5 then 'M'

else null

end as [__CDC_OPERATION],

case[ __$operation]

when 4 then

case [__$update_mask]

when null then null

else [sys].[fn_cdc_is_bit_set]

(@ordinal_1, [__$update_mask])

End

else null

end as [OrderQty_uflag]

from

[cdc].[fn_cdc_get_net_changes_WorkOrder]

(@from_lsn, @to_lsn, @row_filter_option)

return

end

Customizing Wrapper Functions for the First Extraction Interval

Our strategy for synchronizing the initial load from a database snapshot forces us to deal with the first extraction interval specially, allowing the lower bound of the extraction interval to be expressed as an LSN value, while continuing to honor a datetime value as the upper bound. While these customized functions are not currently scripted when wrapper functions are generated, it is straightforward to hand modify the generated wrappers to produce a version designed specifically to handle the initial extraction interval.

61


In this sample, the code for these functions is included explicitly in the script file CDCSetupTables.sql. The modified WorkOrder script is shown below, with the changes appearing in black. These changes include a change to the function signature, the conversion of the string representation of the LSN to a binary(10), some range validation against the capture instance validity interval, and an incremental adjustment to get all changes after the defined anchor LSN.

-- Generate custom wrapper functions for initial incremental load of

-- WorkOrder

create function [dbo].[fn_net_changes_WorkOrder_First]

( @start_lsn_str varchar(40) = null,



)

returns @resultset table ( [WorkOrderID] int, [ProductID] int,

[OrderQty] int, [StockedQty] int, [ScrappedQty] smallint,

[StartDate] datetime, [EndDate] datetime, [DueDate] datetime,

[ScrapReasonID] smallint, [ModifiedDate] datetime,

[__CDC_OPERATION] varchar(2)

) as

begin

declare @from_lsn binary(10), @to_lsn binary(10),

@start_lsn binary(10)

if (@start_lsn_str is null)

select @from_lsn =

[sys].[fn_cdc_get_min_lsn]('WorkOrder')

else

begin

select @start_lsn =

62


[dbo].[HexStrToVarBin](@start_lsn_str)

if ([sys].[fn_cdc_get_min_lsn]('WorkOrder') >

@start_lsn) or

([sys].[fn_cdc_get_max_lsn]() < @start_lsn)


else

select @from_lsn =

[sys].[fn_cdc_increment_lsn](@start_lsn)

end


select @to_lsn = [sys].[fn_cdc_get_max_lsn]()

else

begin

if [sys].[fn_cdc_map_lsn_to_time]([sys].

[fn_cdc_get_max_lsn]()) < @end_time


else

select @to_lsn = [sys].[fn_cdc_map_time_to_lsn]


end



return


63


select [WorkOrderID], [ProductID], [OrderQty], [StockedQty], [ScrappedQty], [StartDate], [EndDate], [DueDate], [ScrapReasonID], [ModifiedDate],

case [__$operation]

when 1 then 'D'

when 2 then 'I'

when 3 then 'UO'

when 4 then 'UN'

when 5 then 'M'

else null

end as [__CDC_OPERATION]

from [cdc].[fn_cdc_get_net_changes_WorkOrder](@from_lsn, @to_lsn, @row_filter_option)

return

end

GO

Running the Change Data Capture for Specified Interval Package Sample in BI Studio

To run the Change Data Capture for Specified Interval Package Sample in BI Studio, open the Change Data Capture for Specified Interval Package Sample project and execute the package SetupCDCSample.dtsx. It should take several minutes to run. Once the execution environment has been set up, three of the workload tasks are launched to generate DML against the AdventureWorks2008 tables that are being tracked. These workload tasks have built in 10 second delays to guarantee that the load is spread over several minutes, regardless of the hardware that it is run on.

64


Figure 10: Setup Package Performing Initial Load and Workload Generation Concurrently

While the workload is being applied, a task to generate the database snapshot for AdventureWorks2008 is also launched. After it completes the dataflow tasks are allowed to run against the snapshot database to initially load the target tables in AdventureWorksDW2008. Once the initial load completes, the base time for the initial extraction interval is determined and the master incremental load package is invoked.

Each time the master package is invoked, it first verifies that the desired extraction interval is contained within the validity interval for the database. If necessary, it will delay and give the capture process time to process all the changes through the high end-point of the next extraction interval. Once the workload has been applied the Mark Workload Completion task sets package variables to indicate that the workload has completed and to identify a completion time. When the start time of an extraction interval exceeds that time, the loop cycling the master package will terminate.

65


Figure 11: Setup package at Successful Run Completion

The control flow diagram above shows the setup package at the end of a successful run.

ConclusionThe Change Data Capture feature of SQL Server 2008 makes change data available in a relational format. This document describes how SSIS packages can leverage this feature to handle incremental loads to a Data Mart using a sample available on Codeplex as the vehicle to drive the discussion.

For more information:

SQL Server Web site: http://www.microsoft.com/sql/default.mspx

SQL Server TechCenter: http://technet.microsoft.com/en-us/sqlserver/default.aspx

SQL Server DevCenter: http://msdn2.microsoft.com/en-us/sqlserver/default.aspx

Did this paper help you? Please give us your feedback. Tell us on a scale of 1 (poor) to 5 (excellent), how would you rate this paper and why have you given it this rating? For example:

66

http://msdn2.microsoft.com/en-us/sqlserver/default.aspx

http://technet.microsoft.com/en-us/sqlserver/default.aspx

http://www.microsoft.com/sql/default.mspx


Are you rating it high due to having good examples, excellent screenshots, clear writing, or another reason?

Are you rating it low due to poor examples, fuzzy screenshots, unclear writing?

This feedback will help us improve the quality of white papers we release. Send feedback.

67

mailto:[email protected]?subject=White%20Paper%20Feedback:%20%5BTitle%20of%20white%20paper%5D

mailto:[email protected]?subject=White%20Paper%20Feedback:%20%5BTitle%20of%20white%20paper%5D

CD Css is Sam

Documents

Transcript of CD Css is Sam