DataMigrator 7.7 in Real Time

72
DataMigrator 7.7 in Real Time Chris Bevilacqua iWay Solutions Architect

description

DataMigrator 7.7 in Real Time. Chris Bevilacqua iWay Solutions Architect. Real Time ETL with DataMigator Agenda. Web Services Change Data Capture File Listener IWAF Adapters. Web Services. What are Web Services for?. What is a Web Service?. - PowerPoint PPT Presentation

Transcript of DataMigrator 7.7 in Real Time

Page 1: DataMigrator 7.7 in Real Time

DataMigrator 7.7 in Real Time

Chris BevilacquaiWay Solutions Architect

Page 2: DataMigrator 7.7 in Real Time

Real Time ETL with DataMigatorAgenda

Web ServicesChange Data CaptureFile ListenerIWAF Adapters

Page 3: DataMigrator 7.7 in Real Time

Web Services

What are Web Services for?

Page 4: DataMigrator 7.7 in Real Time

What is a Web Service?

A software system designed to support interoperable machine-to-machine interaction over a network.

It has an interface described in a machine-processable format (specifically Web Services Description Language WSDL).

– World Wide Web Consortium

Page 5: DataMigrator 7.7 in Real Time

.net

Publicly Available Commercial Web Services

Some sources of Public Web ServicesFree and commercial

Page 6: DataMigrator 7.7 in Real Time

WSDLL.COMWeb Services – Free and Commercial

Page 7: DataMigrator 7.7 in Real Time

Xmethods.net List of public web services

Page 8: DataMigrator 7.7 in Real Time

Seekda.com Find WSDL for PhoneVerify Service from cdyne.com

Page 9: DataMigrator 7.7 in Real Time

Add adapter for Web Services

Page 10: DataMigrator 7.7 in Real Time

Add Connection for a Web Service

Page 11: DataMigrator 7.7 in Real Time

Add Connection for Web Service

Page 12: DataMigrator 7.7 in Real Time

Create Synonym

Page 13: DataMigrator 7.7 in Real Time

Open Synonym

Page 14: DataMigrator 7.7 in Real Time

Sample DataSupply values for PHONENUMBER and LICENSEKEY

Page 15: DataMigrator 7.7 in Real Time

Sample DataShow all fields returned by Web Service

Page 16: DataMigrator 7.7 in Real Time

Create Data FlowAdd input file

Page 17: DataMigrator 7.7 in Real Time

User Defined FunctionCleans up phone number

Page 18: DataMigrator 7.7 in Real Time

Add transformation Clean up phone number and assign name

Page 19: DataMigrator 7.7 in Real Time

Add synonym for Web Service

Page 20: DataMigrator 7.7 in Real Time

Join input file to Web ServiceOn PHONENUMBER and LICENSEKEY

Page 21: DataMigrator 7.7 in Real Time

Select Columns from Web Service

Page 22: DataMigrator 7.7 in Real Time

Test SQLShows “join” from input file to Web Service

Page 23: DataMigrator 7.7 in Real Time

Change Data Capture

When would you want to capture changes?

Page 24: DataMigrator 7.7 in Real Time

Why Change Data Capture?

Large source databases, small batch windowWant to only copy the changes Want near real-time updates

Need some way to detect what has changed…

Page 25: DataMigrator 7.7 in Real Time

Change Data CaptureHow to detect changes to database tables

Change the application? Track timestamp of changes, flag deletes Write changes to a separate table

Compare source to target? Effective for small tables

Poll the database for changes periodically? Detects additions only

Use database triggers? Developer adds to database…. or automatically generated Requires staging table

Page 26: DataMigrator 7.7 in Real Time

Change Data CaptureData Adapter to read from database logs

Read the database log… Proprietary formats Every database is different

New Data Adapter for database log tables No actual staging table required Synonym describes a view of database log for one table Provides automatic checkpoint processing

Page 27: DataMigrator 7.7 in Real Time

Change Data CaptureDatabase configuration

Platforms and databases supported ORACLE 10g and11g on UNIX and Windows UDB on UNIX and Windows DB2 on IBM zSeries and iSeries MS SQL Server 2008

DBA issues database specific commands to Configure database to use archiving or enable logging  Add logging or enable capture mode

Create additional synonym for table log records

Page 28: DataMigrator 7.7 in Real Time

Create Synonym For Table Log Records

Page 29: DataMigrator 7.7 in Real Time

Create Synonym for Table Log Records

Page 30: DataMigrator 7.7 in Real Time

Synonym created for Log TableThree CDC columns added

Page 31: DataMigrator 7.7 in Real Time

Synonym for Log TableLayout same as source table… plus three columns

CDC_OPER – Operation Type I = Insert U = Update D = Delete

CDC_TID – Transaction ID CDC_TIMES – Timestamp

Page 32: DataMigrator 7.7 in Real Time

Create Synonym for Log TableCDC Configuration

Polling intervalTimeoutCheckpoint processing

Page 33: DataMigrator 7.7 in Real Time

CDCConfiguration

POLLING – Log polling interval for LUW

TIMEOUT – Timeout interval polling for LUW

START – CHKPT after last checkpoint saved

CHECKPT_SAVE [YES|NO] – save last checkpoint in file

CHECKPT_FILE – Location and name of file

MAX_LUWS – Maximum number of LUWs processed

Page 34: DataMigrator 7.7 in Real Time

UDBSynonym for Log Table

Page 35: DataMigrator 7.7 in Real Time

Data FlowLog table as source

Page 36: DataMigrator 7.7 in Real Time

Column Selection

Page 37: DataMigrator 7.7 in Real Time

Target Transformations

Page 38: DataMigrator 7.7 in Real Time

Enabled for IUD ProcessingIUD control column identified

Page 39: DataMigrator 7.7 in Real Time

UDB CDCStart with Source and Target the same

Run program to update source

Page 40: DataMigrator 7.7 in Real Time

CDC Flow Log shows rows inserted / updated / deleted

Page 41: DataMigrator 7.7 in Real Time

Change Data CaptureSource and Target now have same rows

Page 42: DataMigrator 7.7 in Real Time

Change Data CaptureReprocessing

For testing or reprocessing…Start at specified timestamp

Page 43: DataMigrator 7.7 in Real Time

File Listener

Process files as they arrive in a directory… posted or copied or program output

Page 44: DataMigrator 7.7 in Real Time

Refactoring

7.6 Listener based Required creating and managing Listener Only Direct Load flows Only fixed length files, only character data Flow never completed… could not view log

could not send email or run process flow Could not schedule flows

7.7 Adapter based All specifications in synonym, simpler to set up Either SQL or Direct Load flows Any file type, any data type Flow completes when file or time limit reached…

Can send email or run flow or procedure on completion Can schedule flows as needed

Page 45: DataMigrator 7.7 in Real Time

File ListenerCreate Sample Data

Page 46: DataMigrator 7.7 in Real Time

File Listener ParametersSynonym - dmordfl

Page 47: DataMigrator 7.7 in Real Time

Data Allocation Parameters

Connection – Name of a connection for an FTP Server

Data Origin FILE – one time read LISTENER – process files as they arrive TAIL – future development

Identify File(s) Directory – Application Directory to poll for files Name – file name, wildcards allowed Extension – file type or extension

Page 48: DataMigrator 7.7 in Real Time

Listening Parameters and Read Limits

Listening Polling – interval in seconds Timeout – time in seconds. If no new files arrive end job

Read Limits Maxfiles – Maximum number of data files Maxrecs – Maximum number of records (only for TAIL)

Page 49: DataMigrator 7.7 in Real Time

Pre and Post Processing

Pre Processing - Pickup IMMEDIATE – As soon as files arrives TRIGGER – Wait for trigger file with

EXTENSION

Post Processing - DiscardDELETE ARCHIVE – Copy files to

DIRECTORY KEEP – Delete trigger only and keep file

Page 50: DataMigrator 7.7 in Real Time

File Name Field with Alias of INSTANCE

Page 51: DataMigrator 7.7 in Real Time

SQL or Direct Load Flow… looks like any other flow

Page 52: DataMigrator 7.7 in Real Time

File ListenerCreate directories listen and archive

Page 53: DataMigrator 7.7 in Real Time

File ListenerSubmit the Flow – Waits for TIMEOUT interval

Page 54: DataMigrator 7.7 in Real Time

File ListenerDrop the Files

Page 55: DataMigrator 7.7 in Real Time

File ListenerDrop the Files - within timeout interval (60 seconds)

Page 56: DataMigrator 7.7 in Real Time

File ListenerSample Data

Page 57: DataMigrator 7.7 in Real Time

Log shows number of rows processedFor all files

Page 58: DataMigrator 7.7 in Real Time

IWAF - iWay Adapter Framework

Page 59: DataMigrator 7.7 in Real Time

iWay Adapter FrameworkApplication Adapters

Data Adapters for some ERP systems New support for Application Adapters including

Salesforce.com MySAP Seibel Peoplesoft

So that DataMigrator can use Services Events

Page 60: DataMigrator 7.7 in Real Time

Configure IWAF Adapter

Page 61: DataMigrator 7.7 in Real Time

Add IWAF Adapter

Page 62: DataMigrator 7.7 in Real Time

Select IWAF Adapter

Page 63: DataMigrator 7.7 in Real Time

Add Connection for IWAFSalesforce Service

Page 64: DataMigrator 7.7 in Real Time

Salesforce AdapterCreate Synonym

Page 65: DataMigrator 7.7 in Real Time

Salesforce AdapterCreate Synonym

Page 66: DataMigrator 7.7 in Real Time

Salesforce AdapterCreate Synonym

Page 67: DataMigrator 7.7 in Real Time

Create Synonym for Salesforce Service

Page 68: DataMigrator 7.7 in Real Time

Salesforce ServiceSample Data

Page 69: DataMigrator 7.7 in Real Time

Salesforce EventSynonym

Page 70: DataMigrator 7.7 in Real Time

SalesforceData Flow

Page 71: DataMigrator 7.7 in Real Time

Salesforce.com

Page 72: DataMigrator 7.7 in Real Time

Salesforce Output table and log