8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
1/18
2008 IBM Corporation1
CDC Transformation and Delivery
Data at the speed of business
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
2/18
2
What is CDC
Change Data Capture
Capture data events in source database and move onlythe changes to the target
Many different ways of doing CDC
Timestamps
Triggers
API
Log-based
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
3/18
3
What fuels the IBM CDC Roadmap? The widest breadth of functionality:
Batch/pull and real-time push processing
Guaranteed delivery/transactional integrity
Multiple topologies (peer to peer, 1 to many, many to 1, uni-directional, bi-directional)
Homogeneous & heterogeneous data synchronization
Broadest range of sources and targets
Log-based capture agents for DB2 (on all platforms), Oracle, SQL Server, Sybase,IMS, VSAM, IDMS, ADABAS
Native/parallel applies for all RDBMS and JMS
Multiple data delivery protocols (TCP/IP, JMS)
Industry leading performance and scalability
End to end throughput and low latency
Parallel Apply to target system
Low impact on source database systems
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
4/18
4
What fuels the IBM CDC Roadmap .
3000+ customers using the existing CDC products for;
HA/DR (DB back-up, fault tolerance)
Real-time reporting/off-load querying
Application Co-existence (migrations, upgrades, modernization)
eCommerce (web apps, portals, data distribution)
Dynamic Data Warehousing, Master Data Management
700+ people in engineering focused on Information Integration including 170+focused on CDC technologies
The most comprehensive suite of data integration products
BoB transform / cleanse / discovery, metadata management, scalableperformance, services enabled for SOA architectures
5000+ customers using Information Server components
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
5/18
5
The IBM Solution: IBM Information ServerDelivering information you can trust
IBM Information Server
Discover, model, andgovern information
structure and content
Standardize, merge,and correct information
Combine andrestructure information
for new uses
Capture, virtualize andmove information for in-
line delivery
Unified Deployment
Unified Metadata Management
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
6/18
6
InfoSphere CDC Solution
Provides real-time change datacapture and delivery for
Dynamic warehousing and real-timereporting
Synchronization and replication Event detection
Minimal impact on productionsystems
High scalability and end-to-endperformance
Guaranteed data integrity
Proven Heterogeneous support
Delivers real time changed data toInformation Server, applications and
targets or message queues
ArchitectsDevelopers
DataMirror
Without impacting performance ofproduction systems
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
7/18
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
8/188
Architecture
DatabasesOracle, DB2, DB2 UDB, SQL Server, Sybase, Teradata, Netezza, PointBase
IMS, VSAM, IDMS, Adabas, DataCom - Classic
Platformsz/OS, System i5, Red Hat and SUSE Linux, AIX, HP/UX (PA-RISC and Itanium), Solaris SPARC,Tru64 UNIX, Windows
Messaging Middleware
MQSeries, Sun Open Message Queue (JMS), TIBCO, BEA AquaLogic, Oracle Fusion Middleware
Journal LogRedo/Archive Logs
Source EngineAnd Metadata
Target EngineAnd Metadata
TCP/IP
Java-based GUIfor admin & monitoring
Database
ODS
JMS
BusinessProcess
Publisher Subscriber
Flat files
Audit
Direct to existing ETL
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
9/18 2008 IBM Corporation9
Customer examples
Use Cases
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
10/1810
1. Building A Low Latency ODS for Operational Reporting and Auditing
ODSManufacturing
ERP
OLTP
Native
Log
DB
Manufacturing
Finance
OLTP
Native
Log
DB
Production Server
Production Server
All OLTP insert, update and delete operations canbe stored as inserts to maintain completetransaction history.
Add relevant information such as timestamp,transaction type, source system id, and id of user
who changed the transaction.
Operational Data Store
Solution deployed to improve visibility into lines of business for organizations withOperational BI and Data Auditing requirements
Each OLTP insert, update and delete operationcan be stored as an insert, update and delete tomaintain synchronized copy of data.
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
11/1811
2. Complementing An Existing ETL Technology
Continuous
Retail
Point Of Sale
OLTP
Native
Log
DB
EDWStageETL
ETL ServerProduction Server
Scheduled Batch
2. Business Objects Data Integrator
1. Informatica Power Center
3. Ab Initio
4. IBM DataStage (has native integration)
Complementary ETL Technologies:
Data Warehouse
2. Flat File
1. Relational Table
3. Message Queue
4. Direct to ETL
Stage can be:
Solution deployed to improve visibility into lines of business (i.e. Dynamic Warehousing)
and help manage impact concerns caused by ETL on mission critical systems
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
12/1812
3. Continuous Feed Of A Business Intelligence Appliance
Manufacturing
ERP
OLTPNative
Log
DB
CDC
Stage
Flat File Appliance
Continuous (to Appliance)
CDC
Staging ServerProduction Server Appliance Nodes/Cluster
Flat file containing transaction changes viewed asan external file to the appliance.
Load threshold based on # of Transactions or timeinterval.
Once threshold reached, call appliance load API
to bulk load transactions into appliance.
Solution deployed to improve visibility into lines of business by combining the
cost/performance benefits of a BI Appliance with real-time data feeds.
Appliance
Load API
2. Netezza
1. Teradata
Supported Appliances
3. GreenPlum
4. Paraccel
5. IBM Balanced Warehouse
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
13/1813
Data Event Synchronization via an Enterprise Service Bus
CDC/Replication License
CDC/Replication Process
Other Technology
E
S
B
Queue 1Queue 1
Billing
Continuous
CDC
OLTP
Telco
Native
Log
DB
Production Server
Telco
OLTP
CRM
Production Server
2. TIBCO Business Works
1. IBM MQ Series
3. BEA Aqualogic
4. WebMethods Fabric
Complimentary ESB Technologies:
Solution deployed to provide real time data feeds for SOA and application
integration business requirements.
A license would reside on the server thathosts the message oriented middleware.
ContinuousCDC
ETL
Solution deployed to provide real time data feeds for SOA and applicationintegration business requirements.
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
14/1814
5. e-Commerce Application Synchronization
Retail
Inventory
OLTP
Native
Log
DB
Native
Log
DB
Corporate
Native
LogDB
Downtown Store
OLTP
OLTP
Production Server
Website Orders
Point Of Sale
Solution deployed to provide continuous customer, sales and inventory visibilityin web base e-commerce applications.
Provides continuous bi-directional synchronizationbetween web based applications and missioncritical business applications.
Helps organizations improve customer online shoppingexperience with improved visibility into inventory and customershopping activities.
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
15/1815
6. Data Synchronization for Upgrades, Migrations and WorkloadBalancing
Manufacturing
OLTP
Native
Log
DB OLTPNative
Log
DB
ERP
Upgrades, Migrations
ERP
Manufacturing
Production Server Testing Server
Workload Balancing
Solution deployed to help IT support application, database and platform migrations.
Keep data synchronized between currentproduction server and a server deployed to test anew application upgrade/version, or ahardware/OS upgrade.
Workload balancing capability (i.e. master to master support)allows database instances to remain synchronized where dual ordouble data entry is a requirement (i.e. data entry occurring onboth systems at the same time).
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
16/18
16
7. Offloading Production Query & Reporting Cycles
Services
OLTP
Report
Services
Finance 3
OLTP
Native
Log
DB
Native
Log
DB
Finance 1
Services
Finance 2
OLTP
Table Copy
Query
Production Server(s)
Reporting Server
Reporting server can also be used forconsolidation requirements i.e. consolidatingfinancials from multiple branches into a singlecorporate instance.
Solution deployed to allow organizations to offload the impact of query andreporting to a non mission critical system.
Replication frequency generally varies fromcontinuous (near real-time) to periodic. Table levelrefresh or copy can be used in addition to logbased change data capture.
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
17/18
17
8. Data Backup And Availability
Finance
Backup
Backup
Native
Log
DB
Partition 1
Partition 2
Continuous (to backup instance)
CDC
OLTP
Production ServerAvailabilityServer
Availability of data only, does not support DDLreplication.
Exact image replication to produce a backup copyon a separate server or in a different partition onthe same server.
A separate license is not required for each partitionused on the production server.
Solution deployed to allow organizations to backup copies of critical data for
recovery where a full disaster solution is not a requirement.
8/3/2019 Info Sphere CDC - Real Time Data Integration UC 17-09-08
18/18
2008 IBM Corporation18
Thank You
Top Related