Click to edit Master title style - IBM to edit Master title style Integrating Your Information -...

Click to edit Master title style

Integrating Your Information - Tools to make life easier

Agenda

� Information on demand and data delivery– Concepts

– Information Integration challenges

– The Integration Landscape

– Process-Centric Integration versus Data-Centric Integration

� Applying the Integration Models– Sample Implementations

– The Role of Replication

� Wrap-up


Information on Demand …and Data Delivery

Integration – Some Definitions

� Process-Centric– EAI (Enterprise Application Integration)

• Process-centric, automate workflows and process dependencies

• Leverage process assets across applications, to create new applications

� Data-Centric

– Federation• Data and content centric "pull"

• Virtualizes access to information sources: structured, unstructured, content

– ETL (Extract, Transform, Load)• Data placement and transformation solutions

• Typically scheduled, but moving to more real-time models

– Data Event Publishing• Data centric "push"

• Driver behind other data movement and integration models

– Replication• Creating and synchronizing like-to-like copies of data

• Focus on performance, multi-directional capabilities

InformationInformation

Bu

sin

es

s f

lex

ibil

ity

& r

es

po

ns

ive

ne

ss

DataData

InsightInsight

Integrate ever evolving data and Integrate ever evolving data and

content, from many different content, from many different

sourcessources……

Transform it into useful Transform it into useful

inin--context information . . . context information . . .

Empower people and Empower people and

processes to take the processes to take the

proper actionproper action……

Information on DemandIncreasing the Business Value of Information

ActionAction

Business value / Ability to Innovate

InformationInformation

Bu

sin

es

s f

lex

ibil

ity

& r

es

po

ns

ive

ne

ss

Data

InsightInsight

Integrate ever evolving data and Integrate ever evolving data and

content, from many different content, from many different

sourcessources……

Transform it into useful Transform it into useful

inin--context information . . . context information . . .

Empower people and Empower people and

processes to take the processes to take the

proper actionproper action……

Information on DemandIncreasing the Business Value of Information

ActionAction

Business value / Ability to Innovate

It begins with It begins with

DATA IntegrationDATA Integration

Data Integration Challenges You Face Every Day

� New applications on diverse platforms drive integration demandsHow do you reuse critical "z" operational data on these platforms?

� Mergers & acquisitions and business intelligence drive consolidationHow do you manage complex transformations to rationalize diversity?

� A shortage of mainframe skills is leading to workload backlogs.Can you reduce dependence on mainframe skills?

� “Data and content volumes are expected to increase by a factor of 10 during the next five years” Gartner, November 2005

How will you deal with these growing volumes?


The Integration Landscape

Business Integration Market Landscape

� Process &/or Information – EAI’s “primary” focus is application assets

…secondarily data

– Federation's is pure data access

– ETL’s focus is transformation

– Replication's focus is placement

– Publishing is an enabler to the rest

� Emphasis on complementary uses– Empower broad EAI-based data access

via Federation

– Use Published “data events” to drive

EAI workflow

– Create aggregated and cleansed data

stores using ETL

BPI

PortalApp Server

ECMText Search & Analytics

WCM

DB

Analytics & Reporting

ETL/Data

Placement/

Replication

Data

Access &

Federation

EAI

Mission Critical "z" Data Can Be DifficultOver 60% of operation data is sourced from System z

Why is system z data delivery an issue?� Proprietary databases – multiple, complex APIs

� Dearth of skills – scarce and becoming more costly

� Billions of lines of mainframe code – cannot migrate

� Integration with modern initiatives not native to this data

What are you doing to meet these challenges?� Hand-coding – COBOL/PLI programs for every data need

– Bottleneck for new initiatives … slows everything down– Dependence on unique skills … proprietary API experts

� The One-Size-Fits-All Approach:

�� Integrate the transactions

… but this can be overly complex, maintenance heavy

�� A unique "copy" for every requirement

… but this can lead to:– Latency errors– Synchronization challenges– Quality issues

VSAM, IAM, flat files

IMS

DB2

CA-IDMS

ADABAS

CA-Datacom

SelfServiceWebSites

SolarisSelf

ServiceWebSites

NTIVR*IVR*

SelfServiceWebSites

Solarise-Commerce

DataWarehouse

The Information-Centric Models

Virtualize(Real-time)

Federation

SQL

Seamlessly integrate with:– e-commerce– Self service– All data is an equal participant

Synchronize(Near Real-time)

ReplicationPublication

Changes

Proactively deliver data to:– Drive integration– Data changes trigger action– Change only updating to:

• Eliminate batch window• Reduce latency

Create & synchronize copies – Distributed operations– High availability– Local copies for performance– DBMS migrations

Move(Scheduled)

WarehouseODS

Data Mart

Consolidation

Repackage & repurpose– Data Warehousing– Operational Data Stores– MDM– CDI


Data Delivery RequirementsApplying the Models

Choosing an Integration Model A multi-dimensional self service environment

� Insurance carrier wants to reduce costs and improve customer and agent satisfaction by implementing self-service environments. BOTH Interactive-voice-response and Web sites

� Today’s Environment– Call center representatives take calls from agents who interface with clients

– Mainframe systems consisting of over 1,000 transactions handle:• Accounting, Policy management, Claims and more

– Technology is• IMS transactions

• IMS, VSAM and DB2 data

• Very complex after decades of application evolution

� Requirements– Minimal disruption on existing call-center

– Staged implementations showing value every 3 – 6 months

– Performance and accuracy are essential to credibility

– Unskilled users (agents and customers) mandate an easy-to-use interface

A Multi-Dimensional Self-Service EnvironmentEAI and ETL are traditional choices

Complexity and volume made them too costly and too time consuming

– Copy the data to Oracle……..… $2M for hardware and software

– Connect through transactions.… 10,000 man hours per system with little reusability

NT

Solaris

AgentsCustomers

Doctor

IVR

SelfServiceWebSites

Agent

Customer

PolicyManagement

Billing

VSAM

z/OS

IMS

DB2

Claims

COBOLExtract

Applications

A Multi-Dimensional Self-Service EnvironmentFederation delivered in a fraction of the time and cost

Provide up-to-the-minute policy, claims and accounting information– Connect interactive voice response (IVR) system to IMS, VSAM & DB2… $250K versus $2M– Connect operational data with self-service Web sites… 200 man-hours versus 10,000

NT

Solaris

AgentsCustomers

Doctor

IVR

SelfServiceWebSites

Agent

Customer

PolicyManagement

Billing

VSAM

z/OS

IMS

DB2

Claims

WebSphere Classic Federation Server for z/OS

� Read from and write to mainframe data sources using SQL

� Standards-based access via ODBC, JDBC, or Call Level Interface

� Multi-threaded with native drivers for scalable performance

� Metadata-driven means:– No mainframe programming required – Fast installation & configuration– Ease of maintenance

� Works with existing and new:– Mainframe infrastructure– Application infrastructure– Toolsets

WebSphere Classic

Federation Server for z/OS

DB2 UDBfor z/OS

VSAM CA IDMS

CA Datacom

IMS Software AGAdabas

MetadataCatalog

WebSphere ClassicFederationServer

z/OS

OS/390

DB2 UDB z/OS VSAM IMS CA-Datacom CA-IDMSAdabas

Data Connector

Data Connector

Data Connector

Data Connector

Data Connector

Data Connector

WebSphere Classic Federation Server for z/OS Complete Implementation

Portal BI Tool Servlet Servlet Clientclass

EJB

AIX, HP-UX, Solaris, JVM 1.2,

Widows NT, 2000, XP,

z/OS, OS/390 ODBC Client JDBC Client

Classic Data ArchitectClassic Data Architect

Copybooks, DBDs, …

Text WebSphere MQ

IBM

Extended

Search

Excel WWW, email,…XML

Biological Data

And Algorithms

Integrating Enterprise Data

WS Classic Federation

Server for z/OS

WS Federation Server

SQL, SQL/XML

Federation Engine

Wrappers and functions

O

D

B

C

Integrated SQL View

Sybase

Informix

SQL Server

Oracle

Teradata

ODBC

DB2 Family

DB2 UDB z/OS

Software AGAdabas

VSAM

CA-IDMS

CA-Datacom

IMS

…

Choosing an Integration ModelYou may need more than one

� Catalog retailer is building an e-commerce site

� Today’s Environment– Call center representatives take orders

– Mainframe order-processing system used by call center reps• CICS transactions

• CA-IDMS data

• Highly-optimized for use by skilled employees

� Requirements– Minimal disruption on existing call-center order processing

– E-commerce site must be optimized for unskilled customer usage

– Performance and accuracy are essential to maintaining business

continuity and reputation

– Must support “shopping”, not just order taking

"Best" solution leverages multiple integration modelsETL

– ETL: copy highly-used product data to application platform • Product information is copied to “local” RDBMS for performance

• Changes are infrequent and can be scheduled

• Value-add image data merged as part of the data transformation

Customer

Customer

Customer

Call-CenterReps

z/OS

e-CommerceSite

DataStage

enhancedproductdata

Ship-toDate Calc

Pricing OrderProcessing

"Best" solution leverages multiple integration modelsETL, Federation

Customer

Customer

Customer

Call-CenterReps

z/OS



• Value-add image data can be added as part of the data transformation

e-CommerceSite

DataStage

enhancedproductdata

Ship-toDate Calc


– Federation: critical “real time” data is dynamically accessed• There can only be one version of the “truth” for inventory

"Best" solution leverages multiple integration modelsETL, Federation and EAI

Customer

Customer

Customer

Call-CenterReps

z/OS



• Value-add image data can be added as part of the data transformation

e-CommerceSite

DataStage

enhancedproductdata

BI Adapter

Ship-toDate Calc


– Federation: critical “real time” data is dynamically accessed• There can only be one version of the “truth” for inventory

– EAI: critical processes are shared• integrate the CICS transactions for pricing, ship-to-date, etc

� Codeless visual design of data flows with hundreds of built-in transformation functions

� Optimized reuse of data integration objects

� Leverages parallel processing without requiring design changes

� Capable of supporting batch and real-time operations

Transform

Transform and aggregate any volume of information in batch or real time

through visually designed logic

Hundreds of Built-inTransformation Functions

ArchitectsDevelopers

WebSphere DataStage®

Deliver

IBM's Extract-Transform-Load Implementation

WebSphere DataStage

Systematic ETL provides a Compelling Value PropositionTool usage (each Project) vs. Point Solutions

Number of Projects

Tim

e a

nd

Eff

ort

Pro

ject

1

Pro

ject

2

Pro

ject

3

Pro

ject

4

Point Solutions

This is the value proposition

Project implementation

Choosing an Integration ModelFederation & Event Publishing can be enabling technologies

� Global technology distributor wants to implement a Web e-commerce environment, reduce inventory costs and improve decision making

� Today’s Environment– Two mainframe-centric transaction processing centers

• Ensures 24 by 7 operations for their Global business

– One Wintel-based support processing center

– Technology environments• Mainframe CICS environment servicing 40 to 50 million transactions/week

• IMS databases hold most of their data

• Wintel-based reporting using Oracle Operational Data Store (ODS)

� Requirements– Implement their Next-Gen transaction platform on WebSphere

• Staying with z/OS & IMS data for performance, reliability and security

– Improve interfaces with their suppliers • Implement "real time" interfaces for a just-in-time inventory environment

– Leverage reduced cost of India data center for non-transactional workloads• Implement an ODS to support business reporting and decision processing

Call-CenterReps

z/OS

Federation & Event Publishing Behind …Transaction processing

– Federation enables switch to Web-based applications• Java-based access to IMS data with transactional performance

Customers

Customers

Payments

ReturnsOrderProcessing

Application Server

Call-CenterReps

z/OS

Federation &Event Publishing Behind …Transaction processing, ETL

– Changes delivered to ODS via ETL • Continuous, 24 by 7 delivery:

– Ensure up-to-the-second data is available– Eliminate dependence on batch cycle

• Steady, predictable usage of communication resources


CHANGES CHANGES

Event Publisher

Purchasing …

Marketing …

BusinessReporting

DataStage

ODS

Customers

Customers

Payments


Application Server

Federation &Event Publishing Behind …Transaction processing, ETL and EAI

– Changes delivered to ODS via ETL • Continuous, 24 by 7 delivery:

– Ensure up-to-the-second data is available– Eliminate dependence on batch cycle

• Steady, predictable usage of communication resources


Partner

Application

Systems

– Changes delivered to Partner Gateway (EAI)• Enables Just-in-Time inventory management

Partner ConnectivityGateway

Call-CenterReps

z/OS

Customers

Customers

Payments


Application Server

CHANGES

CHANGES CHANGES

Purchasing …

Marketing …

BusinessReporting

DataStage

ODSEvent Publisher

� Real time capture and publishing data changes made to:– DB2– VSAM through CICS– Native VSAM– IMS database– CA-IDMS database– SAG Adabas

� Publish changes to WebSphere MQ for z/OS� Consistent “relational” XML format� WebSphere listener application or tool

– Picks up message(s)– Takes action

� Two Event Publisher infrastructures:– DB2 Universal Database for z/OS

(WebSphere Data Event Publisher)– IMS, VSAM and CA-IDMS

(WebSphere Classic Data Event Publisher)

WebSphere Data Event Publishers for z/OS

CICSVSAM

IMSDB2for z/OS

changes changes

CA-IDMS

WebSphere MQ

SAG-Adabas

NativeVSAM

� Log-based & recoverable:– Log exits for active processing

– Log files for recovery processing

� XML message output:– Self-describing XML message format for easy integration

– Consistent format across all publishing solutions regardless of source

� WebSphere MQ publishing:– Common delivery mechanism

– Guaranteed delivery mechanism that spans broadest choice of platforms

– Dominant mainframe messaging infrastructure

– Broad set of IBM & 3rd party solutions can “read” from WebSphere MQ queues

– Publish once – read any number of times

– High performance

WebSphere Data Event Publishers for z/OS

Published data events facilitate business

integration

Usage� Application to application messaging

� Event streaming

� Change-only data distribution

Function� Capture data events in real time

� Publish these data events:– to a message queue for widespread delivery– in XML format for widespread use

Log-based

capture

User Application

DB2 UDB

VSAM

IMS

CA-IDMS

Capture WebSphere MQWebSphere MQ Integrator Broker

Target DBs

ETL or other datapropagation Tool

WebSphere Business Integration

JMS-aware Application


How does Replication Fit?

� Availability– Scheduled outage, failover, disaster recovery

• Can use Hardware, Software, or a combination of methods– Move query or reporting work to a separate system

• Other methods such as flash copy also possible– Peer to peer - split workload

• This is only possible through replication

� Distribution / Consolidation– Move data between central to branches, branches to central, or both– Federate or Replicate?

• where does the application need the data to be? - what db?, what platform?• does the data need to be real time or not?• what is the change volume?

� Warehouse / Business Intelligence / Application Integration– Move data to new platform/database, transform data– ETL or Replicate?

• latency needs• change volume versus total volume• complexity of transformation and/or cleansing

Why Replicate?

Staging tables

WebSphere Replication Server SQL Replication

Log based

Trigger based

External application

IMS

DB2

Sybase

Oracle SQL

ServerInformix

Any

source

Admin

Control

Federation

Engine

DB2

Sybase

Oracle SQL

Server

Informix

Teradata

Nicknames

Apply

CD1CD

CD1CD

CD1CD

Control

Capture

� Flexible scheduling, transformation,

distribution

� Typically used for business intelligence, distribution and

consolidation, application integration

� Homogeneous & Heterogeneous

Sources/Targets

Control

Control

Federation

Engine

Log based

DB2

WebSphere MQ

Q-Capture

TargetQ-Apply

Control

Classic

Capture

VSAM

IMS

CA-

IDMS

SAG-

Adabas

WebSphere Replication Server Q Replication

� Asynchronous replication

� Highly parallel apply process

� Leverage re-architected V8 Capture

� Differentiated conflict detection and resolution

� Integrated infrastructure for replication and publishing

� Target : DB2, Oracle, Sybase, Microsoft SQL Server,

and Informix

Why Q Replication ?

� Performance: Combine high throughput with low latency

� Capability: Significantly improve multi-directional replication support

� New function: Event publishing, table difference utility

� Manageability: Reduce the number of replication objects to be defined and managed, ease the definition process with new Replication Center wizards


Wrap-up

Determining the "Right" Delivery Model:Ask: Who – What – When – Where - Why

� "Why" controls everything – Empower self-service customer environments

– Integrate with suppliers

– Reduce delivery time… support costs… manual interventions

� "Who" and "What" dictate the form and the source(s)– Inside consumer (e.g. claims clerk) versus

Outside consumer (e.g. policy holder, broker, agent)

– Detailed data versus aggregate

– Enterprise versus department

� "When" and "Where" guide the delivery mode– Now… with up-to-the-second accuracy

– Daily… before the nightly accounting run

– Monthly… for the sales reports

Let us help –IBM's Information Integration Assessment

What is it?An interactive session led by an IBM Information Integration Solutions

specialist, to help you take advantage of emerging technologies,standards, and current trends in information integration

What will you get as a result?– Outline of your current information integration environment – Identification of key projects where information integration could

improve IT productivity and speed development – Exploration of alternative approaches– Recommendations for architecture enhancements

� On-site workshop for your team� Information Integration needs assessment� Report and recommendations

Workshop Basics

� Part One: Information gathering and sharing – Information Integration Overview – Discovery –Extensive interviews and data collection

� Part Two: Analysis, summary, recommendations

– Summary of information collected– Review of:challenges, issues, problems, needs

– Approaches considered– Tradeoffs– Recommendations

� Up to 15 team members:

for example:

– Architects• Data architects, information architects• Enterprise architects• Data architects or information architects

– LOB managers in areas like• Compliance (financial services and other

industries)• Supply chain management (CPG)• Clinical intelligence (healthcare)

– Application owners

– Middleware managers

– Data warehouse managers and developers

Typical Agenda Typical Workshop Participants


Thank you for your time

Click to edit Master title style - IBM to edit Master title style Integrating Your Information -...

Documents

Transcript of Click to edit Master title style - IBM to edit Master title style Integrating Your Information -...