Get Started on SOA - IBM - United States · PDF fileGet Started on SOA 05 ... Run QualityStage...

1

Information Entry Point – Information as a Service

Get Started on SOA

05 - Information as a Service v1.5.ppt 2

Information Management Requirements are Increasing

Disconnected Silos of Information

Dynamically DeliverInformation

Delivering information in context to optimize business processes, applications and productivity

Information Platform

People, Processes, Applications

Silos of Business

Information

70% of people’s time can be spent finding relevant information

60%+ of CEOs say they need to do a better job leveraging information

5X More Value creation by organizations effective at using information

Rich Standards,Flexible Architecture

2


Reuse:ServiceCreation

SOA Entry Points

Connectivity:Service

Connectivity

People:Interaction & Collaboration

Services

Process:BusinessProcess

Management

Information:Informationas a Service

SOADesign

SOAGovernance

Service Security, Management and

Virtualization

Getting the right data quickly and consistently for all applications continues to be a key challenge for many enterprises.

Forrester, January 2006

You will waste your investment in SOA unless you have enterpriseinformation that SOA can exploit.

Gartner, March 2005

Service Oriented ArchitectureInformation as a Service is Key


The Need for Information Servers

Trust

Productivity

Collaboration

Scalability

Reuse

Today’s Inflection

Point

Federation Delivery SOA

Replication Quality Transformation

Connectivity MetadataProfiling

3


IBM Information ServerThe SOA Foundation for Information Management

� A New Kind of Platform� Delivers trusted information to

processes, and applications� Foundation for SOA� Dramatically Simplifies Integration

� Innovation Technology� Metadata-driven integration� Breakthrough productivity� Industry-leading scalability� Rich, bi-directional connectivity

� Based on Experience� 5,000+ Integration Clients� 75 Clients in Beta Program� 35+ Partners Enabled…

IBM Information ServerUnderstand, Cleanse, Transform, Deliver

Trusted Informationin Context

New!


Understand Cleanse Transform Federate

Parallel ProcessingRich Connectivity to Applications, Data, and Content

IBM Information Server

QualityStage Federation ServerDataStage

Information Services Director

Metadata Server

Replication ServerData Event Publisher


Business GlossaryInformation Analyzer

Data Architect

Delivers Data Access and Data Integration Services for SOA Environments

4


Shared ServicesAdministration, Logging, Security, Load Balancing, ….

Integrated Metadata Management

Web Services EJB

IBM WebSphere Information Services Director

Quality Stage Data Stage Federation Server

Enterprise Service Bus

Design Operational

Applications

J2EE Applications

Run QualityStage Job Run DataStage Job Run Federated Query

WISD Agent

WISD Agent

WISD Agent

Process Portal

IBM Information Server – Common Programming Model



� Foundation for SOA data applications� Common Services infrastructure� Standard access to Common Services

� Services can be invoked as EJB or Web services� Services can be stored in the WebSphere Service Registry and

Repository� Simplified administration and deployment � Unified reporting, logging, and administration

SOA Framework inside IBM Information Server

WebSphere Information Services Director

Shared ServicesAdministration, Logging, Security, Load Balancing, ….

Integrated Metadata Management

IBM WebSphere Information Services Director

Design Operational

J2EE ApplicationsService

Registry and Repository

5


Service Oriented Finance Needs a Complete View of the Customer

Service Oriented FinanceCIO

Our Relationship Managers need to be able to see all the customer information at once.

IBM

With IBM’s Information Server you can deliver a federated view of the customer data.







Metadata Server




Data Architect

6


Different Types of Integration for Different Needs

� Move data to physical master copy� Large amounts of data moved � Data may require transformation and

cleansing� Typically used for analytical purposes

Master Data

Application

Consolidation(Extract, Transform, Load)

SQL

Sources

Consolidation

Federated Data Access

WebSphereFederation

Server

SQLApplication

Sources

Federation

� Source data remains in place� Small amounts of data accessed

� Typically used for operational purposes


Data sources

DB2Checking

Data

DB2 Savings Data

SQL ServerCredit Card

Data

Informix Mortgage

Data

Customer banking data in DB2

Credit card data stored in SQL Server

Mortgage information in Informix IDS

Query

Query

QueryQuery

Multiple queriesMultiple connectionsHarder to program

Problem: Customer Information Is Stored in Multiple Systems

7


WebSphere Federation Server

Application ProgramData sources

Checking Data

Savings Data

Credit Card Data

Mortgage Data

SQL Query

WebSphereFederation

Server

DB2 wrappers enable access to a variety of data sources as DB2 virtual tables

Data administrator may then define views across tables


DEMO: Federated Data Delivered via IBM Information Server

Data Sources

DB2

SQL SVR

Informix

Create federated query using query designer

Deploy query as a Web Service using IBM Information Server

Service Oriented Finance Portlet uses Web Service to get the federated customer information with a single service request

FederationServer

DataStage

QualityStage

Info

rmat

ion

Serv

erWebSphere Portal Server

Web Service

8



Checking & Savings

…

Credit Card Data….

Mortgage Data

SQL ServerDB2

Web Service

Portlet

IBM Information Server Benefits

� For the Portlet Developer –� Easily use Web Services to access

federated data� No need to connect to multiple

sources� No need to know various source

schemas� Lab tests show 40-60% reduction in

development time and code size

� For the Administrator –� Easily create Web Services to expose

federated data� Data schemas may be changed

without forcing change to application programs Informix


� DB2 UDB � Informix� Oracle� Sybase� Teradata� MS SQL Server� and more …

� Red Brick Warehouse� Classic Federation

� IMS, VSAM, CA-Datacom, CA-IDMS, Adabas

� Ingres� Lotus Notes� Microsoft Access/Excel� MySQL� PostgreSQL� SAS� Sybase SQL Anywhere� and more…

� Web Services� Life Sciences� WBI sources

� SAP, PeopleSoft, Siebel

� Legacy Data Access via WS II Classic Federation for z/OS

� and more …

� Or build custom wrappers� Java and C++ SDK

Non-relational WrappersODBC SourcesRelational Sources

Content Sources

� FileNet� Interwoven� Microsoft � and more …

Sample Data Sources and Wrappers for WebSphere Federation Server

9







Metadata Server




Data Architect


Different Types of Integration for Different Needs

� Move data to physical master copy� Large amounts of data moved � Data may require transformation and

cleansing� Typically used for analytical purposes

Master Data

Application

Consolidation(Extract, Transform, Load)

SQL

Sources

Consolidation

Federated Data Access

WebSphereFederation

Server

SQLApplication

Sources

Federation

� Source data remains in place� Small amounts of data accessed

� Typically used for operational purposes

10


Service Oriented Finance Needs a Data Warehouse

Service Oriented FinanceCEO

All of the branches are reporting good profitability.

But each branch is reporting the business results differently.

IBM

With a consolidated corporate data warehouse. You can consistently analyze branch performance.


Loading your Data Warehouse with WebSphere DataStage

Master Dataor

Data WarehouseExtract

Transform

Load

Multiple databases or files

WebSphere DataStage

SQL ServerBranch Data

SalesFacts

ProductProfitability

Branchs

Product

Star schema tables

Customer Accounts

ProductCampaigns

DB2

11


� Create, execute, and monitor jobs to extract, transform, and load data

� Graphically design, develop, deploy, and reuse ETL jobs quickly and easily

� Deploy jobs in real-time, batch mode, or as services using SOAP and/or EJB bindings

� Parallel Processing delivers Unmatched Function, Performance, and Scalability

DataStage®

ExtractTransform

Load

Extract, Transform, and Load (ETL) Data with WebSphere DataStage


SALEDATEAMTQTYBRANCHID

CUSTID

PRODID

Data Warehouse

Branch Data

Campaign Sales Data Transformation

AMOUNT DATECUSTNOQTYPRODUCT

Transform

000 101 100 01 01 10,000.00 2007-02-28

000 121 100 01 03 500.50 2007-02-28

000 101 101 01 01 20,000.00 2007-03-01

000 101 01 100 10000 2006-02-28

000 121 03 100 500.50 2006-02-28000 102 01 100 20000 2006-03-31

000 101 100 01 01 10,000.00 2006-02-28000 121 100 01 03 500.50 2006-02-28000 101 101 01 01 20,000.00 2006-03-01

AMOUNT DATECUSTNOQTYPRODUCT

SALEDATEAMTQTYBRANCHID

CUSTID

PRODID

Branch Data

Data Warehouse

Transform

000 101 100 01 01 10,000.00 2007-02-28

000 121 100 01 03 500.50 2007-02-28

000 101 101 01 01 20,000.00 2007-03-01

Different field namesDifferent field order

Add Branch IdentifierDifferent currency format

12


DEMO: Run DataStage ETL Job

Branches

DataStage

ExtractTransform

Load

CorporateSQLServer

DataWarehouse


DEMO: Use Same Excel Tool to Do Corporate Level Analysis� Excel accesses data from DB2

for z/OS seamlessly

� Campaign to offer mortgage point deduction most effective…Park City branch shows this

� The Inlet branch has the largest number of new accounts for all of the branches, however that does not necessarily equate to the most profit

13


WebSphere DataStage Enterprise Editions Provide Real Scalability for Big Jobs

Uniprocessor SMP System MPP, GRID, and Clustered Systems

� Design integration process without concern about data volumes or time constraints

� Leverages database partitioning schemes for optimal load performance

� Simple steps to define partitions within each process if needed

� Single configuration file may be reset to add processors, hardware

� No hand coding of programs to enable more processors

� Supports SMP, Clustered, GRID, and MPP platforms


General AccessSequential FileComplex Flat FileFile / Data SetsNamed PipeFTP Compressed / Encoded DataExternal Command CallParallel/wrapped 3rd party appsEMC InfoMoverWeb logsUnstructured: e-mail, docs, etc.Content Management SystemsLife Sciences

Standards & Real Time WebSphere MQJava Messaging Services

(JMS)JavaXML & XSL-TEBXMLWeb Services (SOAP)Enterprise Java Beans (EJB)EDIFIXSWIFTHIPAA

Enterprise ApplicationsJDE/PeopleSoft EnterpriseOneOracle ApplicationsPeopleSoft EnterpriseSASSAP R/3 & BISAP XISiebelJDAAribaManugisticsI2And more…

LegacyAllbase/SQLC-ISAMD-ISAMDatacom/DBDS MumpsEnscribeEssbaseFOCUSIDMS/SQLImageSQLInfomanKSAMM204MS AnalysisNomadNucleusRMS S2000SupraTOTALTurboImageUnifyAnd many more….

RDBMSDB2 (on Z, I, P or X series)OracleInformix (IDS and XPS)IngresMySQLNetezzaProgressRDBRedBrickSQL/DSSQL ServerSybase (ASE & IQ)TeradataUniverseUniDataNonStopSQLAnd more…..

CDC / ReplicationDB2 (on Z, I, P, X series)OracleSQL ServerSybaseInformixIMSVSAMADABASIDMSNonStopSQLEnscribe

IBM Information Server Connects to Almost Everything

14


The Need to Improve Data Quality

Critical Problem� Extracted data may have inaccuracies, errors, and

duplicate data which must be corrected to create a “master”consolidation

Why?� No common standards across organization � Human error� Operational data is estimated to degrade in accuracy 2%

per month� Unexpected values stored in fields� No reliable keys for consolidated views


Data Cleansing with WebSphere QualityStage

New Customers

1. Standardize Format

2. Match/Compare

to data in the Warehouse

Customers Update

Warehouse

Customers

3a. Duplicate

3b. New Customer

OR

15


Service Oriented Finance Needs to Store XML Data

Service Oriented FinanceCIO

We use XML to describe our products.

I need a strategy to store XML data.

IBM

DB2 9 with pureXMLcan manage XML and relational data together


Chart Sources: XML Market Opportunities, Forecasts and Strategies, 2004-2009 WinterGreen Research Inc. ZapThink

� More XML data generated than in databases.

� Growing at twice the rate of the total database market. [IDC]

� XML is pervasive in all kinds of organizations.

� Almost every sector has XML-based standards.

Importance of XML?

“XML is emerging as the standard way to represent data for the purposes of interchange between applications and enterprises across

structured and unstructured sources.”—Gartner Group, “Gartner Study on EIM Highlights Early Adopter Trends and Issues,” David Newman, February 7, 2006.

16


XML Solves Business Problems Today

� Business-to-Business Integration� Platform-independent transport

mechanism.Transaction orders may be defined in XML

� Forms and Document Processing� Government and legal industry require digital

signatureTax forms require signature & change year to year

� Documents often contain sub-documentsLiterary materials contain books, chapters,

and sub-chapters

� SOA� Web Services messages are XML


� Relational is a data model− Relations (tables)− Attributes (columns)− Set based w/ some sequences− Strict schema

� XML is a data model− Hierarchical tree structure− Nodes (elements, attributes,

comments, etc.)− Relationships between nodes− Sequence based w/ some sets− Flexible schema

POID CustomerID ItemID12 1 2162 3 4162 3 5

Id LastName FirstName Street City State Zip

1 Pirahesh Hamid 1 Harry Rd San Jose CA 951413 Selinger Pat 555 Bailey Ave San Jose CA 95141

ItemID Name2 #6 wire nut5 Small Walrus4 Apollo moon rocket

XML – The Difference Is Fundamental

<? xml version=“1.0” ?><paymentRequest id=“12345” >

<payment><payment_name>Loan</payment_name>

</payment><payee_info>

<payee_amt>$97.85</payee_amt></payee_info><remitter>

<name>John Smith Co</name><address>

<street>1234 W. Main</street><city>Yonkers</city><state>NY</state><zip>11111</zip>

</address></remitter>

</paymentRequest>

17


� Simplified XML Data Access� Centrally store and access XML data� SQL or XQuery can be used to retrieve XML data� Join XML data with it’s related relational data

� Search Speed� Search XML documents quickly and efficiently

using search optimization engines

� XML Data Needs Protection� Transactional protection� Backup and recovery to ensure continuity� Security mechanisms for access control

� Optimize Existing Investments� Use existing technology infrastructure and skills to

store and manage both relational and XML

��

XML Data Management Needs Relational MaturityWhat Is Needed?


Solution: DB2 9 Native XML Storage� A database server which supports both the relational and XML data

structures� Adds a new “XML” data type

� A new storage mechanism to efficiently manage XML data� “Native“ means that XML documents are stored on database pages

as parsed tree structures to reflect XML’s hierarchical structure� Separate from the relational storage mechanism

� Native XML support avoids conversions between XML and relationalstructures, and the corresponding limitations. � Input and retrieval are faster, performance is better, and querying is

better and faster.� With conversion techniques like BLOBs and shredding, every

operation (parsing, etc.) is expensive and there is a potential loss of data.

� The XML document might be too complex to shred.

18


DB2 XML Performance Beats Oracle

XML document Insert Performance

2.2

0.0

0.5

1.0

1.5

2.0

2.5

DB2 VS Oracle 10G Structured XML storage

x Fa

ster

DB2Oracle

XML document Mixed Read/Update/Delete Performance

1.5

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

DB2 vs. Oracle 10G Structured XML storage

x Fa

ster

DB2Oracle

XML document Retriaval using multiple Read queries Performance

3.6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

DB2 VS Oracle 10G Structured XML storage

x Fa

ster

DB2Oracle

DB2 9 performs 1.5 to 3.6 times faster than Oracle 10G

on XML operations


60% (3x better)18%ORDERS

Compression Ratio

59% (2x better)29%Entire Database

58% (1.5x better)38%LINEITEM

DB2OracleTable

� TPC-H is a well known data warehouse benchmark� Each vendor uses the same tables and same data� Oracle published their compression rates for TPC-H tables at the

VLDB conference in 2003� IBM ran the same tests on the same tables

� Test results – DB2 reduces cost by requiring less storage

DB2 – Better Compression Ratio Than Oracle

19


Challenge

Solution

Benefits

� Reduce development time and improve response time when using XML data

� Reduce development time and improve response time associated with storing XML

� Schema changes in minutes not days� New database search and retrieval

methods using native XML in minutes…

Implications

� Deeply integrated XML support enables unique client application opportunity

� XML data services critical to SOA implementations

� DB2 9 a combined XML / Relational Information Server

Storebrand Leverages Integrated XML Support


What About Our Competitors?

NONOYES8. Native XML data

LimitedNOBEST7. Data Compression

**LimitedLimited

connectivityLimited

Basic ODBCYES1. Federated database support

NONOYES2. Information Server platform

LimitedNOYES6. Data cleansing and harmonization

LimitedMultiple products

NOYES5. Highly scalable ETL - uniprocessor to true grid

LimitedSQL transforms

LimitedSQL transforms

YES4. Graphical tool for automatic ETL job development with built-in transformations functions

NONOYES3. Cross-product metadata repository

OracleMicrosoftIBM

20


Get Started on SOA - IBM - United States · PDF fileGet Started on SOA 05 ... Run QualityStage...

Documents

Transcript of Get Started on SOA - IBM - United States · PDF fileGet Started on SOA 05 ... Run QualityStage...