Application Integration ”The devil is in the details”

Application IntegrationApplication Integration””The devil is in the details”The devil is in the details”

ContentContent

• Why is integration an issue?• How do I recognize a decent integration architecture

when I see one?• Could there possibly be something called an

Integration Model?• Problems you might run into as an application

integrator!• Saved by an Application Integration methodology!• Corus Quicklink – What sort of hack is that?

What the heck does it do?

Why is integration an issue?Why is integration an issue?

• Demand of new and improved business processes

• Time To Market • Protection of IT investments• The dream of the Connected Economy

Why doesn’t standards solve the Why doesn’t standards solve the integration problem?integration problem?

• Having many standards is like having none• Standards does not protect you from

semantical differences• Standards are not the solution to the

system legacy

• They are however a goal to aim for.

Why are there so many Why are there so many buzzwords in this area?buzzwords in this area?

• Application integration has become HOT• No generally accepted problem definitions• Complete mixup in terminology

– Standards– Infrastructures– Architectures– APIs– Protocols– Products

What is application integration?What is application integration?

• Seemless interaction between applications• Doing the unintended• Old thing, new requirements• Both but independently information and

access

Real life integration scenarioReal life integration scenario

X

CallCenter

Oracle Applications

Pris

AbalonCRM

ColumbineBroadcast

IntranetExtranet

WebbportalLuvitLMS

AR(Kundreskontra)

AP(Leverantörs-

reskontra)

Personal

SökinterfaceContent

Projects

006BroadcastContent

012Medlemsinfo

och kursregistrering

023Debiteringsunderlag

kursavgifter


reklamProvisionsunderlag


timmar

004Inköps-

information

026Prissatta

debiteringar

021Debiterings-

underlagorder

007Webb-

Content

1999-12-08, lax

028Försäljningsprovision

underlag

CMCASH HR

(HumanResources)

GL(Huvudbok)

FAAnläggnings-

register

005Likviditetsinfo

Systemsamband ÖversiktK-World Prio 2 Snart

SubscriberAuthorization

System(Nätoperatörer)016

KundinfoAuktorisationer

Avauktorisationer

015Betalningsstatus

LedningsinfoDataware-

house

1/831/10-99/00

11/10-99/00

2000-03

2000-03

99/00

1/91/101/9

008Tablå i SI-form

043Innehållsinfo

?

022Kundinfo

020Aktörer och

kontaktpersoner

013Validerad

medlemsinfokursdeltagare

017Studie-resultat

utvärderingar

Mindport(Canal+)

001Playlist

025Tittar-

aktiviteter

010Besöks-

aktiviteter

009Leverantörs-

info

030Asrun

029 Bokföring löner

031Tittarkunder

Kortnr

034Aktuellprislista

032PAP

033Produktinfo

Kunskaps-marknadenEasy-

update

003Tx Förbrukning

035Kursutbud

Web

014Studenter

och aktörerlearning

PARBGC, PG

UC

036Medlems- och kundposterAdresser och kreditfakta

038Valuta till Columbine

Inventory

040Organisations-

info

E-commerce

Press-Master

041Säljare

042Kursutbud

LMS044

BasinfoKodtabeller

How do I recognize a decent integration How do I recognize a decent integration architecture when I see one?architecture when I see one?

Rocky road to integration...Rocky road to integration...

• Complexity increases heavily with number Complexity increases heavily with number of integrated systems of integrated systems

• Heavy impact on the integrated systemsHeavy impact on the integrated systems• System dependencies reduces availability System dependencies reduces availability

for integrated systemsfor integrated systems• Hard to support and change existing Hard to support and change existing

integration solutionsintegration solutions• Integration crosses organization Integration crosses organization

boundaries with “political” discussions as a boundaries with “political” discussions as a resultresult

Complexity O(nComplexity O(n22))

System B

System C

System A

System E

System D

Complexity O(n)Complexity O(n)

System B

System C

System A

System E

System D

Distributed Business LogicDistributed Business Logic

System B

System C

System A

System E

System D

Central Business LogicCentral Business Logic

System B

System C

System A

System E

System D

Synchronous deliverySynchronous delivery

System B

System C

System A

System E

System D

Asynchronous deliveryAsynchronous delivery

System B

System C

System A

System E

System D

Distributed knowledgeDistributed knowledge

System B

System C

System A

System E

System D

Central Knowledge BaseCentral Knowledge Base

System B

System C

System A

System E

System D

Knowledgebase

Ad-hoc integration modelAd-hoc integration model

System B

System C

System A

System E

System D

Integration model with common viewIntegration model with common view

System B

System C

System A

System E

System D

Could there possibly be something Could there possibly be something called an Integration Model?called an Integration Model?

Modelling realityModelling reality

Picture of a Forest

Model of a Forest

Trees AnimalsArea

Litter

Organisms

1..1

0..*

1..1

0..*

1..1

Forest

Sun

Lumberharvest

Use of a modelUse of a model

• Declared semantics of information objects• Declared syntax of information objects

• But then there is operational semantics which is uncaptured by the model because it occurs in the eyes of the beholder

Operational SemanticsOperational Semantics

Trees Animals

Area

Litter

Organisms

1..1

0..*

1..1

0..*

1..1

Forest

0..*

Program------------------------------------------------

Integration ModelIntegration Model

• Contains all application views of information objects + common reference view

• Describes business rules in the form of– Integrity checks– Transformations– Filters– ....

Integrity checksIntegrity checks

• Can be local to an information object IF carType = ”SPORTSCAR” THEN horsePower > 200

• Can also be referential i.e. depending on other information objects Car.driver EXISTS IN DriverLicenseRegister.driver

TransformationsTransformations

• Vital to a integration model because they describe how to go from one information view to another

• Note, transformation can (and often do) ”fork out” or ”fork in” i.e. one information element in one view correlate to several in another information view.

Syntax transformationsSyntax transformations

Project management viewCustomer:•Name 20 chars•Phone number is optional•Identified by a number

Sales Support viewCustomer:•Name 30 chars•Phone number is mandatory•Identified by a string

Finance department viewCustomer:•Name 30 chars•Identified by Phone number

Common viewCustomer:•Name 30 chars•Identified by internal id•Phone number optional•Any other attribute

Semantical transformationsSemantical transformations

Project management viewCustomer:•Created at project start•Deleted at project end

Sales Support viewCustomer:•Created at first contact•Never deleted

Finance department viewCustomer:•Created at first invoice•Deleted 10 years after last invoice

Common viewCustomer:•Common lifecycle for all views which implies new state attributes

FiltersFilters

• Filters are criterias describing when information in different views are eligible for delivery to applications.

Integration ModelIntegration Model

applicationA

IntObject: X

IntObject:U

Y_to_Z

Z_to_X

X_to_Y

Y_to_U

W_to_Y_Z

sem_BU

BU_filter

sem_BW

BW_filter

sem_AX

AX_filter

publishingfrom

application B

publishingfrom

application Bsubscription toapplication A

publishingfrom

application A

subscription toapplication B


1

0..n

1

0..n

IntObject:Z

Proxy modelfor

application A

Commonreference

model

Proxy modelfor

application B

IntObject:Y

IntObject:W

Use-casesUse-cases

• Use-cases are good practice in modelling a integration scenario

• Use-cases translates to an chain of operations in the integration model

application AIntObject: X

IntObject: U

Y_to_Z

Z_to_X

X_to_Y

Y_to_U

W_to_Y_Z

sem_BU

BU_filter

sem_BW

BW_filter

sem_AX

AX_filter

publishingfrom

application A


1

0..n

1

0..n

IntObject: Z

Synonymmodel for

application A

Commonglobalmodel

Synonymmodel for

application B

IntObject: Y

IntObject: W

Error handlingError handling

• Since there are no good receiver of errors in an integration scenario. Error situations and their counter actions must be modelled

• It is common that the error situation use-cases outnumbers the normal operation use-cases (if not, you might have neglected some possible error situations)

Problems you might run into as an Problems you might run into as an application integrator!application integrator!

Some important problem casesSome important problem cases

• Multiple master synchronisation• Asynchronous information correlation• Message operation calculations

Multiple master synchronisationMultiple master synchronisation

System BSystem A

Customer CustomerIntegration

environment

System BSystem A


environment

Loops

Differentlatencytimes

Once per hour

Once per second

Asynch. information correlationAsynch. information correlation

System B

System A

Customer

Order withvalid

customer

Integrationenvironment

System C

Order

Once per hour

Once per second

Message operation calculationsMessage operation calculations

System BSystem A


environment

System BSystem A


environment

Good update

Bad create

Good create

Bad create

Good remove

”Do nothing”

Problem summaryProblem summary

• Technical by nature• Asynchronous communication gives extra

complexity (Asynch comm is however necessary to have loose integration)

• The solutions of these problems involves the integration model

Saved by an Application Integration Saved by an Application Integration Methodology!Methodology!

IS-supported business processesIS-supported business processes

Businessplan

Placeorder

PayFind

productUse

product

Selling

Production

Marketing

Deliver

Invoicing

Production

Sales

Customer

Finance

Information systemsplan

Webportal PDM

OACRM

order

Productdef.

customer

Payingcustomer

Productdef.

Top-down approachTop-down approach

From process-analysis to mapping bit and bytes,preferably with the same tool and methodology!

Order_header

order_id number(20)customer varchar(30)status varchar(1)

Order

Order varchar(10)member varchar(30)

rules

and m

appin

g

Utilities and common resources

Methodology anatomyMethodology anatomy

Integration process

ImplementationBusinessanalysis

feedback

Anatomy; continuedAnatomy; continued

Integration process

Overall control of an integration project. Assures that relevant benefit is delivered to customer

Business analysis

Assure that the integration will be relevant according to business goals. The business analysis is independent on the integration tool used.

Implementation Defines an efficient way to implement the ”whats” of the business analysis

Data access Subpart of implementation, dealing with connectivity between applications. - How can applications be accessed?

Information content

Subpart of the implementation, dealing with transformation and interpretation of information - How should information be handled when transferred between applications.

A closer look at business analysisA closer look at business analysis


Integration process


feedback

• Goal model• Process / activity model• Conceptual model• Application-Activity-Object model• Relatively small part of projects and when it is necessary!

Business analysisBusiness analysis

Workshops

1. Current state

2. Future state

3. Investigate the need for integration

4. Evaluate for solution

Goal modelGoal model

«sub-goal»Supply chainmanagement

«mean»Business

automation

«goal»Business

continuance

«mean»Focus on core

business

«quantitative goal»Increased profit

prerequisite forfacilitatesfacilitates

«problem»No XML order/invoice capabilities in financial

application

«quantitative goal» Minimize costs

«mean»1) Obtain Web-system that

facilitates XML-order/invoice 2) integration of order/invoice

between Web and financial appl.

Mitigates problem

enables

contradictory

Is part of

To create a relationship between organization intentions and the

objectives for integration

Process/activity modelProcess/activity model

PayBuyUse

product

Sell

Produce

Marketing

Deliver

Invoicing

Production

Sales

Customer

Finance

Identify organization business processes and their interaction

Process/activity model; continuedProcess/activity model; continued

Identify:

• the activities within the processes

• the need for information

• where the information is available

• information produced

• where the created information is stored

Sell

register customer

place order

Identify prospects

Prepare prospects

Conceptual modelConceptual model

• Defining the participating business objects and their associations

• Optional• UML class diagram

ProductOrder

Customer

is placed by

contains

Organization

Person

Application-Activity-ObjectApplication-Activity-Object diagram diagram

OA

OA

OA

WEB

CRM

WEB

CRM

cust

om

er

PDM

PDM

WEB register customer

CRM register customer

place order

PDM

pro

du

ct

Ob

jec

t s

ou

rce

Ac

tiv

ity

p

erf

orm

ed

in

Ob

jec

t ta

rge

t

product development

pro

du

ct

cust

om

er

production

Pro

du

ct

ba

lan

ce

Pro

du

ctb

ala

nce

cust

om

er

ord

er

Business analysis; continuedBusiness analysis; continued

Strict integration focus:

• Eliminate activities where no information flow is prevalent

• Order between activities are not important• Where activities can be performed are important• (potential) source / target of information are important

Result of business analysisResult of business analysis

• Consistent view on integration need - or alternatives to integration!

• Integration goals and means mapped against organization intentions or visions

• Input to implementation resulting in transparency between analysis and implementationapplication-activity-object diagram application collaboration diagram

• Conceptual model - optional, used for consistency control

A closer look at implementationA closer look at implementation


Web-site

Integration process


feedback

Implementation

Data access

Informationcontent

Application collaboration diagramApplication collaboration diagram

• Integrated applications described as subsystems• Only data flow (normally asynchronous)• Integration objects or their sub-sets• No attributes or similar details • Collaborations outside integration for visualization

billing info (p

ay date)

order

order (shipping date)

Customer, billin

g info

Storagesystem

Financesystem

WebPortal

Spedition

Integration syntax and semanticsIntegration syntax and semantics

Project management:

Customer is: • Anybody within an active project• 20 chars in name• phone number is a mandatory attribute • identified by a number

Sales department:

Customer is: • anyone which may be involved in business• 30 chars in name • phone number is a non-mandatory attribute • identified by a string

Finance department:

Customer is: • Legal entity which has been involved in a business transaction• 25 chars in name• identified by organisation number (mandatory)

Reference:

Customer is: • anyone which may be involved in business• 30 chars in name• identified by internal key (mapped against a string, phone number and a number)

Information contentInformation content

Sale support system

Projectmanagement

Finance system

Proxymodel

Integration model

Reference model

Businessrules

Use case diagrams Use case diagrams - - for requirement specificationsfor requirement specifications

Actors: integrated applications

Publish use case: responds to creation, update or deletion of integration objects in publishing application with C, U or D in ref. model

Subscribe use case: C, U or D of integration objects in subscribing client application as a result of C, U or D in ref. model

Application A«extend»

Application B

Publish use case

Subscribeuse case

Use cases; continuedUse cases; continued

Publish from WEB

Reference model

Use case Scenario (condition) Result

Customer is created - FIN: null

Order is created - LOG: Order is shipped

FIN: Billing info is created

Use case Scenario (condition) Result

A customer enters member information

- Customer is created

Order is entered Related to a customer Order is created

Not related to a customer Error case

Compilation of integration modelCompilation of integration model

• Proxy models and reference model defines static structure of information (relations between integration objects, i.e. referential integrity)

• Data flows in application collaboration diagram defines which transformations must be created

• Use cases describes the behavior of the transformations (and events for integration)

• Attribute mapping is a part of defining the transformations (may be separated later on)

Class diagrams for integration modelClass diagrams for integration model

-Organization no-Name

Customer *

Project *

1

0..*

Proxy model A

-Organization no-Name

Customer

Project

Reference model

-Customer-Contact person-Project no

Project

po6

po4

po3

p

Proxy model B

po5

po7

1

0..*

po1

po2

po3

po4

Integration model:- Proxy model- Referential integrity constraints- Reference model- Post-operations

Corus QuicklinkCorus QuicklinkWhat sort of hack is that ?What sort of hack is that ? What the heck does it do?What the heck does it do?

Where does it come from?Where does it come from?

• Based on experiences made from a number of integration projects

• Theoretical reasoning • Demands from customers, implementers and

maintenance staff

Corus ArchitectureCorus Architecture

i BUILDER• i CONTROLLER• i CONFIGURATOR• i USER

Generators

HTTP Server

Servlet Engine

Repository

i BUILDER server

ODS

HTTP Server

Servlet Engine

i ENGINE

Participant

UI layer

DB layer

App layer

iDM Participant

UI layer

DB layer

App layer

iDM

Integration object - Quicklink atomIntegration object - Quicklink atom

Table

Event log

History log

View Messagetrigger

Pre OP Post OP

Operational Data StoreOperational Data Store

.

.

.

.

.

. .

.

.

.

.

.

. .

System accessSystem access

Client application A

Integration Engine

Log fileINI file

DialogManager

Java VM

ODS

Client application B

DialogManager

Log fileINI file

Java VM

Plug-in Plug-in

AP

I

Genericqueuetable

Genericqueuetable

Publishingadaptor

Subscribingadaptor

Feature example - Pending eventsFeature example - Pending events

• Solves the problem with inconsistent messages– what can the publishing application do with an error?

• A pended message stays in the message table and post- operation is not performed.

• Can self-heal

Integration engine

Publishingsystem

Subscribingsystem

Subscribingsystem

Feature example - Consistency monitorFeature example - Consistency monitor

• Autonomous process checks consistency• Can release pending events

Integration engine

Publishingsystem

Subscribingsystem

Subscribingsystem

CM

The Corus Integration ProcessThe Corus Integration Process

Order process Production

Information Model

Data Model

Application 1 Application 2

Information Model

Data Model d i g i t a l d i g i t a l

d i g i t a l WANIS/IT

IA

RepositoryGenerator

// Kundinfoadaptor// for system sysA

procedure send_kundinfo{ for columns 1 to 3 do { store Kunder.col(i); } … //user part begin call datum_check //user part end}

Corus Methodology Corus Methodology (Implementation)(Implementation)

Goal model for integration automationGoal model for integration automation

Contradictional to

Methodology

leveraged by

SimplicityFlexibility and

advanced functionality

Rapid implementation

Solving all types of integration

Competetive integration

PrerequisiteforFacilitates

Prerequisitefor

Facilitates

Basic principles of Corus methodologyBasic principles of Corus methodology

• Top-down approach– from process, all the way down to the bits and bytes

• Iterative development and successive implementation • Transparency

– no hand-over– single, minimum documentation– all deliverables persistent in one single repository

• Problem split – technology vs. business– decisions will be made by correct persons

• Plug-able. Can be adopted to RUP or PROPS• Middle of the road (solve 80 %)• Standards where possible

– UML for documentation and visualization

But note!But note!

• The methodology is not absolutely necessary for integration with Corus Quicklink– A solution can be obtained ad-hoc– Quicklink is central

.... though highly recommended

Application Integration ”The devil is in the details”

Documents

Transcript of Application Integration ”The devil is in the details”