ESP POC Findings

31
Enterprise Service Bus Proof of Concept To the service bus and beyond... Administrative Technology Services: Enterprise Applications

description

ESP POC Findings

Transcript of ESP POC Findings

Page 1: ESP POC Findings

Enterprise Service Bus Proof of Concept

To the service bus and beyond...

Administrative Technology Services:

Enterprise Applications

Page 2: ESP POC Findings

Agenda – Lisa Justiniano

Page 3: ESP POC Findings

Agenda

Goals Objectives Introduction to ESBs Benefits Recommended Solution Impact Timeline Q & A

3

Page 4: ESP POC Findings

Project Goal Problem: We keep “inventing” different ways to accomplish similar integration tasks, leading to a variety of software that needs to be maintained, supported and enhanced by staff who understand that particular technique.

Solution: Provide standard ways to achieve these common integration tasks, thereby simplifying our software base and making it more easily maintainable and accessible by our partners in a consistent manner.

4

Page 5: ESP POC Findings

POC Goals

1. Get experience with Enterprise Service Bus (ESB) software in general and with 2 or 3 specific ESB products.

2. Determine if an ESB could address our problem statement.

3. Identify an ESB that might best fits our needs.

4. Achieve these goals in three months (April – June, 2014)

5

POC Team • John Marks (GMAS Practice)

• John Shen (PeopleSoft Practice)

• Sreeni Gunnala (FinProc Practice)

• Bill Brickman (HDW Practice)

• Mike Thomas (Lead Architect)

• Lisa Justiniano (Sponsor)

Page 6: ESP POC Findings

Objectives – John Shen

Page 7: ESP POC Findings

Objectives

• Adopt a standard, very high-level language for building integrations between systems.

• Replace existing solutions that we use to integrate systems, currently written in a variety of languages including:

– Perl

– ksh, bash, awk, sed

– SQL*Plus, SQL*Loader

– Java

• Stop implementing a different solution for the same general problem over and over.

• Make our integrations between applications more resilient by reducing or eliminating timing dependencies.

7

Page 8: ESP POC Findings

Objectives

• Utilize a centralized solution in order to: – Reduce cost and time to production

– Improve operational transparency

– Reduce security complexity and overhead

• Deploy a low cost, open and extendible solution.

• Transition slowly and incrementally over time to avoid major disruption and allow measured adoption.

8

Page 9: ESP POC Findings

Introduction to ESB Software - John Marks

Page 10: ESP POC Findings

Introduction to Enterprise Service Bus Software

• Two Logical Components 1. ESB

2. Message Queues

10

<from uri="file:/data/db/input"/>

<setBody><xpath resultType="java.lang.String">/query/lastName</xpath></setBody>

<to uri="sql:select count(*) as cnt from person where last_name=#?dataSource=myDS"/>

<setBody><simple>${body[0][CNT]}</simple></setBody>

<setBody><simple>${bodyAs(String)}</simple></setBody>

<to uri="file:/data/db/output"/>

• Sample Code:

• monitor a folder • xpath extract • sql query • write result to file

Page 11: ESP POC Findings

ESB Functionality

• Host web services and call external web services

• Monitor folders (local and remote) for incoming files

• Transfer files to remote systems

• Query and update databases

• Query LDAP / AD directories

• Send and receive JMS messages

• Send and receive email

• XML parsing, searching etc.

• Call shell scripts, Java bundles

• Do SNMP queries

• TCP sockets

11

Page 12: ESP POC Findings

Benefits – Bill Brickman

Page 13: ESP POC Findings

Benefits

• Faster to market.

• More easily maintainable integrations: – Fewer lines of code.

– Simple, URL-like, syntax.

– Code is provided for us, we just write the configuration.

• Integration libraries (DB, web services, LDAP & AD, etc.) means we can do things we wouldn’t have attempted before.

• Transformations become possible, both in payload and in transmission protocol, without modifying the source or target systems.

13

Page 14: ESP POC Findings

Benefits

• To make both synchronous and asynchronous integrations easier.

• To enable all applications to take advantage of asynchronous integration, meaning:

– Applications are decoupled so dependencies are relaxed: • One application can be taken down for maintenance without notifying or

affecting other applications.

• Less worry about missed time windows (as with current batch transfers).

14

– Move from batch to transactional: fresh data gets to users faster.

Page 15: ESP POC Findings

Benefits

• Enforce generic and modular coding style

– Many small reusable bundles of code.

• Improve operational transparency

– One place to see source of issues and usage trends.

• Reduce security overhead – a common gateway where

security is already controlled (ACLs and logins).

• Centralized simplicity vs. point-to-point complexity

Page 16: ESP POC Findings

Benefits

16

• Eliminate the “daisy chain” method of obtaining information so that:

– Data gets to the end-users faster

– Subscribers don’t need to know much about the source systems

– There is a single source of “truth”

Page 17: ESP POC Findings

Recommended Solution - Mike Thomas

Page 18: ESP POC Findings

Recommended Solution

18

We looked at several products: – Mule ESB

– Apache ServiceMix

– Red Hat jBoss Fuse three closely related and use ActiveMQ

– Red Hat jBoss Fabric8

– Informatica

Our recommendation: ESB Message Queue

– Fabric8 for cloud deployment - ActiveMQ

– ServiceMix for conventional deployment

• Free, open source, standards compliant

• Multiple vendors (Apache – free; Red Hat – QA and support)

• All support fail-over and load balancing

Page 19: ESP POC Findings

Recommended Solution – Deeper Dive

• Fabric8 and Service Mix have common components: – OSGi Containers: Apache Felix, Apache Aries (deploy bundles)

– Command line UI: Apache Karaf

– Integration Library: Apache Camel (routes)

– Web Services Library: Apache CXF

19

– Many more

• Fabric8 adds cloud deployment features.

• Developer (standalone) setup in 5-10 minutes.

Page 20: ESP POC Findings

Recommended Solution – Deeper Dive

• ActiveMQ JMS message broker

– supports both kinds of message queues:

• point to point

• publish / subscribe

– backed by our choice of persistence: Kaha DB, AMQ storage, JDBC (MySQL, Oracle, etc.)

• If necessary we will evaluate alternatives to ActiveMQ like: – Oracle DB JMS implementation

– Red Hat A-MQ

20

Page 21: ESP POC Findings

Recommended Solution - Architecture Some possible configurations:

A. Multiple message brokers, private to specific groups of applications.

B. Single message broker, each application has its own ESB.

C. Single ESB and single message broker.

D. Shared message broker, ESBs private to groups of applications.

21

Page 22: ESP POC Findings

Recommended Solution - Architecture

• In the future we may need to move to a more complex architecture: – for performance reasons

– to isolate populations (of developers, or systems, or even users) from one and other

• Lowest cost and provides most benefit in terms of shared security, logging and auditing.

• Reduced deployment time (infrastructure and ACLs are already in place).

• Developer environment and deployables are the same regardless of which architecture we choose, so is possible to change architecture over time.

22

Our recommendation:

Page 23: ESP POC Findings

Impact – Sreeni Gunnala

Page 24: ESP POC Findings

Impact

• Impact on ATS Developers: – New language, new deployment mechanisms.

– First integration might be hard. Second one is easy. • Training materials will be available.

– Reduction in labor and drudgery required to build integrations

• Impact on ESB POC team: – Outreach: demos, brown bags, wiki, and training materials.

– Best practices for error handling, logging, integrating with scheduling and monitoring tools, and security and authorization.

– Build-out TEST and PROD infrastructure (including evaluating alternatives to ActiveMQ like Oracle DB’s JMS).

24

Page 25: ESP POC Findings

Impact

• Impact on Integration partners: – Will need to learn how to “subscribe” to a service

– Will need to learn how to “consume” a service

– Will need to assist with testing

and they... – Will get new options for integrating with Enterprise Applications

– Will see improved reliability.

– Will see reduced time to market for new integration requests.

25

Page 26: ESP POC Findings

Next Steps – Lisa Justiniano

Page 27: ESP POC Findings

Next Steps

• Immediately build our TEST environment

• Each Practice (Enterprise Applications) will identify one or two integrations that they would like to deploy via this new ESB as part of their FY15 Innovation Goal

– New integration

– Existing Service

– Change management implications

• Confirm our architectural choices

• Expand infrastructure to support load balancing and fail-over.

• Build our PROD environment

• Deploy and stabilize

27

Page 28: ESP POC Findings

Next Steps

1.Prepare TEST Environ: months 0-4 – Install and configure bare-bones

infrastructure

– Determine and document best practices

2.Build Integrations: months 2-18 – Build and test internal integrations

– Verify assumptions

– Assess learning curve

– Assess support issues

– Confirm low impact and minimized risk

3.Rollout Pilot: months 8 –18 – Upgrade infrastructure for fail-over / load

balancing

– Build PROD environment

– Deploy Pilot Integrations

– Stabilize PROD environment and integrations

– Refine training materials

4.Rollout to ATS: month 18 & beyond

28

Prepare

Build

Pilot

Rollout

Page 29: ESP POC Findings

Q & A

Page 30: ESP POC Findings

Example Camel Route <!-- 1. Monitor folder for new files and use XPath get the surname to query from the XML input file. -->

<from uri="file:/data/db/input"/>

<setBody><xpath resultType="java.lang.String"> /query/lastName</xpath></setBody>

<!-- 2. DB query. -->

<to uri="sql:select count(*) as cnt from person where last_name=#?dataSource=myDS"/>

<!-- 3. Extract count and convert to string. -->

<setBody><simple>${body[0][CNT]}</simple></setBody>

<setBody><simple>${bodyAs(String)}</simple></setBody>

<!-- 4. write to file. -->

<to uri="file:/data/db/output"/>

30

Page 31: ESP POC Findings

Contact [email protected] ( x 60093 ) for more information.