© 2015 IBM Corporation

IBM Integration BusEffective Administration and Development

Matthew Golby-Kirk, IBM

What will this session cover?

WMB and IIB have support for two main user roles

• Integration Developer

• Administrator

This session will aim to cover accepted best practice for the typical developer and administrator tasks

• Including some essential “top tips”

Topics to discuss

• Tools of the trade

• Design with IIB features

• Design for performance and scalability

• Design for administration

• Common administrative tasks

I’ll focus on WMB V8 and IIB V9 • but will cover some V10 improvements too!

Development Lifecycle and Environments

• Integration Developer

– Develops message flows, message

models etc.

– Unit Tests on local machine

– Creates archive (BAR) files

containing required artefacts

• Administrator

– Customizes BAR for target

environment (message flow

properties including queues, database

names etc.)

– Deploys BAR to target environment

– Management and operational control

Development Test QA Production

Tools of the Trade for WMB V8 and IIB V9

Integration

Toolkit Command line

Integration

Node

Third Party

Tools

Integration API (CMP)

REST API

Integration

Explorer

Web

Admin

REST API WAS

Admin

Tools of the Trade for IIB V10

Integration

Toolkit

Integration

Node

Command line

Integration API (CMP)

REST API

Third Party

Tools

Web

Admin

REST API WAS

Admin

Integration Explorer

Management option designed for administrators

Plug-in to MQ Explorer

Extra features

− Create/Manage Configurable Services

− Performance Views

− Group integration nodes

− Administration Log

− Administration Queue

− Manage statistics & tracing

− Replaced in v10 with Web UI

BIP1121I: Creates an execution group.

Syntax:

mqsicreateexecutiongroup brokerSpec -e egName [-w timeoutSecs] [-v traceFileName]

Command options:

'brokerSpec' is one of:

(a) 'brokerName' : Name of a locally defined broker

(b) '-n brokerFileName' : File containing remote broker connection parameters (*.broker)

(c) '-i ipAddress -p port -q qMgr' : hostname, port and queue manager of a remote broker

'-e egName' name of the new execution group

'-w timeoutSecs' maximum number of seconds to wait for the execution group to be created

'-v traceFileName' send verbose internal trace to the specified file.

Command line tools

A wide selection of tools for scripting actions

Requires a configured environment

• mqsiprofile or mqsicommandconsole (Windows)

mqsiprofile (Linux/UNIX)

JCL or ISPF (z/OS)

Most commands work against local or remote integration nodes

Some new “IIB” command in V10

Integration API (CMP)

Java interface that enables the administration tools

Use for custom administration requirements

Fully documented and samples available

• WMB v8 and later allows you to create and edit message flows too

– Build your entire system programmatically!

Web Visualisation and Analytics

� A comprehensive tool for web management

• Manage all integration resources from zero-footprint client

• Analyze integration performance in real-time

• Supported on a variety of browsers: IE10, Firefox, Safari…

• Complements MQ Explorer and WAS Admin consoles

� Managing Integration Resources

• View top-level integration node properties

• Add/remove/change integration servers

• Start/Stop integration data flows

• Role based access to control usage

• Advanced options include data replay, policy & monitoring

• Exploits underlying public REST/JSON API

� Integration Performance Analysis

• Operational experience; no developer intervention required– New and existing flows can exploit without change

• Many metrics of integration flow available in real-time– CPU & I/O time shown by default in integration analyzer

– Other metrics include thread, data sizes, errors…

• Flexible display includes data tables and flow profile– Drill down to understand detailed behaviour

• Exploits underlying MQTT web sockets technology– Asynchronous notification at low CPU cost

9

IBM Integration Bus V9

V10 Web UI Administration Improvements

� IIB Web UI becomes the primary means of runtime administration– Browser approach is lightweight and universal– Integration Bus Explorer no longer provided as part of IIBv10

� Programmatic intervention using public Java and REST APIs

� Integration Bus Explorer admin capabilities re-located– Policy Set configuration moved to the IIB Toolkit– Export Port Configuration for external HTTP listeners in Web UI– Integration Server Create, Rename and Delete added to Web UI– BAR file deployment added to Web UI

Significant Tool changes in V10

Single install package and single installed component• Unit test integration node automatically started with IIB development tool

• Can create additional integration nodes as required

No longer any MQ pre-requisite• All existing MQ use-cases still work (and now support remote MQ connections)

• All administration done through the web UI rather than the Integration Explorer

• Remote administration commands need to point at the integration node’s

administration port rather than the queue manager

New “IIB Console”• e.g. iib tools, iib help

• All mqsi* commands continue to work

(noting remote administration caveat above)

Integration Toolkit View for Developers

Create Your Connectivity Solution

Built-in nodes encapsulate transports, technologies and applications• Our intent is always to make the common tasks easy, and the rest possible!

• Use the built-in nodes to reduce the amount of custom code required

• This makes best use of the built-in facilities like activity trace and resource statistics

Integration Bus can interact with a huge range of environments

Transports and protocols

• MQ / SOAP / HTTP / TCPIP / JMS

Data repositories

• File / Database / IMS

Applications

• SAP / Siebel / PeopleSoft / JDEdwards

• CICS / IMS

• .NET

APIs

• SCA

• CORBA

All input nodes can be configured to work with any message format.

Getting data in and out of IIB

Transformation Options

WMB and Integration Bus have several transformation options:▪ Mapping

▪ .NET

▪ ESQL

▪ Java

▪ XSLT

Reflects the importance of transformation in connectivity solutions▪ User-defined nodes supported for Java and C/C++

Every transformation option has strengths and weaknesses!▪ Performance and scalability

▪ Backend integration

▪ Skill sets and learning curve

▪ Developer usability

▪ Portability and maintenance

Use a transformation technology appropriate to the problem at hand!

Subflows are simply message flows that are invoked from another flow

• Input and output nodes in the subflow become terminals in the main flow

• Use subflows to break up large problems into smaller more manageable chunks

Subflows

Message Modelling

Models are needed for parsing, validation and transformation• Models avoid the need to write custom code to parse messages!

Graphical mapper requires models to display the message structure• ESQL editor provides in line validation of code that navigates message trees

XML documents and messages are modelled using XML schemas• XSDs and WSDLs are deployed directly to the runtime – no import required!

Non XML data (both text and binary) is defined using DFDL schemas• Data Format Description Language standardised through the Open Grid Forum

• DFDL schemas replace message set projects and the MRM message domain

• Input nodes support DFDL alongside XMLNSC, JSON, MRM etc

Model based tooling to design and test DFDL schema models• Test parsing, test data generation and parse-by-example all built into editor!

• As with XSDs, DFDL schemas are also deployed directly to the runtime

Message Modelling

Design for Performance

There are several areas that incur a processing cost• Parsing and serialization of messages as they enter and exit the flow

• Business logic such as transformations executed during message processing

• Transformation cost includes general complexity, tree navigation and tree copying

• External resources such as interactions with databases and queue managers

Message flow design is critical to achieve high performance and

scalability• For example using global units of work to achieve transactionality across a flow

Performance is a large (and important) topic in its own right• There is a wealth of information and guidance available online!

Design for Scalability

Large messages can consume large amounts of memory/CPU

There is an industry trend toward larger data sets

• Already apparent in IIB deployments

A flow with a large memory overhead can

• slow down other flows by causing swapping

• in the worst case, bring down the execution group

Recommendations

• Plan ahead - consider whether a message 10 times the current size

would be able to be handled without a problem

• Where appropriate, use large message handling techniques in your

flows ( see Large Messaging sample )

Design for Scalability

Loops in a message flow are exactly like a recursive function call

• So each iteration of the loop consumes some stack space

Avoid loops unless you are certain that the number of iterations will not become large

• Stack overflows will bring down the execution group along with all

running flows!

• There is usually an alternative design that avoids looping

Design for AdministrationMonitoring tools available via Web UI and Integration Explorer• Statistics monitor resource usage across execution groups

• Range of options to update and visualise the resource statistics

• Activity Logging to focus on resource usage

Design for AdministrationUse Applications and Libraries for easier management

Use configurable services to externalize deployment time properties• Allows deployment information to be configured by a system administrator

• For example, credentials for the database nodes and FTP server and port

• Ideal for moving solutions through staging environments (test to production)

Applications, Libraries and Services

Applications package end-to-end connectivity solutions• The concept of an application is shared between the toolkit and runtime

• Applications are deployed and managed as a single unit of isolation

Libraries package resources for reuse (flows, scripts, models)

Resources in an application are not visible to anything else• Use applications to manage your solutions inside an execution group

A Service is an Application with a well defined interface (WSDL)

Patterns for Simplified Development

� Creates top-down, parameterized connectivity solutions

� Reduces common problems in flow development

� Establishes best practices for healthcare integration

� Reduces time-to-value for solution development

� Patterns are easily extended with regular IIB functionality

Pattern Generation

Pattern generation creates the production ready projects• Generated projects reflect the configuration choices of the pattern user

• Configuration is saved so that the pattern can be re-generated if required

Built-In Patterns

Integration Bus provides a core set of built-in patterns

These implement a variety of common scenarios• Web service front end to a MQ based application

• Processing data stored in a file and routing to one or more queues

• Adding a proxy in front of a web service provider

• Processing data from an SAP system and routing to MQ

• Shredding messages and routing to one or more queues

Patterns are selected based on client feedback and field experience

This core set of patterns continues to grow with each release

Pattern Authoring

Pattern becomes even more useful when you can create your own!• Every organization has their own repeating connectivity patterns!

• Pattern authoring is the name we give to this technology in WMB and Integration Bus

We recommend you start with a working solution• One or more projects

Pattern authoring is a design activity• It may be long lived

• It is often not sequential

Using patterns is a top-down activity driven by a requirement, but:• Authoring a working solution is (typically) a bottom-up activity

• So pattern authoring bridges these two different approaches

Patterns have their own development cycle• Pattern Authoring editor supports this design activity

Create Your Working Solution

No change at all - design your solutions as you do today• Pattern authoring does not change the tools you use to create solutions

• The key to a good pattern is to create a good working solution!

Design Your Pattern

Straightforward to create patterns using the Pattern Authoring editor!• Design the user interface which is presented to your pattern users

• Easy to add branding - style sheets, images and other files as required

Packaging a Pattern

Common Administrative Tasks

Planning and configuration

• Bringing a new integration node online

• Making an integration node highly available

• Planning for disaster recovery

• Securing an integration node

Managing integration nodes

• Deployment and redeployment

• Understanding behaviour

• Optimizing and tuning

• Migration

• Maintenance

Bringing a new integration node online

Typical steps required

• Preconfiguration (e.g. OS, userids)

• Installation (IIB, additional software)

• Creating the integration node

• Creating integration servers

• Creating Configurable Services

• Deploying BAR files

• Additional configuration (e.g. security, userids)

And don’t forget:

• Documentation

• Scripting

• Virtualization

Scripting and Automation

Four general approaches for IIB provisioning

• Creating and configuring brokers manually

• Use of commandline tools from within scripts (e.g. shell scripts, Jenkins, Ant,

Maven…)

• Hypervisor images for deployment directly on public or private cloud

• Use of technologies such as Chef and Puppet for all environments (cloud, non-

cloud)

Choose an approach that enables your environment to be reproduced

easily

<project>

<target name=“deploy”>

<cvs command=“checkout” … />

<mqsipackagebar … />

<mqsiapplybaroverride … />

<mqsideploy … />

</target>

</project>

Making the integration node highly available

How do I ensure that the integration node is continually processing messages?

• Active/Active vs. Active/Passive

• Agree SLAs with the business (% uptime)

The integration node includes restart recovery of integration servers

• But this is usually not sufficient on its own

• Does not cover machine restart

Two main options

• Third-party solutions (e.g. VCS/HACMP/MSCS)

• Multi-instance queue managers and integration nodes

Distributed Cache

• Built in cache support in 8.0.0.1 and later.

• Enables HA scenarios

Planning for disaster recovery

What would you do if your primary IIB location goes down?

Distribute IIB to multiple sites if possible

• This introduces data replication and latency concerns

Keep DR concerns separate from HA!

• HA: Systems at a single site with a single configuration

DR: Systems at multiple sites with replicated configurations

• An HA failover can be a planned activity

• DR is unplanned.

Securing WMB and IIB

Simplified administrative security

3 levels of authorisation for administrative actions

• Reading (e.g. View integration node properties - mqsilist)

• Writing (e.g. Change integration node properties -

mqsicreateexecutiongroup)

• Executing (i.e. starting and stopping)

• See IIB Knowledge Center topic bp43540 for more details

On two object types:

• “Integration node” and “Integration servers”

Administrative Security is not enabled by default

• Enable through mqsichangebroker / mqsicreatebroker

Access controlled using MQ queues on the integration node’s queue manager

• Give authority to users or groups

“FileAuth” addition in V10

Managing what’s deployed

• It can be difficult to understand what individual deployed resources are used for

• Applications and libraries can allow you to understand why each file is there

– Application: Encapsulates a single use case or scenario

– Library: Promotes re-use of a shared set of files

– Libraries are statically bound to the application (V8, V9, V10 OB) or shared (V10 only)

• Concepts are shared between developers and administrators

– The developer chooses to create an application or library; the collection is then carried all the way through to the runtime

Backing Up

Consider using mqsibackupbroker to backup the integration node’s configuration

• Ideally, after all configuration changes

• Best run when stopped

• But can be used whilst active as long as not undergoing configuration changes

Also consider any other required resources, for example:

• Database tables

• Source artefacts (message flows, BAR files)

Ensure regular restore testing

• Prove your process works

Pattern to reconstitute (some) message flows from a running integration node

• Available on MQSeries.net

Understanding integration node behaviour

The tools include a lot of information that is useful to the administrator

• Administration queue and log, Message flow and resource statistics

• Warnings when features that affect performance are enabled

• Use this information to understand recent configuration changes

− How the integration server is performing, connected endpoints, etc.

Optimizing and tuning

The tools allow you to modify the configuration operationally

These tweaks more efficient to make than modifying message flows

Encourage developers to create message flows that enable operational tweaks to be made

• User-defined properties, Configurable Services, User-defined configurable services,

Policies

Integration Workload Management• Provide intelligent mechanisms to control processing speed• Most common scenario is to reduce back-end server load• Design allows more policy-based processing over time• Can be applied to new or existing integration data flows

Policy defines threshold limits and relevant actions• Set thresholds for integration data flow throughput• Specify actions at threshold, for example:

– NOTIFY: Higher (or lower) than threshold generates publication– DELAY: Excessive workload will have latency added to shape throughput

Web Console used to manage WLM policy• Sophisticated behaviour controllable by broker WLM policy• Workload can be managed across classes of message flows (e.g. batch vs.

online)• Policies stored in local registry, and dynamically configurable• Developer can also specify limits as integration data flow properties

Controlling Integrations with Policy

0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

100

120

140

160

180

200

Maintenance

Schedule regular maintenance windows

• IBM recommends that you are on the latest maintenance level

• Plan exactly what will be applied and when

• A highly available environment ensures that there is no downtime

Summary

Use the features of WMB & IIB to their full extent!• Message modelling, transformation options, applications and libraries

Libraries and subflows make for simpler designs• Deployable subflows and libraries extend the benefits to the runtime

Consider production and performance early on in your design• In particular, applications and configurable services

Patterns address time-to-value for your solutions• It takes just a few minutes to create a pattern!

Developer Edition• Free edition of IB for use in evaluation, dev, test and production usage

• Fully functional: all nodes available and no time limitations (limited to 1TPS per flow)

Always ensure your environment is reproducible

Treat DR and HA separate

Ensure regular backups

Encourage developers to create message flows that enables operational tweaks to be made

Schedule regular maintenance windows

Thank You.

Questions?