IBM Haifa Research Lab

IBM Haifa Research Lab

© Copyright IBM Corporation 2007

Event-Driven approach for Real-Time Enterprise

Real-Time Middleware Seminar 2007

David BotzerSoftware Tools & Event-Based SystemsIBM Haifa Research [email protected]


� IBM Haifa Research Lab | Real-Time Middleware Seminar 2007 © 2007 IBM Corporation

Main goals

1. Looking at “Real-time” in a broader perspective: Business Processes in Enterprises in addition to Operational System

2. Looking at the “Event world” in a broader perspective

3. “Event Processing” for Real-time requirements in RTE



A short one…

• The manager and the employee are both computer scientists

• The manager to the employee: “Your algorithm is fine, but the complexity of the algorithm is not good enough.”

• The employee: “But it works fast…”

Shocked?



A short one…cont.

• The manager to the employee: “Your solution is fine, but it does not satisfy our real-time requirements

• The employee: “But it works fast…”

Should be the same shock…What about: Sensitivity?Worst case?

Predictability?Is it deterministic?



Categorization of Real-Time Requirements



Active Integration/RTE – “Motivation” citations

“ The Real-time Enterprise was the central theme at Gartner's Symposium, with an emphasis on the importance of time in today's competitive and connected environment.”

“RTE will require a working style that is dependent on smaller faster tasks and is more interrupt driven. Gartner predicts that >80% of enterprises that lead their respective industries in revenue growth will have implemented a real-time "enterprise nervous system" for integrating applications within and outside the enterprise (0.8 probability).”

“A study entitled "active integration of operational and business information system" concluded that RTE is potentially a big opportunity to grow a top line, and recommended that a competency center is needed for incubation of technology and test beds.”

“The IBM Academy workshop on "Straight Through Processing" [STP] in the financial markets concluded that Forty percent of all failures in the market are caused by missing, inaccurate or incompatible data, at a certain time. Time constrained exception handling (e.g. violation of integrity constraints when information moves from one system to another) is a vital part of the goals of STP.”



RTE – Gartner Definition

“The RTE is an enterprise that competes by usingup-to-date information to progressively removedelays to the management and execution of itscritical business processes.”Gartner, 2002



RTE

� We all know the importance of “real-time” in the enterprises� Related concepts:

RTA – Real-Time Analytics ZLE – Zero Latency Enterprise STP – Straight Through Processing

� But:We do NOT want to see histograms of expenses and revenues bouncing up and down in real-time. We do NOT want senior management making real-time decisions, so they can make even more decisions every day.

� So where is “real-time” delivering value today?



Traditional RTE

� Traditional - Improving the efficiency of operational systems.

Examples: Fraud Detection, Electronic Trading, Call Center, Operational “Control Panel”

� Focused on transactions and how these events can be handled more efficiently.

� Address “speeds and feeds” issues. � Capable of monitoring hundreds of thousands of

transactions per second and apply complex rules to these transactions in real-time.

Example: Every credit card usage - a real-time system analyzes it for possible fraudulent use.


IBM Haifa Research Lab | Real-Time Middleware Seminar 2007 © 2007 IBM Corporation

RTE – More Requirements

� Context to the real-time monitoring of operational systems� Example

Not useful - simply alert a manager to an event that has exceeded some thresholdUseful - alert the manager to a specific event within the context of other related information and relevant history

� The Goal: Managers have more information at their fingertips and ability to analyze that information. Improve the efficiency of the detection and reaction to deviations to plan. � Real-time monitoring� Context and analytics necessary to complete the solution.


IBM Haifa Research Lab | Real-Time Middleware Seminar 2007 © 2007 IBM Corporation

Value of RTE within Business Performance Management

� It seems that the greatest value of “real-time” for the enterprise is not (only) within the operational systems but within Business Performance Management.

� “Real-time” within operational systems is localized within a specific application.

� “Real-time” within Business Performance Management can greatly impact the success of the entire organization.



RTE

� Gartner: “The true value of “real-time” is only realized when an enterprise could quickly use that information to better compete by having reduced the delays in the management and execution of its critical business processes.”

� For example:Organizations did not have any problem detecting the events on 9/11 that changed the world and their business environment. They did not need sophisticated “real-time” monitoring systems to determine that things had changed. However, most companies found they could not modify their plans and realign their operations in “real-time”, not even close. Most organizations took months.



BPM

� Reduce the latency between the event and the re-alignment, communication and execution of strategy and plans.

� Organizations, which employ BPM solutions gain competitive advantage because they are the first to gain insight and take action.

� BPM software should enable organizations to translate strategies into plans, monitor execution, and provide insight to improve financial and operational performance.

� Managers and decision-makers need to leverage this insight to anticipate results, continuously adjust, and steer their business in the right direction.

Essential to the RTE.



BPM and Real-time (IBM)

� “BPM is an integrated part of on-demand enterprise”� “Real-Time responsiveness needs complete interaction

with data”� “We need an adaptive, integrate enterprise that can

optimize operations across value-chain in Real-Time…”



Ten Critical Steps Toward a Real-Time Enterprise

1. Real-time Visibility2. Real-time Management3. Business Process Modeling4. Process Automation5. Rapid Deployment of Solutions6. End-to-End Integration: a BPM-based approach to "comprehensive

integration among computing platforms, applications, people and processes.”

7. Flexible Infrastructure: Real-time enterprises need to react quickly to both business and technology changes.

8. Service-Based Architecture: component and service-oriented architectures provide maximum agility and flexible deployment options that isolate and enable infrastructure changes.

9. Support for Standards10.Organizational Agility



RTE value

1. The application of “real-time” to transactions delivers value to the enterprise by improving the efficiency of operational systems.

2. The application of “real-time” to the enterprise through Business Performance Management delivers value by reducing the time from event to insight and by improving the performance and efficiency of the entire enterprise.



RTE is not...

� Enterprises cannot buy it – there is no such product…

� Not a single piece of software but a comprehensive portfolio of technology and applications



RTE is…

A perception that integrates people, technologies and applications in order to …

� Automate, accelerate and improve the efficiency of critical business processes

within the enterprisewith customers between trading partners

� Help companies gain competitive advantage by being more responsive and proactive in the face of current business processes

� RTE is required by any enterprise that requires Active Integration for timely business decisions in order to reduce operational cost and risk, and to increase scalability

� Active Integration: Integrate different sources, protocols, semantics, Authorizations, etc. in a non-intrusive and light weight



RTE – Real Time Challenges

� Provide data immediately it’s entered and as soon as it’s needed (ZLE)

� Consider time constraints� Consider task’s criticality to business profit � Data may have time semantics as well (e.g.

creation time)

��

��

��



RTE Architecture

� The motto:��

��

�� ! �

RTE Controller

Effect

Decide Derive

Detect

Sense

� The look:



RTE Component

� A plug-in that performs one or more RTE activities� Uses manifest that summarizes different aspects of the

component’s behavior� Desired properties

Time constraint enableInterruptibleDynamically prioritizedNon-starvation mechanismSecurity requirements fulfillmentNon-intrusive


�� IBM Haifa Research Lab | Real-Time Middleware Seminar 2007 © 2007 IBM Corporation

RTE Architecture – Cont.� RTE controller –

a programming unit that allocates time and RTE tasks to RTE components in order to complete RTE transactions in a timely manner for the achievement of maximal benefit as defined in the RTE process.

RTEController

Effect

Decide Derive

Detect

Sense

RTE Process

RTE Process

RTE Process

RTE Process

RTE Process

RTE Process

RTE Controller

VisualizeIntegrate informationAnalysisControl TimeManage Policies



Straight Through Processing

� Key drivers resulting in a need for straight through processing:Cross-Border Investing increases the number of parties involved and implies multiple standard use. This means more errors and failures in transactions.T+1 settlement cycle (instead of T+4) requires fundamental changes in the current business process. Detecting and responding to exceptions are many times manual, thus time consuming and costly.Ever growing volume of trade leads to increase in the absolute number of errors.



Real Time Exception Handling

� Exception handling is very expensiveThe later an exception is detected, the more costly its handlingwill be.Research shows that over 35% of trade processing costs involves manual handling of rejected transactions of some sort.

� Real time exception handlingReduces the time required to detect an exception.Reduces the volume of manually resolved exceptions.Even when manual intervention is required, it speeds up problem resolutions.



STP Exception Handling in an RTE environment

� Exception handling for STP: a typical process that could be managed by the RTE perception. The tasks involved in handling an exception are all covered by RTE

� Taking into account different aspects:Data integrationAnalysisCollaborationTime constraints

� RTE aim: Integrate lightweight non-intrusive components into the system so that the integration with the existing process will be as smooth as possible.



RTE tasks for STP

� Sense – receiving a message from an STP component (e.g. a trade order message, or a trade order exceptionmessage)

� Detect – identifying that an exception in the STP process has occurred.

Simple (e.g. stock price is below 0)Complex in the sense that it involves correlation of messages,time notion, message absence, etc.

� Derive – deriving information required for the resolution of the detected exception such as relevant attributes out of the exception, general data…

� Decide – deciding how to resolve the exception by analyzingand/or consulting with other (human or programmed) units, taking into account time constraints and other constraints.

� Effect - executing the stabilization method resolving the exception in the STP process.



Active Integration in RTE

� Active Integration in a Real-time Enterprise environment is the ability to perform a closed loop of tasks (e.g. sense, detect, derive, analyze, collaborate, decide, effect)

� Each of the tasks may include intelligence, analytics or contextual awareness and may have timely manner requirements

� The entire process is controlled by a “real-time” controller� A Real-Time transaction is a sequence of tasks that is handled as

one unit for the purpose of processing an event in a timely manner(e.g. processing of an insurance claim)

� Different time-dependent cost-benefit functions may be applied to different tasks or sequence of tasks



Business Scenarios of Active Integration

� Time Strategic Tactical OperationalReal-Time

� IndustryFinancial MarketBanking InsuranceSupply ChainManufacturingRetailing

� Customer Touch PointsCall CenterInternet (web access)Brick-and-MortarNone

� Business ProcessFraud DetectionCRMSCMOperations

� Degree of AutomationManualSemi-automaticAutonomic



Active Integration vs. …

� Active Integration vs. Active technologies:Event-driven technologies enhanced by real-time capabilities (and Integration capabilities) may be used as part of the architecture andsolutions of Active Integration

� Active Integration vs. Autonomic Computing:Real-time ConstraintsApplication-oriented vs. infrastructure. Includes collaboration (e.g. human interference)

� Active Integration vs. BAM:RT capabilitiesClose loopNot model based



Active Integration (RTE): More Scientific Aspects

Temporal capabilities:Time constrained operationsTemporal aspects of data

The "big spreadsheet" methapor (Gartner)Value dependenies among various data elements as invariants

Self-stabilization of integrity constraints (STP)

Core Database Management issues :Non intrusive detection of changesTransformation in hetrogenous environments

Software Engineering aspects (toolings..).



Basic Real-Time Concepts apply to RTE

• Real-time systems are those in which timelines is as important as the correctness of the outputs

• Real-time systems do not have to be “fast systems”

• Real-time systems design and analysis is a complex holistic engineering science

• Performance estimation and time reduction are crucial in real-time systems



Events apply to RTE

Definition: Any occurrence that causes the program counter to change non-sequentiality is considered a change of flow-of-control, and thus an event.

Synchronous events: Occur at predictable times in the flow-of-control

Asynchronous events: … unpredictable…



“Real-time system” definition applies to RTE

� A real-time system is any information processing system which has to respond to externally generated input stimuli within a finite and specified period

� The computer is a component in a larger (engineering) system => Embedded Computer System=> RT Enterprise

- the correctness depends not only on the logical result but also the time it was delivered

- failure to respond is as bad as the wrong response!



The Event-Driven World

� There is a growing need to monitor, capture, process, and store massive volumes of time-dependent events

Smart sensors, RFID, program trading, fraud management, compliance, intelligent oil field, location based service, logistics, presence (SIP), in-line analytics, etc.

� An increasing number of companies are addressing the needs of the time-dependent infrastructure market

Developing event engines that route, transform and derive events from multiple streamsDelivering content management solutions supporting large volumes of time-dependent dataExtracting event information from applications (ERP, CRM, etc.) and sensors

� Standards are emergingReal-time Specification for Java (JSR 1), Distributed Real-Time Specification (JSR 50)Web Services Notification and Web Services EventingOMG Data Distribution Service (DDS) and IEEE 1516 for battlefield simulation

� Middleware will evolve to deal with the throughput and time-dependent needs of the event-driven world. New programming models and tools will also emerge

Data integration will evolve to event integration

The accelerating need to handle large volumes of time-dependent events will give rise to new classes of middleware, programming models, and tools



The Event-Driven World

• Billing• Security• Fraud alerts• Retrospective

processing

Location Based Service (Traffic)

RFID-enabled Checkout

Intelligent Oil Field

ENRON Chairman

indicted……

..

Fraud/Compliance

Surveillance

RFID-enabled Scanning



Surveillance

Financial

Market Data

RFID Data

Wireless

VoIP Data

TDDB

DBMSDB

Data Warehouse

General Framework for Time-dependent Event Handling

Event ActionEvent Analysis

Event Notification

Event-based Engine

An event-based engine routes, sequences, and filters event data in a time-dependent fashion

TD: Time-dependent

Continuously Analyze Event

Streams

Identify Patterns of Interest

Selectively Notify Systems

& Personnel



Examples of Time-dependent EventsPage 6

Intelligent oil field – Generate early warnings to initiate oil well shutdowns by using signals and complex analysis from multiple sensors

RFID – Track location, count, and flow of parts in global supply chains

Real-time fraud detection – Monitor transaction authorization systems to identify fraudulent behavior and generate real-time alerts before the transaction is approved

Financial Services – Exploit time-dependent opportunities in currency trading and index arbitrage

SIP/Telco – Handle huge volumes of intelligent messaging, voice, and video over IP networks

Surveillance – Monitor people in airports for suspicious activity and generate timely alerts

Health – Detect early warnings of large-scale disease pandemics

Space – Observe corona mass ejections to obtain early warning of disruptions in global weather patterns

Building monitoring – Monitoring people and equipment in buildings and plants

Autonomic – Autonomic monitoring and management of large-scale, virtualized systems



Intelligence Application

Intelligence (applied knowledge)�

�

�

�

�

�

�

Knowledge (fact relationships)

Information (facts)

Data (streams)

Signal (sensors)

Daily Internet Traffic Volume

2002: 23 PB

2007: 647 PB (est.)

Email

1999: 610 Billion Emails (11 PB)

2002: 11 Trillion Emails

2006: 22 Trillion Emails (est.)

Telephony

2002: 187 Billion minutes

Emerging VoIP

Instant Messaging

2002: 41 Million users

2003: 275 Million usersE-mail, Voice, Image, Video, IMS, TV/Radio broadcast, Web traffic, etc.



Middleware for Financial Services

Incr

easi

ng C

apab

ility

Message at a time filter/route

Sequences, thresholds, groups

General multi-stream pattern specifications

Integration with processes, workflows

Inductive reasoning- Untrained patterns- Trained patterns

1’s Data Warehouse

Single server10’s

Data MiningOLAP

Event server clustersCEP event net assembly100’s

TransactionalOLTP

Managed ESB with event services

Tools for distributed deployment1000’s

Near real-time(< sec)

Collaborating domainsTools for integrating content behavior models10,000’s

Soft real-time(scheduled, ms)

Internet scale:100,000’s endpoints

100,000’s

Hard real-time(deterministic, us)

Scalability

Event Processing Language

Richness Ease of Use

Event Throughput

(events/sec/server)

Responsiveness



Message at a time filter/route

Sequences, thresholds, groups

General multi-stream pattern specifications

Integration with processes, workflows

Inductive reasoning- Untrained patterns- Trained patterns

1’s Data Warehouse

Single server10’s

Data MiningOLAP

Event server clustersCEP event net assembly100’s

TransactionalOLTP

Managed ESB with event services

Tools for distributed deployment1000’s

Near real-time(< sec)

Collaborating domainsTools for integrating content behavior models10,000’s

Soft real-time(scheduled, ms)

Internet scale:100,000’s endpoints

100,000’s

Hard real-time(deterministic, us)

Scalability

Event Processing Language

Richness Ease of Use

Event Throughput

(events/sec/server)

Responsiveness

Middleware for Financial Services

Incr

easi

ng C

apab

ility

Financial market information and program trading



“The fewer data needed, the better the information. And an overload of information, that is, anything much beyond what is truly needed, leads to information blackout. It does not enrich, but impoverishes”

Peter F. DruckerFounding father of the study of management

Action triggered not by a single event, but by a complex composition of events, happening at different times, and within different contexts

Complex Event Processing

Complex Event Processing is about getting better information, in real time.

Identification ofComposite Events

&Temporal

Awareness

Real Time Detectionof Problematic

SituationsActions

Probes & Sensors Complex Event Complex Event ProcessingProcessing



Complex Event Processing with IBM Active Middleware Technology™ – From Single Event to Situations

Website : http://www.haifa.il.ibm.com/dept/services/soms_ebs.html

Runtime Engine

Definitions

Detected

Situations

Event

Sources

Run Time

Build Time

Events Event Processing Rules / Patterns

Situation Detection

Authoring Tool

Actions



Examples of Complex Event Processing with IBM Active Middleware Technology™

� Regulatory constraintsreport on a bulk buy transaction that is performed less than one day after a bulk sell.

� Fraud detectionReport when two credit card purchases are performed within an hour at a distance greater than 300 km.

� Aggregation Report at a set time (e.g. the end of a business day) the number of purchase requests that were processed, their total dollar amount and the average dollar amount.

� CRMAlert if three orders from the same platinum customer were rejected for the same reason during a 24-hour periodAlert if an order was sent for processing and no response was received within the time specified by the SLA

� Intelligent Routing Route a trade message through SWIFT. If ACK is not received within 10 minutes and the trade $ value is higher than threshold try again. If ACK is not received after 3 attempts use telex.



IBM Active Middleware Technology™ model – the Situation Concept

Situation

Conditions

Lifespane5e5 e8e8

Initiator Terminator

Event SelectionEvent Selection ActionsActionsOperationOperation

Keys

e1e1

e2e2

e3e3

Input events

OperatorJoining

Counting

Temporal

Absence

Aggregation

NotificationsMessages

Definition updatesUser plug-ins



Transport Services� Assured Delivery� Secure Delivery� Transactional Delivery� Manageable Delivery� Qualities of Services

Event Services�Event Handling�Event Detection�Event Triggering�Complex Event Processing

Mediation Services�Routing �Transformation�Augmentation�Customization

The role of IBM Active Middleware Technology™ in the ESB



IBM Active Middleware Technology™ is enhancing BPM capabilities

IBM Active Middleware Technology™ for intelligent routing and business process initiation

IBM Active Middleware Technology™ for business process rules and calculations

IBM Active Middleware Technology™ for business level monitoring

ActiveMiddlewareTechnology





IBM Active Middleware Technology™ Value Proposition

�Ability to detect real time threats and opportunities�Have consistent behavior across the enterprise (no longer

Manual or Semi automatic event processing)�Add Business agility and quick reaction for changing

requirements/ regulations�Enabling flexible, state of the art enterprise architectures

(SOA, EDA)



Application Generation Framework for Real-Time Complex Event Processing

� The problemReal-time systems are usually monolithic applications, strongly coupled with the application domain and the system hardware.As a result, developing real-time systems is an expensive, time-consuming process.Any change in hardware or specifications may involve redesigning and rewriting the real-time application

� The idea:Develop event processing applications in the RT domainNew approach to develop RT systems, which, until now, were monolithic applications, strongly coupled with the application domain and the system hardwareExploit Event-processing techniques to the RT domain



Idea – cont…

� Improve writing of real-time event processing applications.� We provide a system that will enable users to define event

processing rules, generate an appropriate runtime system, and guarantee worst-case processing time for these rules

� Given a set of rules, we can predict the “worst-case” time requirements for processing of incoming events, and guarantee an upper bound on processing time

CEP is an emerging field, and has not yet been applied to the real-time domain



Justification

� Business JustificationThis will be used by providers of real-time systems dealing with event processingIt is part of the Real-Time SOA effortEstablish RT CEP methodology

� AlternativesWorking around our invention can be done by hard-coding the real-time event-processing system (and that’s not a recommended technique)



Back-up pages: The detailed solution



Summary

� “Real-time” has a broader perspective – RTE, BPM, Active Integration

� The “Event-driven world” has a broader perspective � Event Processing techniques / Active Integration may

bring a very important added value to RTE� Real-time Complex Event Processing is essential to RTE,

both for RT for Operational Systems and RT for BPM

IBM Haifa Research Lab

Documents

Transcript of IBM Haifa Research Lab