Active Technologies - HRL | 2003 | Presentation subtitle: 20pt Arial Regular, teal R045 | G182 |...

Active Technologies - HRL

| 2003 | © 2002 IBM Corporation

Complex event processingSeminar : HKUST – September 2004

Opher EtzionIBM Research Laboratory in Haifa

[email protected]

© 2002 IBM Corporation

YES!! It works every time.The Gartner quote always gets their attention.

As stated in the Gartner report, apples are ….

Wow, Gartner! Perhaps this wouldn’t be a total waste of time after all


Active Behavior – scenario 1

Your Refrigerator called me..


Active Behavior - Scenario 2

At least N people of my team are in this building now


Active technologies – scenario 3

An information about a certain target arrived from two different sources within the last hour AND a contradictory information did not arrive within the last hour, and there have been at least five active sources in the same area within the last hour then handle the target.


Active behavior : Scenario 4

Whenever a traffic congestion occurs, re-evaluate the traffic-lights timing policies and change it.


What is the common denominator ?

All of these are event-driven They are not driven by a single event, so some

processing of the events is needed.We can have some hints about the types of

processing needed in each case…


What is the talk about ?

Events --- what are they ? How do they relate to the rest of

the universe ? What are the types of event

processes ? AMIT – a CEP example Other examples Some use cases Some research and pragmatic

challenges


What is an event ?

An event is a significant (in some context) instantaneous (happens in a specific point in time) atomic (happens completely or not at all) occurrence

Are those events ?


Event collection middleware

A platform to communicate events Event sources: application, workflow, database, IT resource –

needs instrumentation and normalization. Event transfer: publish/subscribe (with content).

Many technical issues: Scalability

Standard protocols

Event store

Event bus


What is the connection between events and data ?

Database does not have an inherent semantics, it can represent everything (including events)

But – semantically there is a distinction between:State: snapshot of the state of (the appropriate subset of) the universe at a certain point in time.

Transition: transfer from state to state.Database processing (e.g. SQL) is state

processing.So – does event represent transitions ?

Sometimes.

o A transition is an event.

oNot all events change states that are of interest.


Is event processing different from data processing ?

In database processing – a snapshot that includes multiple entities is processed. All the processing is a function of values that exists within this snapshot.

In event processing --- the history of events is processed, and can also relate to the state information. Temporal processing is dominant.

Event processing subsumes data processing (but usually not all the power of SQL is needed).

Event processing can be expressed in SQL (the data-stream people are working on it) but in many cases it is not easy (exercise: write in SQL query that finds if a sequence of at least 8 events of 8 different event types has occurred in a sequence within an hour anytime).


Complex event processing – a model based approach

context

data

event

state

entity

activity

flow

situation


What is an event model ?

Data model does not talk only about data, but also about things in the world and dependencies among them (e.g ER, semantic data model).

Event model is an extended data model in which events play role as a first class citizen.

Events are connected to : Other events

Context (what’s that ???)

Messages

Entities

Databases or other state keepers

Flows


Event processing - Situation

Situation is defined as a transition in the universe that requires reaction (either “reactive” situation or “proactive” situation).

One of the main event processing goals is situation detection An approximation for situation is an inferred causality event within a

context, the function can optionally contain other players (data, state etc..) S = F (e1,…en, context, [state information]).

This approximation equals the situation when the function is deterministic; in other cases we shall need to operate uncertainty measures.

Other types of event processing relate to relationships of events with everything else.


Situation detection


Some examples from various domains

Alert if the IBM stock has gone up in 3 percent within two hours, and the Dow Jones did not go up in more than 1 percent at the same period.

Alert if three memory problems occurred during the last hour. Alert if the same request was reassigned to three agents, and no answer

was given to the requester.


Relationships among events

Causality: Observed causality : two events that has statistical correlation that

indicates that one of them is an antecedent of the other (example: getting out of the car, locking the car).

Inferred causality: An event that is signaled as a result of processing in which the other event participated in (example: traffic jam identified, traffic-lights policy re-calculated).

Subset hierarchy : possibly conditional generalization/specialization relations (printer

problem, hardware problem) Cross-section :

Events with certain conditions are considered as other virtual events (e.g. all events that relate to the same object).


Context

Event is instantaneous – occurs in a certain period of time.

Sometimes event processing is done within a context Examples:

Within the working hours

The last 30 minutes of trade

within 60 minutes from the time that the request was sent

From transaction start to transaction end

Context has : Validity interval.

possibly multiple instances by some partition criterion (e.g. by customer).

possibly spatial characteristics. Events are related to contexts:

Start and end of contexts are events.

Event processing can be a function of context.


Context Awareness

Composite perspectives of the environment Semantic perspective designates environment information about a specific

object or entity (e.g. users that are members of the same group).

Temporal perspective designates environment information within a specific temporal element (e.g. network overload in one hour).

Spatial perspective designates environment information within a specific location or area (e.g. vehicles near a traffic problem).

State perspective designates environment information within a specific state (e.g. low market volume).


Temporal Context

free

busydysfunctional

busy free

free

free

freefree

busy

Designate a collection of events that occur within a temporal interval

Bounded by initiator and terminator

Has maximal length, initiation and termination policies

Multiple lifespan may exists simultaneously


Spatial context

Express the spatial perspective of Amit context

Designate a collection of events that are originated from the same region

Either a circle specified by a center coordinate and a radius or a polygon specified by a set of coordinates.

Fixed or moving

dysfunctionaldysfunctional


Events and Messages

There is common misconception that events and messages are the same.

In fact there are some connections: A message may represent a description of event occurrence – this is

one of the common ways to report events.

There are events associated with messages (created, sent, received, acknowledged…) like any other entity.

Complex event processing is part of message brokering/mediation.


Events and entities

An event can refer to one or more entities,Example: John sell the bike to Jim (type of event sale, entities:

John, bike, Jim with different roles).An entity has a role in an event (seller, buyer, merchandise)An event may have a role in the entity (starting, ending, disrupting –

can be interpreted as change the entity status)


Events and databases – again

An event may (but does not have to) change state (or results in a database update)

A database operation occurrence is an event Event processing can look at both events and states.


Events and activities (and flows…)

Event/situation can trigger activitiesActivity can signal events

A flow can contain activities, event, data and messages with all interactions.

In case of a flow --- an event/situation can add/modify/delete sub-flows dynamically and can interrupt running flows

Transition in flow states are events.


Short history of event processing ..

In early days of programming languages it was used for interrupts/exception handling

Real-time applications raised requirements for reaction based on time.

Active databases started in the late 1980-ies System and network management tools emerged in the early

1990-ies Publish/Subscribe system appeared in the lat 1990-ies Currently: monitoring, management application, business

process integration, straight-through-processing… Gartner’s : CEP, BAM, RTE.


Active behavior specification:

Embedded in regular programming languages. Rule-based approaches (reactive/proactive programming):

Condition-Action rules (event is hidden)

Event-Condition-Action rules

Situation-[Condition]- Action rules Model based approaches (reflective programming)

Semantic net approach.

Self-stabilization approach (data-driven).

The “action” can be a “business rule”


Event-Condition-Action rules

Making the processing event driven The paradigm :

when event occurred if a condition is satisfied perform action

A variation can be E(CA)*

Example: When a message about Microsoft stock quote has arrived, If the value is less than 25, then notify all customers in the subscription-list.


Situation based processing

Next step in the evolution – from event based to situation based. The concept of situation is what triggers the action from the

user’s point of view (may not be a single event) This is an abstraction over the universe of transitions in the

same way that SQL query or view is an abstraction over the universe of states

It is roughly equivalent to the term “composite event within a context”, but may have uncertainty associated with it.


Situation examples

A client has withdrawn at least 3 times a sum of more than $100,000 that he deposited at most 2 days before the withdrawal [looking for money exceptional movement]

A client has withdrawn within 2 hours from two ATM machines that are more then 200 KM apart [fraud detection]

A client wishes to be notified when IBM stock is up more than 3 percent if he is in the office [personalized location-aware publish/subscribe]

At the end of the day at least 2% of the orders have not completed [monitoring]


Relationship to Publish/Subscribe Technology

Publish/subscribe is : event - action. Publish/subscribe with filtering is: event - condition – action

(current state of the art). Situation based publish/subscribe is the next generation – enable

“personalization of push technology”.


Relationship to event correlation

The term “event correlation” has originated from network management.

The idea is to find event that have statistical correlation among them (and hence the name) and possibly filter out some of them to cope with “event storm”

It has some notion of causality (usually – two events occurring together within a fixed time interval).

Used in system management applications, and as an event filtering tool.


AmitActive Middleware Technology


Amit – Technologies Context diagram

Amit Metadata

(Definition Manager)

ADI Model

Templates

Events

Rules

GUI

Model-based

Rule-based

Rule-based

Situation

ManagerARAD

External (pub/sub)

Amit

Tooling

Action

Manager

Exception

Handling

Events

Instances

Situations

Conclusions

Alerts

Exceptions

Definitions

Exception Handling Metadata

Exceptions

Exceptions

DetectionsCreate new

Events

Change Definitions

tools


The situation concept

e1.id

e2.name

e3.key

KeyConditions )where...(Attributes )retain, override.,..(

Detection Mode )immediate, delayed, deferred(

Repeat Mode )always, once(

Contexte5

e8

Initiator Terminator

Situation

e1

e2

e3

Events Operators

JOINING

COUNTING

TEMPORAL

ABSENCE

)all, sequence(

)atleast, atmost, nth(

)every, after, at(

)not, unless(

3


Amit Tooling Today


Selected additional complex event processing solutions

Apama Elity Actimize Ispheres …

Active Technologies - HRL

| 2003 | © 2002 IBM Corporation

Some applications

of Complex Event Processing


Transaction Flow Monitoring

Transaction Level Alerts: Acknowledgements is not

consistent with the sending order. 15 minutes before a settlement has

to close, a message is waiting to be handled by a compliance officer due to insufficient credit.

No acknowledgement/reject received from the stock exchange within one hour from sending a message.

Business Level Alerts: Abnormal number of payments

received from a specific bank (account) within the first two hours or business

Three rejects received within a single working day, for FED messages of the same platinum client.

Order Management System Execution Management

Open Order

23

SM CP

validate the security

validate the Party & account

4

Exchange

Transaction ManagementCalculationsFigurationAllocationSetlement Details*

AM = Account MasterCP = CounterpartySM = Security MasterFC = Fees and ChargesFR= Fees and charges rulesSI = Setlement Instructions

AM

Reference DataOperation &Transaction Data

Order1

Investment Manger

Open Order

updates open orders with fills

Request for ExecutionNotification of Completion

Block trade

Message/Transaction/Flow

5

6

7

8Allocations, Net proceeds, affirms , confirms

Confirmation and Allocation

9Fully figure trade

Purchase & SalesClearing

SM3 CP3AM3

Settlement

SI2

10 Settlement Instruction and Payment

CP2 FC FRAM2SM2 SI1*

Front-Office

Middle-Office

Back-Office


Finance Scenarios

Trade Regulation BreachCEPS verifies that sells and subsequent purchases of large quantities of stock meet regulatory requirements

Credit BreachCEPS initially allows a credit limit to be exceeded (by no more than 10%). The second time the limit is exceeded, CEPS routes the order to a credit officer for approval

Trade Execution DelayNotify if a purchase order was sent for processing and no response was received within the time specified by the SLA.

Fraud DetectionReport when multiple credit card purchases are performed within an hour or (any given time frame) at a distance greater than 300 km (or any given distance).

Finance


Server

CEP SERVER

Action Manager

Wireless server

Data Warehouse

CRM

On Demand eCRM Architecture and Dataflow Example:

Hand-Held

Database stores all historical customer information

A CEP server receives relevant customer data from database when customer enters the store

Each event is processed when customer is in the store

Action managers performs actions – recommendation of complementary products, sales on frequently purchased items etc..

Retail


Homeland security

CEP

Unusual activity

Security Scenario

Security


Insurance Scenario

CEP helps improve claim processing by automating key activities such as:

Identifying invalid claims

Detection of potential fraudulent claims

We can help release delayed payments according to pre-specified conditions.

Identifying problematic health insurance providers that do not comply with regulations.

Insurance

Automatic Exception Resolution

ECS Automatic Exception Resolution

ECS

WBI Message Broker

StartQueue

XML

HTML claim form

EDSroutingnode

Stage 1

(log)

Stage 1Queue

Suspendqueue 1

Effectqueue 1

Stage n

(log)

Finalqueue

ECSroutingnode

Suspendqueue 1

Effectqueue 1

Claimdatabase


Higher level technologies that use CEP

The “sense and respond” loop Real-time analytics Autonomic computing


On-demand control loop (sense and respond) :

06nAFS -


Real-time analytics

Departure from traditional use of analytic tools that may not be time constrained.

The traffic light problem is an exampleOther examples:

Re-calculation of network configuration policies when part of the network is disabled (e.g. due to “denial of service attack”.

Re-establishing of queue priorities policies Trade-off between time and quality of solution (cannot get to the

optimal solution in 1 minutes, how should I get to the best possible solution given these time constraints)..


Autonomic computingAlerts, events and problem analysis request interface

Interface to real and virtualized resources and components that regulate control.

SLA/Policy interface, interprets and translates into "control logic"

Plan

Policy Transforms

Plan Generators

Policy InterpreterAnalyze

Execute

Service Dispatcher

Distribution Engine

Scheduler Engine

Workflow Engine

Monitor

Metric Managers

Filters

Simple Correlators

Knowledge

Policy

CalendarTopology

Recent Activity Log

Sensors Effectors

Rules Engines

Analysis Engines

Policy Validations

Policy Resolution


Additional research topics

Real-time aspects Distribution and parallelism Transactional support Temporal issues Uncertainty in complex event processing. Software engineering aspects.


Real-time aspects:

The end-to-end process involving the CEP may have real-time constraints.

This may inflict real-time Real-time awareness built-in operations: Scheduling

Prioritization

Relevance of “late” events.


Distribution and Parallelism

For scalability reasons: The ability to apply N engines and balance the processing load

Note, that there can be many nested situations, thus full partition may not be possible

Establishing minimal traffic among the various engines.

Reference: M. Shmueli, O. Etzion - Parallel Implementation of Composite Events. ICDCS Workshops 2002: 579-580

For high availability: Requires support in clustering and failover.

Usually done by basing on middleware services


Transactional support

May require non-ACID transactional support : Total rollback may not be possible, since we cannot say that the “event

did not happen”, even if we fail to process its consequences.

Events that are part of the process may belong to different transactions, only the last of them “closes the loop”.

However, some of the process can be “atomic”.


Temporal issues

Events may not arrive in the same order they are produced The time-stamps on different events may not be consistent with

relative timings. There may be a communication delay to report events

How long should we wait for an event ?

What happens we get event beyond this time-out ?In general – how do we process:

Retroactive events (events about the past)

Predictive events (“certain” events about the future)


Uncertainty aspects :

There are several sources for uncertainty: Uncertainty that the event has (or has not) occurred

Uncertainty in the details of the events itself

Uncertainty that the SITUATION is equivalent to this specific function of events and contexts

Uncertainty that the right context is identified,

…


Software Engineering aspects:

A programming paradigm that deserves: Methodologies - as a programming tool, and as part of a bigger picture

Modeling tools, automatic creations of rules

Debugging --- all the known difficulties of rule debugging --- interactions among situations, halting problem, determinism and sequencing.


Future --- stand-alone vs. embedded technology

Complex event processing is being developed in two main contexts: As a central service collecting events from various sources (will be

part of all middleware products)..

As an embedded technology inside other frameworks/products/solutions (seem to grow more rapidly)

According to Gartner’s hype-cycle, time to full maturity is 5-7 years.


References D. Luckham – The power of events: An Introduction to Complex Event Processing. Addison-Wesley, 2002 A. Adi, O. Etzion – Amit, the situation manager. VLDB Journal, 13(2), 173-203, 2004. Gartner reports:

“Events will transform application servers” “Hype cycle for application integration middleware and platforms 2003” …

Vendors URLs : http://www.apama.com http://www.elity.com http://www.actimize.com http://www.ispheres.com

Foundations : Chakravarthy-S, and Mishra-D. "Snoop: an expressive event specification language for active

databases." Data and Knowledge Engineering 14.1 (1994): 1-26. Tombros-D, Geppert-A, and Dittrich-KR. "Semantics of Reactive Components in Event-Driven

Workflow Execution". CAiSE 1997: 409-422. Yemini-SA, Kliger-S, Mozes-E, Yemini-Y, and Ohsie-D. "High speed and robust event correlation."

IEEE Communications Magazine. 34.5 (1996): 82-90. Zimmer-D, and Unland-R. "A General Model for Specification of the Semantics of Complex Events in

Active Database Management Systems." C-LAB Report. 1998. Zimmermann-J, and Buchmann-A. "REACH." Active Rules in Database Systems. Springer Verlag,

1999. 263-277.

.

Active Technologies - HRL | 2003 | Presentation subtitle: 20pt Arial Regular, teal R045 | G182 |...

Documents

Transcript of Active Technologies - HRL | 2003 | Presentation subtitle: 20pt Arial Regular, teal R045 | G182 |...