Post on 01-Nov-2014
description
epts event processing technical society
eptsevent processing technical society
Adrian Paschke
(Freie Universitaet Berlin)
Paul Vincent
(TIBCO Software)
Catherine Moxey
(IBM)
Alex Alves
(Oracle)
Event Processing
Reference Architecture
and Design Patterns
Tutorial v1.2
Content by: members of the
EPTS Reference Architecture Working Group
epts event processing technical society
Tutorial
• Event Processing is an increasingly “mainstream”
view in IT, if not widely in production
• Event Processing Functional Reference Architecture
(FRA) describes the functions involved in event
processing
• Functions in the FRA can themselves be described as
event processing patterns
• Most technology providers can support the FRA and
its associated design patterns
2
epts event processing technical society
3
Agenda
• Introduction to the EPTS Functional Reference
Architecture, and what it contains
• Mapping Functions to Patterns: some sample
Event Processing Patterns
• EPTS member pattern implementation examples
• Summary and future work on EP Patterns
epts event processing technical society
Introduction to Event Processing
• Event-centric view of IT
• Events are
– Sent and Received
– Aggregated, Transformed into Data, Deleted
– Processed in queries, rules etc
– Cause actions like processes, service invocations, etc
4
epts event processing technical society
Event Processing, Complex Event Processing, Event Stream Processing
Event Abstraction, Event Pattern Detection, Event Composition
etc. etc.
Introduction to Event Processing
Event
Managem
ent
Event Consumer
(Event sink, event handler, event
listener)
Run time Administration
Event
Type D
efinitio
ns,
Event
Pro
cessin
g R
ule
s,
etc
.
Design time
Events Derived
Events
Event Producer
(Event Source, Event Emitter)
epts event processing technical society
Motivation and Benefits
• Motivation
– Event Processing is evolving and exploiting many technologies
– Potential adopters (stakeholders) need a reference to understand suppliers‟ architectures and solutions.
• Benefits
– a Reference Architecture predefines customizable abstract frames of reference for specific stakeholder concerns and application domains.
• Aids reuse of successful EP architectures for frequently occurring EP design problems
– Underlying Architecture is a Reference Model that defines the terminology and components in Event Processing architectures
epts event processing technical society
10
Reference Architecture Viewpoints
Viewpoint
Element
Viewpoint
Engineering EP
Architecture
Managing EP
Architecture
Business with EP
Architecture
Concepts How to
implement?
How to apply? How to utilize / sell /
own?
Stakeholders Architects /
Engineers
Project Manager Decision Maker,
Customer, Provider
Concerns Effective
construction and
deployment
Operational
Management
Strategic and
tactical
management
Techniques /
Languages
Modeling,
Engineering
IT (service/appl)
management,
project
management
Monitoring,
Enterprise Decision
Management,
Governance
epts event processing technical society
11
Stakeholders and Viewpoints covered
Viewpoints (for CEP / stateful EP)
Stakeholder Component Functional
Decision Maker
/ End User
[Not applicable] Inputs, Outputs and
Processing Requirements
View
Architect /
Designer
Solution Components
View
Functions carried out in
CEP View
TBD (2012?) Focus so far
epts event processing technical society
Functional Reference Architecture
• Architect and Developer perspective
– includes the 3 main functions
(development, run-time and administration),
– targets primarily the automated event processing
operations
• Run-time functions in 2 main groups:
– the event infrastructure (sources and consumers)
external to the event processor under consideration,
– the event processor.
epts event processing technical society
Event Processing, Complex Event Processing, Event
Stream Processing
Event Abstraction, Event Pattern Detection, Event Composition
etc. etc.
Functional View source: definitions of EP
Event
Managem
ent
Event Consumer
(Event sink, event handler, event
listener)
Run time Administration
Event
Type D
efinitio
ns,
Event
Pro
cessin
g R
ule
s,
etc
.
Design time
Events Derived
Events
Event Producer
(Event Source, Event Emitter)
epts event processing technical society
Reference Architecture: Functional View
Event Production Publication, Retrieval
Even
t Process M
on
itorin
g, C
on
trol
Event PreparationIdentification, Selection, Filtering,
Monitoring, Enrichment
Complex Event DetectionConsolidation, Composition,
Aggregation
Event ReactionAssessment, Routing, Prediction,
Discovery, Learning
Event Consumption Dashboard, Apps,External Reaction
Run time Administration
Even
t a
nd
Co
mp
lex E
ven
t(P
att
ern
, Contr
ol, R
ule
, Q
uery
, RegEx.e
tc)
Defi
nit
ion
, M
od
elin
g, (con
tin
uou
s) I
mp
rovem
en
t
Design time
Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification
0..*
0..*
0..*
0..*
Sta
te M
an
ag
em
en
t
epts event processing technical society
Reference Architecture: Functional View / Runtime
Event Production Publication, Retrieval
Event PreparationIdentification, Selection, Filtering,
Monitoring, Enrichment
Complex Event DetectionConsolidation, Composition,
Aggregation
Event ReactionAssessment, Routing, Prediction,
Discovery, Learning
Event Consumption Dashboard, Apps,External Reaction
Run time
Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification
0..*
0..*
0..*
0..*
Sta
te M
an
ag
em
en
t
epts event processing technical society
Reference Architecture: Functional View / Runtime
Event Production Publication, Retrieval
Event Consumption Dashboard, Apps,External Reaction
Event Production: the source of events for event processing.
• Event Publication: As a part of event production, events may be published
onto a communication mechanism (eg event bus) for use by event
consumers (including participants in event processing). This is analogous to
a "push" system for obtaining events.
• Event Retrieval: As a part of event production, events may be explicitly
retrieved from some detection system. This is analogous to a "pull" system
for obtaining events.
epts event processing technical society
Reference Architecture: Functional View / Runtime
Event Production Publication, Retrieval
Event Consumption Dashboard, Apps,External Reaction
Event Consumption: the process of using events from event publication and
processing. Event processing itself can be an event consumer, although for
the purposes of the reference architecture, event consumers are meant to
indicate downstream consumers of events generated in event processing.
• Dashboard: a type of event consumer that displays events as they occur to
some user community.
• Applications: a type of event consumer if it consumes events for its own
processes.
• External Reaction: caused through some event consumption, as the result
of some hardware or software process.
epts event processing technical society
Reference Architecture: Functional View / Runtime
Event PreparationIdentification, Selection, Filtering,
Monitoring, Enrichment
Event Preparation: the process of preparing the event and associated payload and
metadata for further stages of event processing.
• Entity Identification: incoming events will need to be identified relative to prior events, such
as associating events with particular sources or sensors.
• Event Selection: particular events may be selected for further analysis. Different parts of
event processing may require different selections of events. See also event filtering.
• Event Filtering: a stream or list of events may be filtered on some payload or metadata
information such that some subset is selected for further processing.
• Event Monitoring: particular types of events may be monitored for selection for further
processing. This may utilise specific mechanisms external to the event processing such as
exploiting event production features.
• Event Enrichment: events may be "enriched" through knowledge gained through previous
events or data.
epts event processing technical society
Reference Architecture: Functional View / Runtime
Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification
Event Analysis: the process of analysing suitably prepared events and their
payloads and metadata for useful information.
• Event Analytics: the use of statistical methods to derive additional information about an
event or set of events.
• Event Transforms: processes carried out on event payloads or data, either related to
event preparation, analysis or processing.
• Event Tracking: where events related to some entity are used to identify state changes in
that entity.
• Event Scoring: the process by which events are ranked using a score, usually as a part of
a statistical analysis of a set of events. See also Event Analytics
• Event Rating: where events are compared to others to associate some importance or
other, possibly relative, measurement to the event.
• Event Classification: where events are associated with some classification scheme for
use in downstream processing.
epts event processing technical society
Reference Architecture: Functional View / Runtime
Complex Event DetectionConsolidation, Composition,
Aggregation
Complex Event Detection: the process by which event analysis results in the
creation of new event information, or the update of existing complex events.
• Event Consolidation: combining disparate events together into a "main" or "primary"
event. See also event aggregation.
• Event Composition: composing new, complex events from existing, possibly source,
events.
• Event Aggregation: combining events to provide new or useful information, such as trend
information and event statistics. Similar to event consolidation.
epts event processing technical society
Reference Architecture: Functional View / Runtime
Event ReactionAssessment, Routing, Prediction,
Discovery, Learning
Event Reaction: the process subsequent to event analysis and complex event
detection to handle the results of analysis and detection.
• Event Assessment: the process by which an event is assessed for inclusion in some
process, incorporation in some other event, etc.
• Event Routing: the process by which an event is redirected to some process, computation
element, or other event sink.
• Event Prediction: where the reaction to some event processing is that some new event is
predicted to occur.
• Event Discovery: where the reaction to some event processing is the disclosure of a new,
typically complex, event type.
• Note that event prediction is predicting some future event, usually of a known type, whereas event
discovery is the uncovering of a new event type. See also event-based learning.
• Event-based Learning: the reaction to some event processing that uses new event
information to add to some, typically statistical-based, understanding of events.
• Note that event-based learning is a specialisation of
general machine learning and predictive analytics.
epts event processing technical society
Reference Architecture: Functional View / Design time
Even
t a
nd
Co
mp
lex E
ven
t(P
att
ern
, Contr
ol, R
ule
, Q
uery
, RegEx.e
tc)
Defi
nit
ion
, M
od
elin
g, (con
tin
uou
s) I
mp
rovem
en
t
Design time
Covers the definition, modeling, improvement /
maintenance of the artifacts
used in event processing:
• event definitions, including event metadata and payloads,
• event and event object organisations and structures,
• event processing transformations / queries / rules /
procedures / flows / states / decisions /
expressions (although these can sometimes be
considered as administrative updates in some situations)
epts event processing technical society
Reference Architecture: Functional View / AdministrationEven
t Process M
on
itorin
g, C
on
trol
Administration
Administrative concepts of
monitoring and control. This may involve
• starting and stopping the application and event
processing elements, including application monitors
• providing and updating security levels to event inputs and
outputs (also can design-time)
• management of high availability and reliability resources,
such as hot standby processes
• resource utilisation monitoring of the event processing
components
• process updates, such as how-swapping of event
processing definitions to newer versions.
epts event processing technical society
24
Agenda
• Introduction to the EPTS Functional Reference
Architecture, and what it contains
• Mapping Functions to Patterns: some sample
Event Processing Patterns
• EPTS member pattern implementation examples
• Summary and future work on EP Patterns
epts event processing technical society
Event Processing Patterns
• Functions from Reference Architecture are a
guide to possible low-level patterns
25
Event PreparationIdentification, Selection, Filtering,
Monitoring, Enrichment
Complex Event DetectionConsolidation, Composition,
Aggregation
Event ReactionAssessment, Routing, Prediction,
Discovery, Learning
Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
Which Classification of Patterns to Use?
• Common EP Functions?
As described in the Reference Architecture
• Other sources?
Recent literature
26
epts event processing technical society
Comparison of EPTS RA Fns and EPIA EP Agents
27
Event PreparationIdentification, Selection, Filtering,
Monitoring, Enrichment
Complex Event DetectionConsolidation, Composition,
Aggregation
Event ReactionAssessment, Routing, Prediction,
Discovery, Learning
Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification
EPA
Filter Transformation
Translate
Enrich Project
Aggregate Split Compose
Pattern detect
epts event processing technical society
Comparison of EPTS RA Fns and EPIA EP AgentsEvent Preparation
28
epts event processing technical society
Comparison of EPTS RA Fns and EPIA EP AgentsEvent Analysis
29
epts event processing technical society
Comparison of EPTS RA Fns and EPIA EP AgentsComplex Event Detection
30
epts event processing technical society
Comparison of EPTS RA Fns and EPIA EP AgentsEvent Reaction
31
epts event processing technical society
What do we want to cover in an EP Pattern?
32
Event Filtering: a stream or list of events may be filtered
on some payload or metadata information
such that some subset is selected for further processing.
Type /
synonym
Operation /
Operators
Operands
Rationale /
business ruleMapping to
EPL template
epts event processing technical society
EP MetaPattern version 1
• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations
33
epts event processing technical society
EP MetaPattern future versions?
• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations
34
may use EPL classification
per EPTS Languages WG
may use revised EP Fns
classification / ontology
epts event processing technical society
EP Pattern Implementations
• Differ in Implementation type (Event Processing
Language, Execution semantics, etc)
• Sampled systems we cover:
1.TIBCO BusinessEvents Rete-production-rule-engine
(with continuous query and event-expression options)
2.Oracle CEP Stream-processing engine
3.Prova Logic Programming Semantic CEP Rule Engine
4.IBM WebSphere Decision Server event-driven
production-rule-engine
& IBM Infosphere Streams stream processing engine
35
epts event processing technical society
36
Agenda
• Introduction to the EPTS Functional Reference
Architecture, and what it contains
• Mapping Functions to Patterns: some sample
Event Processing Patterns
• EPTS member pattern implementation examples
• Summary and future work on EP Patterns
epts event processing technical society
Some EP Patterns to review in more detail
1. Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
2. Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
3. Aggregation: add prior events together for new information
– E.g. customer purchase history: track the sum total of order
values for a customer
4. Routing: based on the purchase type pass the event on to the
appropriate service
– E.g. customer purchase event: pass on to a provisioning service
based on the type of product / product classification 37
epts event processing technical society
Some EP Patterns to review in more detail
1. Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
2. Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
3. Aggregation: add some information based on prior events
– E.g. customer purchase history: track the sum total of order
values for a customer
4. Routing: based on the purchase type pass the event on to
the appropriate service
– E.g. customer purchase event: pass on to a provisioning
service based on the type of product / product classification 38
epts event processing technical society
Filter Pattern: Classification (1)
• EPTS Reference Architecture:
– "During event preparation, a stream or list of events may
be filtered on some payload or metadata information
such that some subset is selected for further processing."
• EPIA:
– Filter (EPA or Event Processing Agent) performs filtering
only and has no matching or derivation steps, so it does
not transform the input event.
39
epts event processing technical society
Filter Pattern: Description (2)
• Role / Explanation
– Comparing some property of the event (an attribute or
metadata) with some other value (from some other event, or
data)
• Associated Business Rule Specification
– As a constraint: a selection (to be enforced during the
processing of events):
All <event entities>
that have
<filter expression>
must have
<some status>. 40
eventFilter
f(event)event
discarded
event
epts event processing technical society
Filter Pattern: Structure (3)
• EPTS Glossary comparison
– A filter pattern can relate to several terms in the EPTS
Glossary
• can be specified in an event pattern (A template containing
event templates, relational operators and variables...)
• .. by an event pattern constraint (A Boolean condition that
must be satisfied by the events observed in a system...)
• .. as part of an event processing rule (A prescribed method
for processing events.)
• .. or as part of event stream processing (Computing on inputs
that are event streams.)
–
41
epts event processing technical society
Filter Pattern: Implementations (4)
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
42
eventFilter
f(event)event
discarded
event
epts event processing technical society
Filter Pattern: Implementation 1aTIBCO BusinessEvents: rules
• General pattern: <declare> event
<if> event-boolean-expression
<then> update-event-with-fact
OR
action-for-filtered-event
• Effect: ignore filtered events
on operations on non-filtered events
43
event
Filter
f(event)
event
Operation on
event
epts event processing technical society
Filter Pattern: Implementation 1bTIBCO BusinessEvents: event removal rules
• Implementation type: Rete-based rule-driven system
• Alternate pattern: <declare> event
<if>
NOT event-boolean-expression
<then>
consume-event
• Effect: removed filtered events from
system (and ALL subsequent operations!)
44
event
NOT Filter
!f(event)
event
epts event processing technical society
Filter Pattern: Implementation 1c TIBCO BusinessEvents: State transitions
• General pattern: <state models for concept C>
<route from State n to n+1>
<on> event
<where> event-filter
• Effect: change state for some concept
• Notes: this is a specialisation of the
general rule pattern!
45
event
Filter
query(event)
State
transition
epts event processing technical society
Filter Pattern: Implementation 1d TIBCO BusinessEvents: Queries
• General pattern: <select> output
<from> event
<where> event-filter
<policy> notification
• Effect: collect filtered events for
event operations per some policy
• Note: the pattern matching framework
has a similar notion
but a different surface syntax46
event
Filter
query(event)
Event
subset
Operations on
event subset
on policy event
Policy
event
epts event processing technical society
Filter Pattern: Implementation 2aOracle CEP: queries
• General pattern:
SELECT <property>
FROM <STREAM>
WHERE <predicate-condition>
• Effect: discard events whose predicate
evaluate to false
47
event
predicate
event
Operation on
event
epts event processing technical society
Filter Pattern: Implementation 2bOracle CEP: model
48
• Scenario:
– Select only stocks from the stock tick stream whose
symbols are „AAA‟.
• Application Model
epts event processing technical society
Filter Pattern: Implementation 2cOracle CEP: query
49
Specify STREAM source
Define predicate for
filtering
SELECT *
FROM StockTickStream
WHERE symbol = „AAA‟
epts event processing technical society
Filter Pattern: Implementation 3aProva: rules
• Implementation type: backward reasoning logical filter
• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :-
<logical filter condition(s)>,
... .
<logical derivation rule 1> :-
<filter conditions>.
<logical derivation rule 2> :-
<filter conditions>.
...
• Effect: Unify received events with
event pattern definition and apply logical
filter rules (derivation rules with rule chaining)50
event
Event pattern +
logical rule filter
event
Operation on
event
event
Backward
Resoning
Rule
chaning
…
epts event processing technical society
Filter Pattern: Implementation 3bProva: Messaging Reaction Rules in Prova
• Send a message
sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
Description:
– XID is the conversation identifier
– Protocol: protocol e.g. self, jms, esb etc.
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA Agent Communication
– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload
epts event processing technical society
Filter Pattern: Implementation 3cProva: Example
% Filter for stocks starting with „A“ and price > 100
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
S = "A.*",
P > 100,
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
% Filter for stocks starting with „A“ and price > 100
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
filter(S,P),
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
filter(Symbol,Price) :-
Price > 100,
Symbol = "A.*".
Example with rule chaining
epts event processing technical society
Filter Pattern: Implementation 3dProva: Semantic Event Processing Example
Event Stream{(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }
{(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}
Semantic Query Filer:Stocks of companies, which have production facilities in Europe andproduce products out of metal andHave more than 10,000 employees.
{(OPEL, is_a, car_manufacturer),(car_manufacturer, build, Cars),
(Cars, are_build_from, Metall),(OPEL, hat_production_facilities_in, Germany),
(Germany, is_in, Europe)(OPEL, is_a, Major_corporation),
(Major_corporation, have, over_10,000_employees)}
Semantic Knowledge Base
rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^car:Major_corporation,P^^currency:Dollar,
T^^time:Timepoint)) :- … <semantic filter inference> .
epts event processing technical society
Filter Pattern: Implementation 4aIBM WebSphere Decision Server: Business Events component
55
event
Filter
event
Operation on
event
• Example:
Define a filter to only process events with a
purchase value greater than £100
• Effect: Only process events
which match the filter
event
epts event processing technical society
• Example:
composite Main {
graph
stream<rstring name, uint32 age> Beat = Beacon() {}
stream<rstring name, uint32 age> Youngs = Filter(Beat)
{
param filter : age < 30u;
}
(stream<Beat> Younger; stream<Beat> Older) = Filter(Beat)
{
param filter : age < 30u;
}
}
Filter Pattern: Implementation 4bIBM InfoSphere Streams: Streams Processing Language
56
• Effect: filter out into streams of older and younger ages
epts event processing technical society
Some EP Patterns to review in more detail
1. Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
2. Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
3. Aggregation: add some information based on prior events
– E.g. customer purchase history: track the sum total of order
values for a customer
4. Routing: based on the purchase type pass the event on to
the appropriate service
– E.g. customer purchase event: pass on to a provisioning
service based on the type of product / product classification 57
epts event processing technical society
Enrichment Pattern: Classification (1)
• Alternative Names:
– Event Annotation, Semantic Event
• EPTS Reference Architecture:
– "During event preparation, events may be "enriched"
through knowledge gained through previous events or
data. "
• EPIA:
– Enrich (EPA): a translate EPA that takes a single input
event, uses it to query data from a global state element,
and creates a derived event which includes the attributes
from the original event, possibly with modified values...58
epts event processing technical society
Enrichment Pattern: Description (2)
• Role / Explanation
– Updating some attribute or metadata of the event with some
other values, possibly computed, from some other event or
data
• Associated Business Rule Specification
– a selection (to be enforced during the processing of events):
All <event entities> that are also <enriched data
expression>
must have <some status>.
59
eventEnrichment
f(event)event
Other data
or event
epts event processing technical society
Enrichment Pattern: Structure (3)
• EPTS Glossary comparison
– Enrichment can be specified in an event pattern (pattern
definition containing additional meta data, semantic
knowledge)
– .. as part of an event processing rule (A method for
enriching events during processing, e.g., by analysing the
meta data and querying external data sources or
inferencing semantic knowledge bases)
– .. or as part of event stream processing (additional
additional information to event stream data, e.g. by
querying external data sources)
60
epts event processing technical society
Enrichment Pattern: Implementations (4)
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
61
eventEnrichment
f(event)event
Other data
or event
epts event processing technical society
Enrichment Pattern: Implementation 1 TIBCO BusinessEvents: Rules
• General pattern: <declare> event, data
<if> event-data-match
<then> update-event-with-data
• Effect: find appropriate data for
events and apply enrichment
operations on them
62
event
Identify
enrichment data
Event
Enrich
Operation
on event
Enrich
data
epts event processing technical society
Enrichment Pattern: Implementation 2aOracle CEP: queries
• General pattern template:
SELECT <property>
FROM <STREAM>[WIN],<RELATION>
WHERE <join-condition>
• Effect: enrich events from stream
with contextual data from relation
where condition applies.
63
event
join-condition
event
Operation on
event
epts event processing technical society
Enrichment Pattern: Implementation 2bOracle CEP: model
64
• Scenario:
– For every stock event from stock tick stream whose bid is
greater than 5.0, find its company‟s location.
• Application Model
epts event processing technical society
Enrichment Pattern: Implementation 2cOracle CEP: queries
65
SELECT event.symbol, location
FROM
StockTickStream [NOW] AS event,
StockTable AS data
WHERE event.symbol = data.symbol AND
event.lastBid > 5.0
No need for window
operator on a „relation‟
epts event processing technical society
Enrichment Pattern: Implementation 2dOracle CEP: table metaphor
66
Time Input (StockTickStream) Output (QUERY)
1
2
3
Symbol Full-name Location
AAA The AAA company San Francisco
BBB The BBB company San Jose
{“AAA”, 10.0, 10.5} {“AAA”, “San…”}
{“BBB”, 11.0, 12.5} {“BBB”, “San Jose”}
{“AAA”, 4.0, 4.5}
StockTable
epts event processing technical society
Enrichment Pattern: Implementation 3aProva: rules
• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :-
<query built-in(s)>,
<enrichment condition(s)>
... .
<logical enrichment rule 1> :-
<enrichment operations>.
• Effect: query data from external source
and enrich the event
67
event
Query
event*
Operation on
enriched event
External
data
Enrichment
rules
epts event processing technical society
• Java..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)
• File Input / Output ..., fopen(File,Reader), ...
• XML (DOM)document(DomTree,DocumentReader) :-
XML(DocumenReader),
• SQL …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).
• RDF ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),
• XQuery ..., XQuery = ' for $name in StatisticsURL//Author[0]/@name/text() return $name', xquery_select(XQuery,name(ExpertName)),
• SPARQL...,sparql_select(SparqlQuery,name(Name),class(Class),definition(Def)),
Enrichment Pattern: Implementation 3bProva: External Data and Object Integration
epts event processing technical society
Enrichment Pattern: Implementation 3cProva: Example with SPARQL Query
% Filter for car manufacturer stocks and enrich the stock tick
event with data from Wikipedia (DBPedia) about the manufacturer
and the luxury cars
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
carManufacturer(S,Man), % filter car manufacturers
findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query
EnrichedData = [S,Data], % enrich with additional data
sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).
% rule implementing the query on DBPedia using SPARQL query
luxuryCar(Manufacturer,Name,Car) :-
Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query
WHERE {?car <http://purl.org/dc/terms/subject>
<http://dbpedia.org/resource/Category:Luxury_vehicles> .
?car foaf:name ?name .
?car dbo:manufacturer ?man .
?man foaf:name ?manufacturer.
} ORDER by ?manufacturer ?name“,
sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).
epts event processing technical society
Enrichment Pattern: Implementation 4IBM WebSphere Decision Server: Business Events component
70
event
Mapped Key
look up
Enriched
event
• Example: Define a Mapped Key expression, where a
field in the event is used to look up objects in a
database table
• Effect: required enrichment data is obtained
from the data base and added to event
epts event processing technical society
Some EP Patterns to review in more detail
1. Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
2. Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
3. Aggregation: add some information based on prior events
– E.g. customer purchase history: track the sum total of order
values for a customer
4. Routing: based on the purchase type pass the event on to
the appropriate service
– E.g. customer purchase event: pass on to a provisioning
service based on the type of product / product classification 71
epts event processing technical society
Aggregation Pattern: Classification (1)
• Alternative Names:
– Event Summarization, Composite event, Complex event
• EPTS Reference Architecture:
– "During complex event detection, combining events to
provide new or useful information, such as trend
information and event statistics. Similar to event
consolidation "
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as
input a collection of events and creates a single derived
event by applying a function over the input events.72
epts event processing technical society
eventevent
Aggregation Pattern: Description (2)
• Role / Explanation
– Aggregation is used to describe several events as a single
composite event, generally summarizing a metric of a set of
component events.
– For example: generate a (composite) event that contains the
medium price of a stock over a 10 minute stream of events.
• Associated Business Rule Specification
– a summarisation (to be enforced during event processing):
The <aggregation fn> of <event entities> that
are <selection constraint>
must have
<some constraint>.
73
eventAggregation
f(events)event
epts event processing technical society
Aggregation Pattern: Structure (3)
• EPTS Glossary comparison
– The term aggregate event is sometimes used for some
forms of composite or derived event.
– Composite event: a derived, complex event
• is created by combining base events using a specific set of
event constructors such as disjunction, conjunction,
sequence, etc.
• always includes the base (member) events from which it is
derived.
– Derived event (/synthesized event): an event that is
generated as a result of applying a method or process to
one or more other events.
74
epts event processing technical society
Aggregation Pattern: Implementations (4)
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
75
eventevent
eventAggregation
f(events)event
epts event processing technical society
Aggregation Pattern: Implementation 1TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
aggregation-object
<if>
event-member-of-aggregation
<then>
update-aggregation-object
with event-data
• Effect: construct aggregation
(event) object for each incoming event76
event
Identify
event for
aggregation
Updated
Aggregation
Object
Aggregate
Operation
on event
Aggregation
object
epts event processing technical society
Aggregation Pattern: Implementation 2aOracle CEP: queries
• General pattern template:
SELECT <summary-property>
FROM <STREAM>[WIN]
WHERE <predicate>
GROUP BY <aggregation-identity>
• Effect: summarizes the properties of
several simple event into a new
complex event
77
event
Identify
event for
aggregation
Updated
Aggregation
Object
Aggregate
Operation
on event
Aggregation
object
epts event processing technical society
Aggregation Pattern: Implementation 2bOracle CEP: model
78
• Scenario:
– Output the average bid and ask price of a stock in the
last 10 seconds.
• Application Model
epts event processing technical society
Aggregation Pattern: Implementation 2cOracle CEP: queries
79
SELECT symbol, AVG(bid), AVG(ask)
FROM
StockTickStream [RANGE 10 SECONDS]
GROUP BY
symbol
epts event processing technical society
Aggregation Pattern: Implementation 2dOracle CEP: table metaphor
80
Time Input Output
1s
4s
9s
15s
20s
{“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0}
{“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0}
{“BBB”, 4.0, 5.0}{“AAA”, 11.0, 13},
{“BBB”, 4.0, 5.0}
{“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5}
{“BBB”, 8.0, 10.0}
> 10s
epts event processing technical society
Aggregation Pattern: Implementation 3aProva: rules
• General pattern template:
<rule_head> :-
<Create Aggregator>,
@group(<reaction group>)
@timer|size(Start,Interval,Aggregator)
rcvMsg <EventPattern> [<Aggregate Operation>].
<rule_head> :-
@or(<reaction group>)
rcvMsg <Aggregator>,
... <consume Aggregator>.
• Effect: Repeated incremental
aggregations over new events
81
event
Check & Update
Timer / Size Count
Updated
Aggregation
Object
Aggregate
Operation
on event
Aggregation
object
Identify event
for aggregation
Consume
Aggregation
Object
or
epts event processing technical society
Aggregation Pattern: Implementation 3bProva: Example with Time Counter
% This reaction operates indefinitely. When the timer
elapses (after 25 ms), the groupby map Counter is sent
as part of the aggregation event and consumed in or
group, and the timer is reset back to the second
argument of @timer.
groupby_rate() :-
Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj.
@group(g1) @timer(25,25,Counter) % timer every 25 ms
rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event
[IM=T,Counter.incrementAt(IM)]. % aggr. operation
groupby_rate() :-
% receive the aggregation counter in the or reaction
@or(g1) rcvMsg(XID,self,From,or,[Counter]),
... <consume the Counter aggreation object>.
epts event processing technical society
Aggregation Pattern: Implementation 4aIBM WebSphere Decision Server: Business Events component
83
• General Pattern:
Define how the
events should be
accumulated
... then when
processing the
events, use the
relevant
aggregator
• Effect of the example: sum up the value of a
customer‟s purchases over a week
Add to array
Intermediate
Object Array
Operation
on aggregate
event event
epts event processing technical society
Aggregation Pattern: Implementation 4bIBM InfoSphere Streams: Streams Processing Language
84
• Example:// sliding window
stream<Beat, tuple<int32 maxSalary, uint32 ageOfMaxSalary> >
Agg4 = Aggregate(Beat)
{
window
Beat : sliding, time(10.5), count(10);
output
Agg4 : maxSalary = Max(salary), ageOfMaxSalary = ArgMax(salary, age);
}
• Effect of example: Aggregate the salary values in the
stream within a time-based window.
epts event processing technical society
Some EP Patterns to review in more detail
1. Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
2. Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
3. Aggregation: add some information based on prior events
– E.g. customer purchase history: track the sum total of order
values for a customer
4. Routing: based on the purchase type pass the event on to
the appropriate service
– E.g. customer purchase event: pass on to a provisioning
service based on the type of product / product classification 85
epts event processing technical society
Routing Pattern: Classification (1)
• Alternative Names:
– Event Summarization, Composite event, Complex event
• EPTS Reference Architecture:
– "During event reaction, event routing is the process by
which an event is redirected to some process,
computation element, or other event sink. "
• EPIA: no direct analogy, possibly maps to split /
compose / project
– Project (EPA): a translate EPA that takes an input event
and creates a single derived event containing a subset
of the attributes of the input event.86
epts event processing technical society
Routing Pattern: Description (2)
• Role / Explanation
– Routing is used to describe the process of adding one or
more new destinations to an event.
– For example: route an input event to the appropriate
specialist agent for that event type.
• Associated Business Rule Specification
– a constraint (to be enforced during event processing):
The <event> that
satisfies <selection constraint>
must be assigned to
<destination>.
87
eventRouting
dest = f(event+data)event
Other data
or event
epts event processing technical society
Routing Pattern: Structure (3)
• EPTS Glossary comparison
– Routing (a process on events) is not defined, but is
related to associating an event to an event sink.
– Event sink (event consumer) is an entity that receives
events.
• Examples:
• ƒ Software module
• ƒ Database
• ƒ Dashboard
• ƒ Person
88
epts event processing technical society
Routing Pattern: Implementations (4)
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
89
eventRouting
dest = f(event+data)event
Other data
or event
epts event processing technical society
Routing Pattern: Implementation 1TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
routing-data
<if>
event-matches-routing-data
<then>
create-new-routed-event and
send-new-routed-event
• Effect: match events to routing rule
and construct / send new event90
event
Identify
event for
routing
event
Create routed
event
Routing
data
epts event processing technical society
Routing Pattern: Implementation 2aOracle CEP: query pattern
• General pattern template:
Route 1: <filtering pattern> query
Route 2: <filtering pattern> query
Route n: <filtering pattern> query
• Effect: Different predicates
associated to different queries
evaluate and select appropriate
destinations
91
event
Identify
event for
routing
event
Select destination
Routing
data
epts event processing technical society
Routing Pattern: Implementation 2bOracle CEP: model
92
• Scenario:
– Output stocks whose symbols start with „a-m‟ to
destination 1, and whose symbols start with „n-z‟ to
destination 2.
• Application Model
epts event processing technical society
Routing Pattern: Implementation 2cOracle CEP: queries
93
Destination-1:
SELECT *
FROM StockTickStream
WHERE symbol.matches(“^[a-m]”)
Destination-2:
SELECT *
FROM StockTickStream
WHERE symbol.matches(“^[n-z]”)
epts event processing technical society
Routing Pattern: Implementation 3Prova: rules
• General pattern template:
rcvMsg <Event Msg Pattern> :-
<routing decisions>,
sendMsg <route event >.
<routing decision rule 1> :-
<decision conditions>.
<routing decision rule 2> :-
<decision conditions>.
...
• Effect: Events are routed
according to the routing decision
rules94
event
Identify
event for
routing
event
Create routed
event
Routing
data
Routing
Decisions
Route
event
epts event processing technical society
Routing Pattern: Implementation 3bProva: Example with Agent (Sub-) Conversations
rcvMsg(XID,esb,From,query-ref,buy(Product) :-
routeTo(Agent,Product), % derive processing agent
% send order to Agent in new subconversation SID2
sendMsg(SID2,esb,Agent,query-ref,order(From,
Product)),
% receive confirmation from Agent for Product order
rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).
% route to event processing agent 1 if Product is
luxury
routeTo(epa1,Product) :- luxury(Product).
% route to epa 2 if Product is regular
routeTo(epa2,Product) :- regular(Product).
% a Product is luxury if the Product has a value over …
luxury(Product) :- price(Product,Value), Value >=
10000.
% a Product is regular if the Product ha a value below
…
epts event processing technical society
Routing Pattern: Implementation 4IBM: Business Events component of WebSphere Decision Server
• Example: Define an interaction set,
which will send the request to supplier A,
when the purchase indicates supply
from Supplier A
•
• Effect of example:
Events are routed as required
96
event
Identify
event
routing
event
Routing
information
Consume
event
epts event processing technical society
97
Agenda
• Introduction to the EPTS Functional Reference
Architecture, and what it contains
• Mapping Functions to Patterns: some sample
Event Processing Patterns
• EPTS member pattern implementation examples
• Summary and future work on EP Patterns
epts event processing technical society
Another Pattern Type
Other patterns may be required for
implementation, such as handling
produced-consumer interactions
epts event processing technical society
Application-Time Pattern
• Problem:
– Need to use application‟s view of time, instead of CPU wall-clock.
– This is particularly useful when events originate from different machines and need some way of synchronizing.
• Scenario:
– Again, consider bid and ask stream…
– However, bid and ask requests are time-stamped on trader at the time that the order is placed.
99
epts event processing technical society
Application-Time Pattern
• Scenario:
– Seller places two ask requests respectively at time
8:00:00 and 8:00:12
– Buyer places one bid request at time 8:00:11
100
epts event processing technical society
Application-Time Pattern
• Scenario:
– Remember that we want to correlate using a 10 seconds
window, as anything older is stale…
– Hence first event from seller should not be considered,
and we should correlate the ask price of 11.0 with the bid
price of 9.5
101
epts event processing technical society
Application-Time Pattern
• Scenario:
– However, the reality is that the first two events could
arrive together in a burst in the exchange, and the third
could be delayed in the cloud…
102
epts event processing technical society
Application-Time Pattern
• Scenario:
– In this case, bid price of 9.5 would correlate to ask price of 10.0
and the exchange would loose money as the spread is lower…
103
epts event processing technical society
Application Time-stamped Pattern: Implementation 1Oracle CEP
• Solution:
– The query does not change…
– However STREAMS must be configured to use
application time-stamps based upon some event
property, instead of having events being system-
timestamped…
104
epts event processing technical society
Application Time-stamped Pattern: Implementation 2IBM Business Monitor
• Example: Define a correlation expression
• Effect of example: The duration of a customer order can be obtained from
the timestamps on start and end events
105
„Order Complete‟
event
Correlate
Calculate
Duration
„Order Placed‟
event
epts event processing technical society
Application Time Pattern: Implementation 3 Prova: rules
• General pattern template:
rcvMsg <Event Msg Pattern(s)> :-
<add application timestamp to event>,
sendMsg <Adjusted Event>.
• Effect: Any timestamp can be adjusted
(for time computations)
106
Event
adjust event
Updated
Event
epts event processing technical society
Application Time-stamped Pattern: Implementation 4TIBCO BusinessEvents: PreProcessor Pattern
• General Pattern: adjust Event metadata (e.g.
Timestamp) in preprocessor for that event
– events are generally non-monotonic!
– PreProcessor applies regardless of engine used!
• Effect: Any custom timestamp can be adjusted (for time
computations)
107
Event
Preprocess:
adjust event
Updated
Event
Note: If the problem is “out of order” events
(rather than synchronizing clocks) a
different pattern may be better!
epts event processing technical society
Use Cases
Illustrating using some of the patterns
epts event processing technical society
IBM Use Case: Utilizing Smart MetersLarge energy company improved customer responsiveness using event processing
Use Case
• Utilized events (and lack of events) from
smart meters to detect service outages and
disruptions
• Empowered operations team to define
outage detection and response patterns
without coding or scripting
Smarter Business Outcomes
• Automated detection and response to service
outages
• Improved customer satisfaction by delivery of
estimated time to recovery alerts based on
customer preferences
Challenge: Reliance on customer calls
to detect and respond to outages and
reduce time to recovery while providing
more information to the customer
epts event processing technical society
110
Customer Care
Manager(s) with IBM
Business Monitor
WebSphere
Business Events
Business
Events
Power
Disruption
3
Data Collection
Engine (DCE)
Outage
Management
System (OMS)
Tivoli OMNIbus,
Network Manager,
and Impact
Customer
Data
1
1
2
4
Location, Estimated Duration
of DisruptionAlert and Assist Impacted
Customers
Energy Company Leveraging Smart Meters to Improve Customer Responsiveness
epts event processing technical society
Oracle Use-case: Capital Markets
• Low-Latency Trading
– Algorithm Trading
• Trade Matching
– High throughput
– In-flight fraud
• Position Capture and Aggregation
• Risk and Analytics
– Pre/post trade analytics
– On-demand risk management
111
epts event processing technical society
TIBCO Use Case: Application Service GatewayTelco high performance service policies using event processing
112
Mobile Services
difficult to monitor
end to end
Auto detect and
fixing of issues
for better
user service
Excellent service quality
needed to avoid
customer churn
Service quality affects
take-up of new services
epts event processing technical society
• Standardized channel for monitoring subscribers
• Standardized access to services (e.g. Facebook, Google, etc)
• Consolidation of existing multiple gateways
• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc
• Creates events for revenue generation and reconciliation
• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc
Service Gateway architecture exploits event processing
epts event processing technical society
• Standardized channel for monitoring subscribers
• Standardized access to services (e.g. Facebook, Google, etc)
• Consolidation of existing multiple gateways
• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc
• Creates events for revenue generation and reconciliation
• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc
Service Gateway architecture exploits event processing
Routing Pattern
Transformation
Patterns
Decision / Policy
Pattern
Security Pattern
epts event processing technical society
115
Agenda
• Introduction to the EPTS Functional Reference
Architecture, and what it contains
• Mapping Functions to Patterns: some sample
Event Processing Patterns
• EPTS member pattern implementation examples
• Summary and future work on EP Patterns
epts event processing technical society
Summary: Realization Patterns / Idioms
Event Production Application Time
Publication, Retrieval
Event PreparationIdentification, Selection, Filtering,
Monitoring, Enrichment
Complex Event DetectionConsolidation, Composition,
Aggregation
Event ReactionAssessment, Routing, Prediction,
Discovery, Learning
Event Consumption Dashboard, Apps,External Reaction
Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification
0..*
0..*
0..*
0..*
Sta
te M
an
ag
em
en
t
epts event processing technical society
Categorization of Event Processing Patterns
Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008
http://arxiv.org/abs/0806.1100v1
epts event processing technical society Categorization of Patterns
• Categorization according to Good and Bad Solutions– CEP Patterns
– CEP Anti-Patterns
• Categorization according to the Abstraction Level– Guidelines and Best Practices
– Management patterns
– Architecture patterns
– Design patterns
– Mapping patterns
– Idioms / Realization patterns
– Smells / Refactoring patterns
• Categorization according to the Intended Goal– Adoption patterns
– Business patterns
– Integration patterns
– Composite patterns:
– …
• Categorization according to the Management Level– Strategic patterns
– Tactical patterns
– Operational patternsSource:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing,
Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60.
http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf
epts event processing technical society
Next Steps
• Continue evolution of EP Reference Architecture
– Eg Logical Architecture vs Systems Description
• Evolve the EP Pattern <<metapattern>> to be more useful for developers and tools
• Expand coverage to all Functions as Patterns, + extend Functions as necessary
• Planned Outputs:
– EPTS-RA Description white paper
– Wiley Book "Pattern Oriented Software Architecture: Architectures, Models and Patterns for Event Processing" (already in preparation)
– Input for Event Processing Metamodels and associated standards
epts event processing technical society
Summary
• Reference Architecture provides a common set of
EP Functions that may be mapped
to other EP ontologies / taxonomies
• Provides the basis for discussing EP Patterns
• EP Patterns can be semantic, design or runtime
focused
• Further evolution of EP Pattern and examples is
likely!
120
epts event processing technical society
121
Thank you !
Acknowledgment to the Event Processing Technical Society
Reference Architecture working group members
epts event processing technical society
References
• EPTS Glossary
http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/