Real-Time Processing Layer - ProaSense · Real-Time Processing Layer Goal-Driven Event Processing...
Transcript of Real-Time Processing Layer - ProaSense · Real-Time Processing Layer Goal-Driven Event Processing...
FP7-ICT-2013.1.3
Brussels, 24.11.2014
Real-Time Processing LayerGoal-Driven Event Processing
Dominik Riemer, FZI
• Introduction– Introduction to Complex Event Processing
– Real-Time Processing Layer: Objectives
– Role of layer in the ProaSense architecture
– Research problems related to the layer
• Methodology: Overview
• Setup Phase
• Execution Phase
• Current Status and Roadmap– Current Status
– Future Work
Outline
2
Complex Event ProcessingConceptual model
Sensors
Applications
Processes
Notifications
Actions
Even
t Pro
duce
rs
Even
t Con
sum
ersEvent Processing
Network
Relevant Situations
Humans
4
Real-Time Processing LayerChallenges and Objectives
ObjectivesChallenges
• The real-time processing layerhandles high-volume datastreams produced by a large variety of sensors
• Objective is to identify potential failures immediately
• many patterns are needed todetect such situations
• large-scale, distributed real-time infrastructure is needed
• Methodology for Goal-Driven Event Processing
• Models and implementation forsemantic descriptions ofheterogeneous event processingagents
• Modelling and deployment of real-time processing pipelines
5
Real-Time Processing LayerIdentified research problems
Pattern Complexity
Pattern Expressivity
Pattern Reusability
Pattern Identification
Design-time challenges of Complex Event Processing:How can non-technical users identify and model event patterns?
detailed next slides
Proactive monitoring of potential failures and situations of interest
restricted by
7
High complexity• complex structure for (relatively) simple use
cases• development effort similar to traditional
programming languages
Limited expressivity• many use cases typically require custom
event processing logic• limited expressivity of pattern languages
require implementation of external functions• e.g., detection of drilling start
Limited reusability• fixed event schemas• static definition of event properties• hard-coded values
Implementation example (Esper EPL)
Research ProblemsComplexity, Expressivity, ReusabilityTask: Detection of an abnormal oil temperature increase (e.g. 10% above normal) measured over 30% of the drilling period when drilling RPM exceeds a threshold …
INSERT INTO HighTemperatureEventSELECT * FROM pattern [a=DrillingProcessEvent ANDb=AggHighTemperatureEvent where timer:within(2 min)]
INSERT INTO DrillingProcessEventSELECT timestamp, variable_typeFROM DrillingRPM match_recognize(partition by deviceIdmeasures B.value as rpmpattern (A B) define A as A.drillingStart, B asB.drillingStop
create schema DrillingRPM as (variable_type string, timestamp long, value double, deviceId int)
create schema GearLubeOilTemperature as (variable_type string, timestamp long, value double)
INSERT INTO AggHighTemperatureEvent(g.variable_type, g.value, avg(g.value) SELECT * FROM GearLubeOilTemperature.win:time(2 min) as g group by variable_type
8
Identification
Scope of event patterns:• real-time/proactive monitoring• detection of threats• detection of opportunities
Requirements:• set of patterns is complete• set of patterns is error-free• support for heterogeneous technical systems
Problems:• gap between business requirements and
technical pattern representation
Research ProblemsPattern identificationTask: Identify all patterns which are of interest in a particular domain
PIPIPM5
PIPIP
ContextTechnical Representation
PIPIKPI
PIPIP
PIPIP
Business Requirements
9
What‘s wrong with existing approaches?Graphical tool support for CEP
- Programming-oriented developmentenvironments
- Definition of custom (e.g., JAVA-based) functions required
- Translation into exactly onelanguage (e.g., Panteon->Esper, Oracle->CQL)
- Completely missing for distributedstream processing
Example: Oracle CEPProblems
10
Semantic Event Processing Agent
Semantic Event Producer
Main contributionMethodology for Goal-Driven Event Processing
Methodology: Overview
Execution PhaseSetup Phase
Ontologies
Example
Conceptual Model
Ontologies
Example
Conceptual Model
Processing Pipelines
Conceptual Model
SemanticIntegration
Example
11
Setup PhaseSetup Phase Execution PhaseExecution Phase
Pattern EngineersPattern Engineers
Business AnalystsBusiness Analysts
Technical ExpertsTechnical Experts
Methodology
Source Modelling
Stream Modelling
Stream Implementation
SEPA Modelling
Performance Indicators
Business Goals
Key Performance Indicators
Pipeline Implementation
Pipeline Deployment
Pipeline Evolution
Pipeline Identification
Register
SEPAOntologySEPA
Ontology
Requirements
Requirements
detailed next slides
Event OntologyEvent
Ontology
SEPA Implementation
13
Concrete Implementation
Methodology: Outcome
High-level processing pipelines
Common semantic descriptionHow can heterogeneous event processing logic be integrated?
Concrete Implementation Concrete Implementation
StandaloneCEP Engine TopologyAlgorithm
Proa
Sens
eW
P3Im
plem
enta
tions
14
Semantic Event Producer (S-EP)Semantic Event Processing
Agent (S-EPA)Processing Pipeline
Increase
Aggregation
ShaftSpeed
Increase
Aggregation
ShaftSpeed
Concepts
• performs an eventprocessing logic
• provides input eventstream requirements, restrictions and outputdescription
• produces one or moreevent streams
• describes streams basedon functional and non-functional (e.g., quality) aspects
• transformation processfrom input event streamsto an output eventstream
• consists of a set of S-EPAs and S-EPs
• single pipeline elementscan be implemented byarbitrary underlyingsystems
• tree structure (consumeras top-level elements)
15
Setup PhaseSetup Phase Execution PhaseExecution Phase
Pattern EngineersPattern Engineers
Business AnalystsBusiness Analysts
Technical ExpertsTechnical Experts
Methodology: Overview
Source Modeling Stream Modeling
Stream Implementation
SEPA Modeling
Business Goals
Strategy
Pipeline Implementation
Pipeline Deployment
Pipeline Evolution
Pipeline Identification
Register
SEPAOntologySEPA
Ontology
Requirements
Requirements
detailed next slides
Event OntologyEvent
Ontology
SEPA Implementation
Semantic Event Producer
17
Setup PhaseSemantic Event Producer: Ontologies
Semantic Event Producer Event Stream
Event Source
Event Quality
Event Grounding
Event Schema
Event Feature
rdf:Datatype
muo:Measurement Unit
non‐functional aspects:‐ e.g., frequency, latency, accuracy
source‐related aspects:‐ e.g., sensor/application/human
technical aspects:‐ e.g., transport protocol, data format
functional aspects:‐ e.g., event header, payload
produces
has..
eventFeature
measurementUnit
featureType
rdfs:classtypeOf
mhwirth:DDM mhwirth:DrillingSpeedShaft
Concept
Instance
mhwirth:speed
muo:rpm
xsd:doublexsd:string
value
featureName
18
Setup PhaseSemantic Event Producer: Ontologies
Semantic Event Producer Event Stream
Event Source
Event Quality
Event Grounding
Event Schema
Event Feature
rdf:Datatype
muo:Measurement Unit
non‐functional aspects:‐ e.g., frequency, latency, accuracy
source‐related aspects:‐ e.g., sensor/application/human
technical aspects:‐ e.g., transport protocol, data format
functional aspects:‐ e.g., event header, payload
produces
has..
eventFeature
measurementUnit
featureType
rdfs:classtypeOf
mhwirth:DDM mhwirth:DrillingSpeedShaft
Concept
Instance
mhwirth:speed
muo:rpm
xsd:doublexsd:string
value
featureName
Event Source• determines the source an event is
produced by• e.g., sensor, application, website• allows modelling of source-based
restrictions (e.g., static or movingsensors)
19
Setup PhaseSemantic Event Producer: Ontologies
Semantic Event Producer Event Stream
Event Source
Event Quality
Event Grounding
Event Schema
Event Feature
rdf:Datatype
muo:Measurement Unit
non‐functional aspects:‐ e.g., frequency, latency, accuracy
source‐related aspects:‐ e.g., sensor/application/human
technical aspects:‐ e.g., transport protocol, data format
functional aspects:‐ e.g., event header, payload
produces
has..
eventFeature
measurementUnit
featureType
rdfs:classtypeOf
mhwirth:DDM mhwirth:DrillingSpeedShaft
Concept
Instance
mhwirth:speed
muo:rpm
xsd:doublexsd:string
value
featureName
Event Quality• determines the quality level of a
stream• e.g., frequency, accuracy, certainty• linked to ssn:Quality
20
Setup PhaseSemantic Event Producer: Ontologies
Semantic Event Producer Event Stream
Event Source
Event Quality
Event Grounding
Event Schema
Event Feature
rdf:Datatype
muo:Measurement Unit
non‐functional aspects:‐ e.g., frequency, latency, accuracy
source‐related aspects:‐ e.g., sensor/application/human
technical aspects:‐ e.g., transport protocol, data format
functional aspects:‐ e.g., event header, payload
produces
has..
eventFeature
measurementUnit
featureType
rdfs:classtypeOf
mhwirth:DDM mhwirth:DrillingSpeedShaft
Concept
Instance
mhwirth:speed
muo:rpm
xsd:doublexsd:string
value
featureName
Event Grounding• determines the technical grounding
of an event stream (protocol, port, topic)
• e.g., JMS, MQTT, HTTP, Kafka
21
Setup PhaseSemantic Event Producer: Ontologies
Semantic Event Producer Event Stream
Event Source
Event Quality
Event Grounding
Event Schema
Event Feature
rdf:Datatype
muo:Measurement Unit
non‐functional aspects:‐ e.g., frequency, latency, accuracy
source‐related aspects:‐ e.g., sensor/application/human
technical aspects:‐ e.g., transport protocol, data format
functional aspects:‐ e.g., event header, payload
produces
has..
eventFeature
measurementUnit
featureType
rdfs:classtypeOf
mhwirth:DDM mhwirth:DrillingSpeedShaft
Concept
Instance
mhwirth:speed
muo:rpm
xsd:doublexsd:string
value
featureName
Event Schema• determines the (functional) schema
of a single event• Event Feature as single event
properties (e.g., sensor observations)
• features can have a measurement unit (MUO) and a specific type
22
Setup PhaseSetup Phase Execution PhaseExecution Phase
Pattern EngineersPattern Engineers
Business AnalystsBusiness Analysts
Technical ExpertsTechnical Experts
Methodology: Overview
Source Modeling Stream Modeling
Stream Implementation
SEPA Modeling
Business Goals
Strategy
Pipeline Implementation
Pipeline Deployment
Pipeline Evolution
Pipeline Identification
Register
SEPAOntologySEPA
Ontology
Requirements
Requirements
detailed next slides
Event OntologyEvent
Ontology
SEPA Implementation
Semantic Event Processing Agents
23
Setup PhaseSemantic Event Processing Agent: Ontologies
Event StreamSemantic Event Processing Agent
requires
Static Feature Output Strategy
outputStrategy
Append
Fixed
Custom
Rename
staticFeature
AnyStaticFeature
FreeTextStaticFeature
MappingStaticFeature
MatchingStaticFeature
OneOfStaticFeature
24
Setup PhaseSemantic Event Processing Agent: Ontologies
Event StreamSemantic Event Processing Agent
requires
Static Feature Output Strategy
outputStrategy
Append
Fixed
Custom
Rename
staticFeature
AnyStaticFeature
FreeTextStaticFeature
MappingStaticFeature
MatchingStaticFeature
OneOfStaticFeature
Required event streams• restrictions on stream inputs
25
Setup PhaseSemantic Event Processing Agent: Ontologies
OneOfStaticFeature
Event StreamSemantic Event Processing Agent
requires
Static Feature Output Strategy
outputStrategy
Append
Fixed
Custom
Rename
staticFeature
AnyStaticFeature
FreeTextStaticFeature
MappingStaticFeature
MatchingStaticFeature
OneOfStaticFeature
Output strategy• controls output of a SEPA• output depends on input schema(s)
and SEPA type• e.g., ENRICH SEPA type:
AppendOutputStrategy• currently supported: Append, Keep,
Replace, Fixed, Custom
26
Setup PhaseSemantic Event Processing Agent: Ontologies
Event StreamSemantic Event Processing Agent
requires
Static Feature Output Strategy
outputStrategy
Append
Fixed
Custom
Rename
staticFeature
AnyStaticFeature
FreeTextStaticFeature
MappingStaticFeature
MatchingStaticFeature
OneOfStaticFeature
Static feature• defines static data required to
instantiate a SEPA• e.g., threshold values, time window
specifications,
27
Setup PhaseSemantic Event Processing Agent: Ontologies
Event StreamSemantic Event Processing Agent
requires
Static Feature Output Strategy
outputStrategy
Append
Fixed
Custom
Rename
staticFeature
AnyStaticFeature
FreeTextStaticFeature
MappingStaticFeature
MatchingStaticFeature
OneOfStaticFeature
Mapping static feature• defines mapping of an event feature
to a SEPA operation
28
Setup PhaseSemantic Event Processing Agent: Ontologies
Event StreamSemantic Event Processing Agent
requires
Static Feature Output Strategy
outputStrategy
Append
Fixed
Custom
Rename
staticFeature
AnyStaticFeature
FreeTextStaticFeature
MappingStaticFeature
MatchingStaticFeature
OneOfStaticFeature
Matching static feature• defines mapping between two event
features of different streams• e.g., A AND B where a.x=b.y
29
Setup PhaseSemantic Event Processing Agent: Example
Event Restrictions- payload contains numerical value- payload produced by a sensor
Quality Restrictions- min frequency 1 event/sec- min sensor accuracy
Static data requirements- time window- static threshold: value- dynamic threshold
Capabilities max throughtput
Event restrictions- events produced by speed shaft- measurement unit RPM
Quality restrictions- min frequency
Static data requirements- min RPM
Capabilities- at-least-once processing
Increase
Aggregation
Drilling Status
Algorithm
30
Setup PhaseSemantic Event Processing Agent: Example
Increase
Event Stream
EventFeature
xsd:number
featureType
eventFeature
requires
AnyStaticFeature
percentage
Option Option
staticFeature
option
absolute
FreeTextStaticFeature
value Drilling Status
Event Stream
EventFeature
muo:rpm
measurementUnit
eventFeature
requires
speedShaft
rdf:type
Increase
Aggregation
Drilling Status
Algorithm
31
Setup PhaseSetup Phase Execution PhaseExecution Phase
Pattern EngineersPattern Engineers
Business AnalystsBusiness Analysts
Technical ExpertsTechnical Experts
Methodology: Overview
Source Modeling Stream Modeling
Stream Implementation
SEPA Modeling
Business Goals
Strategy
Pipeline Implementation
Pipeline Deployment
Pipeline Evolution
Pipeline Identification
Register
SEPAOntologySEPA
Ontology
Requirements
Requirements
detailed next slides
Event OntologyEvent
Ontology
SEPA Implementation
33
Processing PipelinesConceptual Model
ShaftSpeed
OilTemperature
Drilling Status
Algorithm
Drilling Status : Drilling
Filter
Increase
AggregationAND
Pattern Detect Dashboard
Event Consumer
abnormal oil temperature rise (10% above normal) measured during the drilling period
StandaloneCEP Engine
Topology
Algorithm
34
Semantic IntegrationExample
Connection valid?
Additional user input required?
Increase S‐EPA output schema?
?Increase
Aggregation
ShaftSpeed
35
Semantic Integration
IncreaseShaftSpeed
Define static properties and concrete mappings
Validate connectionCompute mapping properties
Compute output schema
Connection is validated based on four metrics:• Feature match, schema match, quality
match, grounding match
If connection is valid:• for all required mapping properties of P: find possible mappings in S
Task
Increase S-EPA requires an event stream with an event feature of type xsd:number Shaft Speed provides such a feature
(mhwirth:speed) -> connection valid
Find all event features in S and add mapping property list
Example
′ , ′ 1 | ∀ ∈ : ∃ ∈ ′ ∧0 |
Actor: System
36
Semantic IntegrationDefine static properties and
concrete mappingsValidate connection
Compute mapping propertiesCompute output schema
For all static properties of P• provide static properties as user‐definedinput (text fields, selections)
For all concrete mapping properties of P• provide list of mappings as user‐defined input (selection)
Task
Increase S-EPA requires a static property indicating a minimum increase threshold
As only mhwirth:speed matches the input requirements, no mapping must be chosen
Example
IncreaseShaftSpeed
Actor: User
37
Semantic IntegrationDefine static properties and concrete mappings
Validate connectionCompute mapping properties
Compute output schema
Compute output based on:
• concrete input event schema
• defined mapping properties
• predefined output strategy
Connect next element
For each S-EPA
• compute invocation graph
Task
Increase S-EPA defined output strategy Custom• users are able to manually define the
output
A Filter S-EPA would define an output strategy Keep, which is defined by the input.
Example
IncreaseShaftSpeed
Actor: System
38