Event Driven Architecture

Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Driven Architecture

Lourens Naudé – SAPO Codebits 2010


About

• Developer @ http://www.wildfireapp.com• Ruby / C• Well versed full stack• Active interest in Event Driven technologies

and related business processes

http://www.wildfireapp.com/


Agenda

• Business and systems growth• Events• (Lack of) control• Event Driven Architecture

• Domain Driven Design primer• Command and Query Responsibility

Segregation• Complex Event Processing


Maintenance cost is multiple times more than

development cost


http://en.wikipedia.org/wiki/Open/closed_principle

“Systems should ideally be open for extension, but

closed for modification”

http://en.wikipedia.org/wiki/Open/closed_principle


Driving forces for change

• Fulfillment of business requirements• Business expansion: rollout of new

processes to drive sales, market penetration etc• Mergers and acquisitions• Legislation


Modification risks

• Introduces regression testing and a QA burden – never compromise a working system for a functional requirement

• Profiling overhead for (soft) realtime systems

• Excessive “refactoring” of existing stable interfaces

• May fracture a business domain


EDA imposes a system design that's geared for

extension, interoperability and unanticipated use.


Interoperability VS Integration

• Two or more systems performing a task / process together

• Integration implies a formal link / bridge between two systems - high coupling

• Building with interoperability in mind is the easiest way for long-term integration

• Should be accounted for in system lifecycle as well: delays, cost, communication etc.


Governance (lay down the law)• Technology standards, SLAs, development

processes• Cannot achieve interoperation without

governance• Poor governance: inappropriate policies,

people (habits) and business structures• Policies should be enforced at runtime

in an auditable manner• Otherwise like criminal law without

police and courts


Business layers


Business Layers

• Physical world: sensors and users as actors

• Transactional: physical world interactions with the business / domain layer

• Intelligence: reporting tier and insights, most notably sales and marketing performance

• This is where EDA really shines: visibility into business processes

• Transparent operation for easier compliance and lower support costs


Events


Event Definition

• Anything that happened (or didn't happen)

• A change in state• A condition that triggers a notification• An event is always named in the past tense

and is immutable

CustomerAddressChangedInventoryRelocatedOrderShipped


Three levels of detail

• Basic fact that an event occurred• Event definition required in order to

recognize an event• Event detail / context• Always attempt to capture events at the point

of greatest information value (context)• This reduces downstream lookup and

correlation overhead


Event Types

• Notification: informational• Alert: notification that should cause a

response• Event Object: record of an event - just data,

no behavior• Business Event: meaningful for conducting

trade activities• Complex: abstraction of one or more other

events that represents summary level insights


Notification Types

• Observational• Reports an event

butdoes not changeanything

LowStockCartCleared

• Transactional• Reports an event

and causes change

• Reliable transport• Originates from

within the org.

OrderShippedCardProcessed


(Lack of) Control


Event Driven Programming

• Control flow is driven by external events

• NO central controller (main) - contrary to what we're taught starting off as devs

• No well defined flow of data


EDA Components

• Event producers• Event consumers / listeners• Event processors• Event reactions• Messaging infrastructure


Lack of control in EDA

• An EDA is driven by extensibility and not controllability

• Transaction state is managed by events, not some central controller

• Loosely coupled, asynchronous and stateless• Tell downstream components what

happened, but not what to do.• Business events enable a system to operate

under control, but without central control


Reduced coupling

• There's always some coupling• We strive to reduce the degree of

coupling• Always coupled to data


Stateless Order process

• Modifying processes managed by a central controller requires changes to a component as well as the controller

• Move state / data from the central controller to events


Event Driven Architecture


Use cases

• Government: security, border control, taxes etc.

• Health: stream processing + research• Compliance: guard against violations in

cross country laws• Track and trace: Fedex, UPS etc.• Service Level Agreements


When not to use an EDA

• If the potential for reuse is very little • Problem being solved is lower down the

application stack• Network bandwith and capacity is

limited – serialization overheads and tends to be chatty

• Security boundaries: immutability of events that carry sensitive data


EDA Workloads

• Simple event processing: narrow design• Stream processing: high message

volume, but only relevant events is filtered• Complex Event Processing (CEP): able

to correlate multiple streams


Interaction Types

• Time-driven: actions initiated at specified times, eg. batch workflows etc.

• Request-driven: initiated by client and completes when the provider replies, eg. client server

• Event-driven: initiated by events and participants is open ended


Hybrid sytems

Most event driven systems also include time and request

driven components.


Components

• Event producers• Event consumers / listeners• Event processors• Event reactions• Messaging infrastructure


Nervous System Analogy

• Hand is the event producer• Spinal cord is the ESB• Brain is the event listener• Brain is the event processor• Hand is the event reaction – we pull away


Event Producers

• Generates event data OR transforms it into a format that the EDA can understand

• Data isn't always generated to feed an EDA, especially if the producer is a legacy component


Event Consumers / Listeners

• Knows how to differentiate an event from data it receives

• Can only “hear” events it's supposed to

• Should be able to detect and interpret an event


Event Processors

• Able to asses event importance and derives the next action to take

• No value in not handling a perceived event• "Do nothing" is a valid reaction as

well


Event Reactions

• Automated responses• Notifications to participants or people• Human reactions / interventions for

offline processes


Messaging Infrastructure

• Communication link between EDA participants

• Doesn't have to be a single piece of infrastructure - distinct backbones should be able to share information

• Embrace standards and be as universal as possible

• Promote decoupling between participants - pub/sub


Data in Event Driven Systems


Event Data

• Business event data held in an event object

• Context of what happened

ProductCategorized('dsaed32wed', 'Coffee')OrderCancelled('ko90323', 'No funds')


State Data

• Core system data that changes as a result of operational activity

Order('ko90323', 'Tom Jones', 13.45 'open')Product('dsaed32wed', 'Delta', 'Coffee', 5.8)


Reference Data

• Stable / somewhat permanent - could change, but a lot less frequently than state data

• Reference data would be replaced whereas state data would change incrementally.

SupportedCurrency('USD')ProductCategory('Coffee')Address('Some Street', 'CA', 'USA')


Lifecycle of Data

• New state data = state data + event data applied

• State data is the accumulation of event data over time

• Event data as deltas

Product('dsaed32wed', 'Delta', 'Coffee', 5.8)ProductCategorized('dsaed32wed', 'Tea')Product('dsaed32wed', 'Delta', 'Tea', 5.8)


Domain Driven Design primer


Domain Driven Design

• Suitable for behavior driven systems• Modeling is often data and not

behavior driven• A Domain Model is a representation of

relationships between Entities and Values• Try to have a business analyst review

processes before introducing new architecture


Common Language

• Participants: Developers, Experts and Stakeholders

• Nouns: things with / without identity• Verbs: actions, hints at state changes• Domain models evolve through language• We often write different to how we speak and

think• Most failures of systems is communication

failures of people


Entities

• Has an identity within the system• State can change

ProductCustomerOrder


Value Objects (Values)

• No identity• Immutable• Represented by it's structure - structural

equivalence

MoneyOrderStatusTimezone


Domain Services

• Actions, operations and activities• Business driven intent• Stateless• Should handle behavior that doesn't map

well to any particular entity

ShippingQuotesServicePaymentProcessingService


Aggregates

• A group of Entities that belong together

• An instance of a process• A single unit of work – consistency /

transactional boundary• eg. Order and Order items• Order item is useless without an Order• Remove Order item: reconstitute the Order,

find item via Order, remove item• Usually identified by cascading delete

requirements


Rental administration example


Command and Query Responsibility Segregation


Either perform an action, or return a result

Not both.


Overview

• CAP theorem: only can have 2 of 3• Consistency steps aside for Availability and

Partitionability• Somewhat stale data is often acceptable• Business requirements define by how much• Split systems into distinct command and query

components (read and write)


Conceptual view


Benefits

• Split teams: high level + good communicators on the write side, junior devs on the read side

• Ability to scale command and reporting sides independently

• Auditable system (dependent on history)• Selectively disable functionality during

upgrades on the write side whilst the reporting side is still available

• Lends itself well to behavior driven extension• Easy to test


CQRS Components

• Commands: drives change in the system• Internal events: captures intent (event

store)• External events: republished to let other

components know• Queries: examines state changes


Eventual Consistency

• All data is stale, unless the whole system uses pessimistic locking

• Most web apps request data *before* they render screens

• Add 250ms to 500ms on average for rendering and additional server side blocking operations

• Users don't react immediately to UI changes

• Strictly consistent on the write side, relaxed (eventual) consistency on the read side


Rethinking UI / adaptors

• Users don't think in data, but in tasks and processes

• Cannot get benefits of CQRS without considering the UI first

• Focus on capturing user intent explicitly

• User experience should be an integral part of any business process


Ambigious UI

UpdateComment / CommentUpdated


Task Driven UI

ApproveComment / CommentApproved


Commands

• A well defined task with parameters• Represents a process, not an entity• Distinct and transactional unit of work:

wholly accepted or rejected• The only way to change data in a system• Just the right amount of granularity and intent

to not cause concurrency conflicts of updates on large entities

• Rejectable with exception or fault event• Idempotent – should never be reprocessed


Rethinking Validations

• There's a fundamental difference between a validation and a business rule

• Validation: well structured and formatted etc.• Business rule: uniqueness constraints etc.• Validate in client and in command handler -

not all clients are well behaved• Valid commands can fail, most often

through optimistic concurrency conflicts• Common for offline processes to reject requests


Queries and Reports

• Read only view of data - examines state in the system

• Never goes through the domain model• Table per UI view (assuming relational

storage)• Single call to the reporting store returns

all (or most) data for a screen• Minimal / no transformation - reporting

tier is a cache


Aggregates and Events

---------- begin consistency boundary ----------

=> Commands::CategorizeProduct("blends", "1517fd7efbffacdf928056656bbb61cf")

rebuild Aggregates::Product(1517fd7efbffacdf928056656bbb61cf) from Events::ProductCreated({:category=>"coffee", :product_id=>"1517fd7efbffacdf928056656bbb61cf", :sku=>"XYZ", :name=>"Delta"})

Aggregates::Product @ version 1

<= Events::ProductCategorized({:category=>:blends, :product_id=>"1517fd7efbffacdf928056656bbb61cf"})

Aggregates::Product @ new version 2

---------- end consistency boundary ----------


Code example

class Aggregates::Product......def rename(name) apply Event(:ProductRenamed, uuid, name)endprivatedef apply_product_renamed(event) @name = event.nameend ......end


Internal Events

• Domain / Event store is append only: no deletes, ad hoc reads.

• Compensating events to delete• Event store is a history of intent• Replay: events allow for reproducing error

conditions by rebuilding an aggregate via snapshot up to just before a problem occurred

• Republishes any handled events to other components


External Events

• Events handled by our domain that's been republished to the rest of our system

• Communicates any state changes to the reporting tier

• Integration / interoperation hook for business growth


ORM Issues

• Suitable for structural models only• Structural models tend to loose

information• Mapping objects to structed data is difficult

to optimize• CRUD – what changes on UPDATE ?


Testing

• Command -> Domain -> [Event, Event …]• Able to assert behaviors in two ways• What happened ?• What didn't happen ?• Same pattern for testing the reporting side• Event -> View -> assert state from getters


Assertion example

def test_relocate_inventory Domain << Command(:CreateInventory, uuid, 123, 'Delta', 'A', 'X',10) Domain << Command(:RelocateInventory, uuid, 'B', 'Y') assert_published :InventoryCreated, :InventoryRelocatedend


Complex Event Processing


Overview

• Connecting the dots• Summary level information gathered

from base events• Patterns identified by business experts, but

enforced at runtime by CEP software• Value is proportional to accuracy and

relevance• Accurate data can however still be

irrelevent• Allows for Management by Exception


Consciousness / awareness

• Without consciousness can't react to situations without human intervention

• SOA's the nervous system• EDA brings awareness• The core of Military systems – situational

awareness on the battlefield derived from input streams


Event Stream

• Linear (ordered by time) sequence of events• Produced by a single component / system• Little aggregation required• High throughput• Minimal insights into what happened• Example: market data feed


Event Cloud

• Collection of Event Streams within a system• Aggregates streams into event sets• Much lower throughput• Very good situational awareness• Example: trading strategies that's

dependent on multiple market feeds, realtime news sources etc.


Event Cloud layout


CEP in Retrospective

• Adds value to Event Driven systems• “Complex” is misleading – it simplifies input

and streams• Matches real-time events from an Event

Cloud with patterns from historical data and detect or predict situations

• Event -> correlate -> assess -> decide -> action


Summary

• Restist modification• Tasks and intent – behavior matters• Communicate• Reduce coupling and avoid central

control


@methodmissinghttp://www.github.com/methodmissing

CQRS Example

https://github.com/methodmissing/aftermath

Thanks !

Questions ?

http://www.github.com/methodmissing

https://github.com/methodmissing/aftermath

Event Driven Architecture

Technology

Transcript of Event Driven Architecture