EVENTS: DISTRIBUTED PROGRAMMING MADE EASY

STORM + F#

Storm + F#

ABOUT ME

Eugene TolmachevPrincipal Architect @ ProlucidDeveloping software for over 20 years Open source contributor:• FsStorm maintainer

• Bitlore author

• FsUnit

• Dapper

• MassTransit

• Nowin

Storm + F#

SCALE OUT & AVAILABILITY

UI Backend DB

A = Aui Abe Adb

Storm + F#

DISTRIBUTED COMPUTING

DRPC/Actors• CORBA/DCOM• Web-services• AKKA• Orleans

Data freshness

ESP/CEP• Storm• Spark

Batch processing• Custom ETL• Hadoop• mbrace• Spark

Storm + F#

ASK VS TELL

Model

Price

Risk Date

getRisk(date)risk

date

getModel(date) model

Stack: … getPrice getRisk getModel …

• Easy to start with• Composed at design-time• Fault handling: on the caller• Challenging to cache• Fragile integration

Storm + F#

EVENTS (FIRE-N-FORGET)

Implemented via Store-and-forward messagingRepresent a summary of a changeSelf-contained*Time-relevant, not time-sensitiveUniquely identifiable

• Tracing/correlation• Idempotency

Storm + F#

ASK VS TELL

• Upfront setup• Can compose at runtime• Fault handling:

Compensations• Caching – unnecessary• Can evolve independently• Embarrassingly parallel

Price

Risk

DateModel

modelUpdated(m,date)

riskUpdated(r,m,date)

newDay(date)

newDay(date)calculate Model

calculate Risk

calculate Price

generate Date

Multiple Stacks:

Storm + F#

MICRO-SERVICES

Services encapsulate• Business-logic• Can evolve independently as long as the surface API remains the same

Services control the context• Persistence• Security• Performance

Micro-services+ Do one thing only, do it well- Introduce latency- Deployment/management overhead

Storm + F#

THERMOSTAT ALERTSensor

is up

Sensor value

Settings updated

Write down raw data Evaluate against

the threshold Send notification

Read all sensors

Write down the settings

Write down the result

Storm + F#

APACHE STORMRuns on JREStorm

• Spouts: throttled• Streams: default and named• Tuples: anchored or not• Bolts: input – on demand/output – at will• Grouping (affinity): all, shuffle, fields, direct

Storm multilang protocol:• JSON over stdin/stdout• Components: a process per instance

Storm CLI, to start:• storm dev-zookeeper (runtime coordination)• storm nimbus (management/api)• storm supervisor (runs components)• storm ui (optionally, http://localhost:8080)

Storm + F#

THERMOSTAT TOPOLOGY

sensorStartedEvents[1]

sensorUpdateEvents[1]

settingsUpdateEvents[1]

sensorWriter[2]

sensorViewProjection[2]

sensorViewUpdatedNotifier[1]

settingsUpdatedNotifier[1]

settingsWriter[1]

sensorViewWriter[2]

allSensorsReader[1]

Storm + F#

FSSTORMRuntime• Implements multilang protocol• Lets you write components in F#• Additional implementations for unit-testing

DSL:• Define topology in F#• Validated

Scripts• Submit your topology to nimbus