EFL Munich - February 2013 - "Conversational Big Data with Erlang"
-
Upload
darach -
Category
Technology
-
view
1.809 -
download
1
description
Transcript of EFL Munich - February 2013 - "Conversational Big Data with Erlang"
![Page 1: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/1.jpg)
Copyright Push Technology 2012
EFL Munich February 2013
![Page 2: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/2.jpg)
Copyright Push Technology 2012 [email protected]
About me?
• Distributed Systems / HPC guy.
• Chief Scien*st :-‐ at Push Technology
• Alumnus of :-‐ Motorola, IONA, BeOair, JPMC, StreamBase.
• School: Trinity College Dublin. -‐ BA (Mod). Comp. Sci.+ -‐ M.Sc. Networks & Distributed Systems
• Responds to: Guinness, Whisky
![Page 3: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/3.jpg)
Copyright Push Technology 2012
Conversa[onal Big Data with
![Page 4: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/4.jpg)
Copyright Push Technology 2012
Big Data. The most important V is Value.
• 4Vs: • Volume • Velocity • Variety • Variability
• “It’s the connec[ons that ma`er most” -‐ David Evans, Cisco
![Page 5: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/5.jpg)
Copyright Push Technology 2012
Connect all the things
![Page 6: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/6.jpg)
Copyright Push Technology 2012
An Internet Minute
10k, 1ms 1k, 100us
![Page 7: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/7.jpg)
Copyright Push Technology 2012
The Last Mile is HARD
y = -‐1.1667x + 45703 R² = 0.02496
1.00E+00
1.00E+01
1.00E+02
1.00E+03
1.00E+04 Sep-‐06 Oct-‐06 Oct-‐06 Nov-‐06 Dec-‐06 Dec-‐06 Jan-‐07 Feb-‐07
Latency
Latency (ms)
Expected (ms)
Linear (Latency (ms))
-‐10.00
0.00
10.00
20.00
30.00
Sep-‐06 Oct-‐06 Oct-‐06 Nov-‐06 Dec-‐06 Dec-‐06 Jan-‐07 Feb-‐07
Download Bandwidth
Expected (Mbps)
Download (Mbps)
Buffer Bloat Network
Satura[on Ba`ery!
![Page 8: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/8.jpg)
Copyright Push Technology 2012
Conversa[ons
M2M M2H Tradi[onal Messaging MQTT, AMQP
WebSockets
W3C Server Sent Events
Bidirec[onal Real Time Data Distribu[on
![Page 9: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/9.jpg)
Copyright Push Technology 2012
Tradi[onal Messaging
• Loosely coupled. • All you can eat messaging pa`erns • Familiar
• No data model. Slinging blobs • Fast producer, slow consumer? Ouch. • No data ‘smarts’. A blob is a blob.
Pros Cons
Producers Consumers?
A
Bba
bb
![Page 10: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/10.jpg)
Copyright Push Technology 2012
Invented yonks ago…
Before the InterWebs For ‘reliable’ networks For machine to machine Remember DEC Message Queues? -‐ That basically. Vomit!
![Page 11: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/11.jpg)
Copyright Push Technology 2012
When fallacies were simple
- The network is reliable - Latency is zero - Bandwidth is infinite - There is one administrator - Topology does not change - The network is secure - Transport cost is zero - The network is homogeneous
![Page 12: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/12.jpg)
Copyright Push Technology 2012
Then in 1992, this happened:
The phrase ‘surfing the internet’ was coined by Jean Poly. First SMS sent First base.
![Page 13: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/13.jpg)
Copyright Push Technology 2012
It grew, and it grew
![Page 14: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/14.jpg)
Copyright Push Technology 2012
Then in 2007, this happened:
The god phone: Surfing died. Touching happened. Second base unlocked.
![Page 15: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/15.jpg)
Copyright Push Technology 2012
Then in 2007, this happened:
So we took all the things and put them in the internet: Cloud happened. So we could touch all the things. They grew and they grew like all the good things do.
Messaging Apps Hardware Virtualize all the things Services Skills, Special[es
![Page 16: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/16.jpg)
Copyright Push Technology 2012
Then in 2007, this happened:
And it grew and it grew. Like all the good things do.
Messaging Apps Hardware Virtualize all the things Services Skills, Special[es
![Page 17: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/17.jpg)
Copyright Push Technology 2012
So we scaled all the things
Big data is fundamentally about extrac[ng value, understanding implica[ons, gaining insight so we can learn and improve con[nuously.
It’s nothing new.
big
DATA
![Page 18: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/18.jpg)
Copyright Push Technology 2012
But, problem: The bird, basically.
Immediately Inconsistent. But, Eventually Consistent … Maybe. Humans Are grumpy
IIBECM
![Page 19: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/19.jpg)
Copyright Push Technology 2012
Stop.
- The network is not reliable nor is it cost free.
- Latency is not zero nor is it a democracy.
- Bandwidth is not infinite nor predictable especially the last mile!
- There is not only one administrator trust, rela[onships are key
- Topology does change It should, however, converge eventually
- The network is not secure nor is the data that flows through it
- Transport cost is not zero but what you don’t do is free
- The network is not homogeneous nor is it smart
![Page 20: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/20.jpg)
Copyright Push Technology 2012
Look.
- What and How are what geeks do. - Why gets you paid - Business Value and Trust dictate What and How - Policies, Events and Content implements Business Value
- Science basically. But think like a carpenter:
- Measure twice. Cut once.
![Page 21: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/21.jpg)
Copyright Push Technology 2012
Listen.
- Every nuance comes with a set of tradeoffs. - Choosing the right ones can be hard, but it pays off. - Context, Environment are cri[cal - Break all the rules, one benchmark at a [me.
- Benchmark Driven Development
![Page 22: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/22.jpg)
Copyright Push Technology 2012
Ac[on: Data Distribu[on
Messaging remixed around: Relevance -‐ Queue depth for conflatable data should be 0 or 1. No more
Responsiveness -‐ Use HTTP/REST for things. Stream the li`le things Timeliness -‐ It’s rela[ve. M2M != M2H.
Context -‐ Packed binary, deltas mostly, snapshot on subscribe.
Environment -‐ Don’t send 1M 1K events to a mobile phone with 0.5mbps.
![Page 23: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/23.jpg)
Copyright Push Technology 2012
Ac[on. Virtualize Client Queues
BufferBloatProducers Consumers
A
Bba
bb
Nuance: Client telemetry. Tradeoff: Durable subscrip[ons harder
![Page 24: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/24.jpg)
Copyright Push Technology 2012
Ac[on. Add data caching
One hop closer to the edge …
Is it a cache?
Producers Consumers
A
Bba
bb x
x
![Page 25: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/25.jpg)
Copyright Push Technology 2012
Ac[on. Exploit data structure
Snapshot recovery. Deltas or Changes mostly. Conserves bandwidth
State!Producers Consumers
A
Bba
bb x
x
Snapshot Delta
![Page 26: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/26.jpg)
Copyright Push Technology 2012
Ac[on. Behaviors
Extensible. Nuance? Roll your own protocols. Tradeoff? 3rd party code in the engine :/
The topic is
the cloud!
Producers Consumers
A
Bba
bb x
x
X
![Page 27: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/27.jpg)
Copyright Push Technology 2012
Ac[on. Structural Confla[on
Ensures only current + consistent data is distributed. Ac[vely soaks up bloat. Extensible!
Currentdata!
Producers Consumers
A
Bba
bb x
x
X
+
![Page 28: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/28.jpg)
Copyright Push Technology 2012
Ac[on. Structural Confla[on [EEP]
• Replace/Overwrite ‘A1’ with ‘A2’ • ‘Current data right now’ • Fast
• Merge A1, A2 under some opera[on. • Opera[on is pluggable • Tunable consistency • Performance f(opera[on)
Replace Merge
Cop Bop Aop ... ...C2 A2 C1 B1 A1 'op'
C2 B1 A2C2 A2 C1 B1 A1 R
C2 B1 A+C2 A2 C1 B1 A1 M+12345 5
=A2 + A1
=4 + 1
1
12345 415
8=
C2 + C1=
5 + 3
Current
Current +
Consistent
![Page 29: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/29.jpg)
Copyright Push Technology 2012
Implement Client J
![Page 30: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/30.jpg)
Copyright Push Technology 2012
Example Diffusion Client
Backend to Basho’s Lager logging – Stream log events for near real-‐[me analysis …
![Page 31: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/31.jpg)
Copyright Push Technology 2012
Benchmark Driven Development
‘Measuring Value’
![Page 32: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/32.jpg)
Copyright Push Technology 2012
Performance is not a number
![Page 33: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/33.jpg)
Copyright Push Technology 2012
Benchmark Models
Box A Box B
• Ramp clients con[nuously • 100 messages per second per client • Payload: 125 .. 2000 bytes • Message style vs with confla[on
• Really simple. 1 client • Ping – Pong. Measure RTT [me. • Payload: 125 .. 2000 bytes
Throughput (Worst case) Latency (Best case)
![Page 34: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/34.jpg)
Copyright Push Technology 2012
Throughput. Non-‐conflated • Cold start server
• Ramp 750 clients ‘simultaneously’ at 5 second intervals
• 5 minute benchmark dura[on
• Clients onboarded linearly un[l IO (here) or compute satura[on occurs.
• What the ‘server’ sees
![Page 35: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/35.jpg)
Copyright Push Technology 2012
Throughput. Non-‐conflated • What the ‘client’ sees
• At and beyond satura[on of some
resource?
• Things break!
• New connec[ons fail. Good. • Long established connec[ons ok. • Recent connec[ons [meout and client
connec[ons are dropped. Good.
• Diffusion handles smaller message sizes more gracefully
• Back-‐pressure ‘waveform’ can be tuned out. Or, you could use structural confla[on!
![Page 36: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/36.jpg)
Copyright Push Technology 2012
Throughput. Replace-‐conflated • Cold start server
• Ramp 750 clients ‘simultaneously’ at 5
second intervals
• 5 minute benchmark dura[on
• Clients onboarded linearly un[l IO (here) or compute satura[on occurs.
• Takes longer to saturate than non-‐conflated case
• Handles more concurrent connec[ons
• Again, ‘server’ view
![Page 37: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/37.jpg)
Copyright Push Technology 2012
Throughput. Replace-‐Conflated • What the ‘client’ sees
• Once satura[on occurs Diffusion
adapts ac[vely by degrading messages per second per client
• This is good. Soak up the peaks through fairer distribu[on of data.
• Handles spikes in number of connec[ons, volume of data or satura[on of other resources using a single load adap[ve mechanism
![Page 38: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/38.jpg)
Copyright Push Technology 2012
Throughput. Stats
• Data / (Data + Overhead)? • 1.09 GB/Sec payload at satura[on. • 10Ge offers theore[c of 1.25GB/Sec. • Ballpark 87.2% of traffic represents business data.
• Benefits? • Op[mize for business value of distributed data. • Most real-‐world high-‐frequency real-‐[me data is recoverable • Stock prices, Gaming odds
• Don’t use confla[on for transac[onal data, natch!
![Page 39: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/39.jpg)
Copyright Push Technology 2012
Latency (Best case) • Cold start server
• 1 solitary client
• 5 minute benchmark dura[on
• Measure ping-‐pong round trip [me
• Results vary by network card make, model, OS and Diffusion tuning.
![Page 40: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/40.jpg)
Copyright Push Technology 2012
Latency. Round Trip Stats
• Sub-‐millisecond at the 99.99% percen[le
• As low as 15 microseconds
• On average significantly sub 100 microseconds for small data
![Page 41: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/41.jpg)
Copyright Push Technology 2012
Latency. Single Hop Stats
• Sub 500 microseconds at the 99.99% percen[le
• As low as 8 microseconds
• On average significantly sub 50 microseconds for small data
![Page 42: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/42.jpg)
Copyright Push Technology 2012
Latency in perspec[ve
• 2.4 us. A tuned benchmark in C with low latency 10Ge NIC, with kernel bypass, with FPGA accelera[on
• 5.0 us. A basic java benchmark – as good as it gets in java • Diffusion is measurably ‘moving to the le{’ release on release • We’ve been ac[vely tracking and con[nuously improving since 4.0
2.4 us 5.0 us 8.0 us 50.0 us
~1000bytes(avg)
~125bytes(avg)
![Page 43: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/43.jpg)
Copyright Push Technology 2012
Embedded Event
Processing
![Page 44: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/44.jpg)
Copyright Push Technology 2012
CEP, basically
• Add aggregate func[ons and window opera[ons to Erlang • Separate context (window) from computa[on (aggregate fn) • 4 window types: tumbling, sliding, periodic, monotonic • Process oriented. • Fast enough. J
What is eep.erl?
S Cw ww w Q
![Page 45: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/45.jpg)
Copyright Push Technology 2012
Aggregate Func[ons
• A func[on that computes values over events. • The cost of calcula[ons are ammor[zed per event • Just follow the above recipe • Example: Aggregate 2M events (equity prices), send to GPU on emit, receive 2M op[ons put/call prices as a result.
What is an aggregate func*on?
![Page 46: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/46.jpg)
Copyright Push Technology 2012
Aggregate Func[on Example
![Page 47: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/47.jpg)
Copyright Push Technology 2012
Tumbling Windows
• Every N events, give me an average of the last N events • Does not overlap windows • ‘Closing’ a window, ‘Emits’ a result (the average) • Closing a window, Opens a new window
What is a tumbling window?
1 2 3 4
2 3 4 5
2 3 4 5
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ...
init()
init()
init()
emit()
emit()
emit()
x() x() x() x()
x() x() x() x()
x() x() x() x()
![Page 48: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/48.jpg)
Copyright Push Technology 2012
Tumbling Windows
![Page 49: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/49.jpg)
Copyright Push Technology 2012
Sliding Windows -‐ O(Long Story)
• Like tumbling, except can overlap. But O(N2), Keep N small.
• Every event opens a new window. • A{er N events, every subsequent event emits a result. • Like all windows, cost of calcula[on amor[zed over events
What is a sliding window?
1 2 3 4
1 2 3 4
1 2 3 ..
1 2 .. ..
5
..
..
..
..
..
init()
x()
x()
x()
..
.. ..
..
..
..
..
..
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ...
![Page 50: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/50.jpg)
Copyright Push Technology 2012
Compensa[ng Aggregates
1
1 2 3 4
1 2 3 4
1 2 3 ..
1 2 .. ..
5
..
..
..
..
..
init()
x()
x()
x()
..
.. ..
..
..
..
..
..
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ...
do { … } while (…)
compensate()
![Page 51: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/51.jpg)
Copyright Push Technology 2012
Sliding Windows.
![Page 52: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/52.jpg)
Copyright Push Technology 2012
Periodic Windows
• Driven by ‘wall clock [me’ in milliseconds • Not monotonic, natch. Beware of NTP
What is a periodic window?
1 2 3 4
2 3 4 5
2 3 4 5
t0 t1 t2 t3 ...
init()
init()
init()
emit()
emit()
emit()
x() x() x() x()
x() x() x() x()
x() x() x() x()
![Page 53: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/53.jpg)
Copyright Push Technology 2012
Periodic Windows
![Page 54: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/54.jpg)
Copyright Push Technology 2012
Monotonic Windows
• Driven mad by ‘wall clock [me’? Need a logical clock? • No worries. Provide your own clock! Eg. Vector clock
What is a monotonic window?
1 2 3 4
2 3 4 5
2 3 4 5
t0 t1 t2 t3 ...
init()
init()
init()
emit()
emit()
emit()
x() x() x() x()
x() x() x() x()
x() x() x() x()
my my my
![Page 55: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/55.jpg)
Copyright Push Technology 2012
Monotonic Windows
![Page 56: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/56.jpg)
Copyright Push Technology 2012
EEP looking forward?
• Migrate windows from simple processes to gen_server. • Extract library (no process / message passing overhead). • Consider NIFs for performance cri[cal sec[ons. • Consider na[ve aggregate func[ons. • Consider alterna[ve to gen_event for interfacing • Streams / Pipes? Hey, shoot me, but I like Node.js Streams/Pipes. • Add more func[ons. • Add more languages • Node.js EEP – h`ps://github.com/darach/eep-‐js • React/PHP EEP -‐ h`ps://github.com/ianbarber/eep-‐php
![Page 57: EFL Munich - February 2013 - "Conversational Big Data with Erlang"](https://reader034.fdocuments.in/reader034/viewer/2022042813/5497150fac795982318b474a/html5/thumbnails/57.jpg)
Copyright Push Technology 2012 [email protected]
Ques[ons?
• Thank you for listening to, having me • Le twi`er: @darachennis
• EEP in Erlang / PHP / Node.js:
h`ps://github.com/darach/eep-‐erl
• h`ps://github.com/ianbarber/eep-‐php h`ps://github.com/darach/eep-‐js
• Push Technology’s Diffusion product now supports Erlang Clients too. [Alpha, ping me if interested, email below]
PROMO CODE: ENNI100