Reactive programming at scale
-
Upload
john-mcclean -
Category
Technology
-
view
705 -
download
0
Transcript of Reactive programming at scale
![Page 1: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/1.jpg)
ReactiveProgramming @ Scale
![Page 2: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/2.jpg)
Paul Downey● Director of UI & APIs for demand advertising● Based in Dublin, Ireland● Email: [email protected] McClean● Systems Architect Demand Side Forecasting● Based in Dublin, Ireland.● Twitter: cyclops_aol● Medium: https://medium.com/@johnmcclean● Email: [email protected]
About Speakers
![Page 3: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/3.jpg)
A
A
A
P
P
P
Exchange
![Page 4: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/4.jpg)
RTB Ad Cycle
100 ms
20,000,000,000+
![Page 5: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/5.jpg)
● Responsive
● Scale
● Resilient
● Capacity to process lots of data
System Requirements
![Page 6: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/6.jpg)
![Page 7: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/7.jpg)
Moore’s Law
7
![Page 8: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/8.jpg)
Race Conditions
Deadlocks
Contention
Non-Deterministic State
Thread StarvationLivelocks
![Page 9: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/9.jpg)
Amadahl’s Law
9
![Page 10: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/10.jpg)
Concurrency in Practice.
![Page 11: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/11.jpg)
Java Concurrency In Theory
11
![Page 12: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/12.jpg)
Java Concurrency In Practice
12
![Page 13: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/13.jpg)
Why is it like this?
13
![Page 14: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/14.jpg)
Why is it like this?
14
![Page 15: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/15.jpg)
Can we fix this?
15
![Page 16: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/16.jpg)
What is Reactive?
16
![Page 17: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/17.jpg)
What is Reactive?
17
Functional Reactive Programming?
Actor based concurrency?
![Page 18: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/18.jpg)
Functional Reactive Programming.
![Page 19: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/19.jpg)
Is it something new?
19
1997
![Page 20: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/20.jpg)
Big in 1997?
20
![Page 21: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/21.jpg)
Is it something new?
21
![Page 22: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/22.jpg)
Well established on the front end
22
![Page 23: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/23.jpg)
What is it?
![Page 24: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/24.jpg)
Functional programming?
24
![Page 25: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/25.jpg)
Functional programming
25
Less buggy code
Easier to parallelise
Easier to optimise
![Page 26: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/26.jpg)
Reactive programming
26
Reactive pipeline manages flow
What not How
![Page 27: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/27.jpg)
Reactive programming
27
What to do
Change each element by multiplying by 100
Remove all elements over 550
Print out each remaining element
![Page 28: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/28.jpg)
Reactive programming
28
Composition
Change each element by multiplying by 100
Remove all elements over 550
Print out each remaining element
map(e -> e*100)
forEach(System.out::println)
filter(e->e<551)
![Page 29: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/29.jpg)
Reactive programming
29
.map(e -> e*100)
.forEach(System.out::println)
.filter(e->e<551)
Example
ReactiveSeq.of(6,5,1,2)
![Page 30: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/30.jpg)
A Reactive Pipeline
30
first stage second stage terminal stage
![Page 31: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/31.jpg)
A Reactive Pipeline
31
first stage(map)
second stage(filter)
terminal stage (forEach)
![Page 32: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/32.jpg)
A Reactive Pipeline
32
first stage(map)
second stage(filter)
terminal stage (forEach)
Data is pushed through the pipeline
500
2
1
5 600
![Page 33: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/33.jpg)
Forecasting
![Page 34: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/34.jpg)
Forecasting
34
Campaign Planning and Optimization
Query data across petabytes data
Accuracy & Responsiveness
![Page 35: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/35.jpg)
Price / Volume Curve
![Page 36: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/36.jpg)
Every Day We Process
36
Bid Request Records
Impression Records Viewability Records
20B 2B 2B
![Page 37: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/37.jpg)
Indexing Data
37
![Page 38: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/38.jpg)
Reactive Indexing
![Page 39: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/39.jpg)
What we need to do
![Page 40: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/40.jpg)
A Reactive Pipeline
40
first stage(map)
second stage(filter)
terminal stage (forEach)
Data is pushed through the pipeline
500
2
1
5 600
![Page 41: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/41.jpg)
Scaling Up
41
first stage(map)
second stage(filter)
terminal stage (forEach)core 1
first stage(map)
second stage(filter)
terminal stage (forEach)core 2
first stage(map)
second stage(filter)
terminal stage (forEach)core 3
first stage(map)
second stage(filter)
terminal stage (forEach)core 4
![Page 42: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/42.jpg)
Scaling Up for I/O
42
first stage(map)
second stage(filter)
terminal stage (forEach)
future 1
future 2
future 3
future 4
future 5
future 6
future 7
future 8
future 9
future 10 future
11
![Page 43: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/43.jpg)
Performance Differences
43
![Page 44: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/44.jpg)
What would the code look like?
![Page 45: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/45.jpg)
Sequential Stream
45
.map(e -> e*100)
.forEach(System.out::println)
.filter(e->e<551)
Example
ReactiveSeq.of(6,5,1,2)
.futureOperations(executor)
![Page 46: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/46.jpg)
Async I/O
46
.map(e -> e*100)
.filter(e->e<551)
Example
FutureW.of(()->loadData(6))
//non-blocking, FutureW executes asynchronously
.forEach(System.out::println)
![Page 47: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/47.jpg)
Parallel Async Stream
47
.map(this::loadData)
.forEach(System.out::println)
.filter(e->e<551)
Example
new LazyReact(100).of(6,5,1,2)
![Page 48: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/48.jpg)
What we need to do
![Page 49: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/49.jpg)
Reactive Indexing Architecture
49
![Page 50: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/50.jpg)
What have we learnt?
![Page 51: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/51.jpg)
Conclusions
51
![Page 52: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/52.jpg)
Conclusions
52
![Page 53: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/53.jpg)
Conclusions
53
![Page 54: Reactive programming at scale](https://reader030.fdocuments.in/reader030/viewer/2022013004/586fdbdc1a28ab18428b62c9/html5/thumbnails/54.jpg)
Thank You.