Monitoring Complex Systems - Chicago Erlang, 2014
-
Upload
brian-troutwine -
Category
Software
-
view
796 -
download
1
description
Transcript of Monitoring Complex Systems - Chicago Erlang, 2014
![Page 1: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/1.jpg)
Monitoring Complex Systems
![Page 2: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/2.jpg)
I do things to/with computers.
![Page 3: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/3.jpg)
I build real-time systems.
![Page 4: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/4.jpg)
I build fault-tolerant systems.
![Page 5: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/5.jpg)
I build critical systems.
![Page 6: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/6.jpg)
AdRoll
![Page 7: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/7.jpg)
Less this.
![Page 8: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/8.jpg)
More this.
![Page 9: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/9.jpg)
Engineering + Mathematics = ads
![Page 10: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/10.jpg)
Engineering + Mathematics = ads
(you’re welcome)
![Page 11: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/11.jpg)
R E A L - T I M E B I D D I N G
![Page 12: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/12.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
![Page 13: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/13.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
• Firm real-time system
![Page 14: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/14.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
• Firm real-time system
• Highly concurrent (90 billion transactions per day)
![Page 15: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/15.jpg)
The Problem Domain• Low latency ( < 100 ms per transaction)
• Firm real-time system
• Highly concurrent (90 billion transactions per day)
• Global, 24/7 operation
![Page 16: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/16.jpg)
I build Complex Systems
![Page 17: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/17.jpg)
Complex Systems
• Non-linear feedback
• Tightly coupled to external systems
• Difficult to model, understand
![Page 18: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/18.jpg)
Bad things happen when Complex Systems fail.
![Page 19: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/19.jpg)
Humans are bad at predicting the performance of complex systems(…). Our ability to create large and complex systems fools us into believing that we’re also entitled to understand them.
C A R L O S B U E N O “ M AT U R E O P T I M I Z AT I O N H A N D B O O K ”
![Page 20: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/20.jpg)
Complex Systems often create worse problems than those they solve.
![Page 21: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/21.jpg)
The key challenge to sustaining a complex
system is maintaining our understanding of it.
![Page 22: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/22.jpg)
What can be done?
![Page 23: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/23.jpg)
Ahead of time verification is not sufficient.
!
(don’t scrimp on it, though)
![Page 24: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/24.jpg)
Compile-time guarantees are not sufficient.
!
(don’t scrimp on them, either)
![Page 25: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/25.jpg)
We need insight into the running system.
![Page 26: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/26.jpg)
• VM killers
What are we looking for?
![Page 27: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/27.jpg)
• VM killers
• Application performance regressions
What are we looking for?
![Page 28: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/28.jpg)
• VM killers
• Application performance regressions
• Abnormal application behavior
What are we looking for?
![Page 29: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/29.jpg)
• VM killers
• Application performance regressions
• Abnormal application behavior
• Surprises
What are we looking for?
![Page 30: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/30.jpg)
INSTRUMENTATION
![Page 31: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/31.jpg)
The BEAM is ready to ride.
![Page 32: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/32.jpg)
erlang:memory/1
![Page 33: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/33.jpg)
erlang:statistics/1
![Page 34: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/34.jpg)
erlang:system_info/1
![Page 35: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/35.jpg)
What about our own work?
![Page 36: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/36.jpg)
Exometer
![Page 37: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/37.jpg)
Important Termsmetric a measurement
entry a receiver and aggregator of metrics
reporter that which samples entries periodically and ships them to another system
subscription the definition of the regular interval on which reporters sample entries
![Page 38: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/38.jpg)
These are all loosely coupled at runtime.
![Page 39: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/39.jpg)
Configuration is static, but you can adapt it on the fly.
![Page 40: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/40.jpg)
Why exometer over the alternatives?
![Page 41: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/41.jpg)
It is extensively documented.
![Page 42: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/42.jpg)
It’s vigorously maintained.
![Page 43: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/43.jpg)
It’s vigorously maintained.
(Ulf Wiger fan club day.)
![Page 44: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/44.jpg)
It is silly fast.
![Page 45: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/45.jpg)
Okay, great. We have instrumentation.
![Page 46: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/46.jpg)
Now what?
![Page 47: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/47.jpg)
MONITORING
![Page 48: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/48.jpg)
This is the hard part.
![Page 49: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/49.jpg)
Visualization
![Page 50: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/50.jpg)
Alerting
![Page 51: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/51.jpg)
Analysis
![Page 52: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/52.jpg)
Visualization tells you how things look but not why.
![Page 53: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/53.jpg)
A good day.
![Page 54: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/54.jpg)
Uh oh.
![Page 55: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/55.jpg)
Growth is good, but steady is better.
![Page 56: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/56.jpg)
Bids are
stable.
![Page 57: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/57.jpg)
Budgets are
stable.
![Page 58: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/58.jpg)
What happened?
![Page 59: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/59.jpg)
We forgot about Labor Day.
![Page 60: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/60.jpg)
![Page 61: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/61.jpg)
Alerting tells you that something happened, but not why.
![Page 62: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/62.jpg)
A normal day.
![Page 63: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/63.jpg)
Wat?
![Page 64: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/64.jpg)
That’s some cliff.
![Page 65: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/65.jpg)
Timeouts look good.
![Page 66: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/66.jpg)
Errorsprior are okay.
![Page 67: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/67.jpg)
What happened?
![Page 68: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/68.jpg)
“Uh, hey guys, you know Facebook is down, right?”
![Page 69: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/69.jpg)
Analysis gives you why but only if you know how to ask for what.
![Page 70: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/70.jpg)
The memory use of a bidder.
![Page 71: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/71.jpg)
ಠ_ಠ
![Page 72: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/72.jpg)
It’s all binaries.
![Page 73: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/73.jpg)
Not in processes.
![Page 74: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/74.jpg)
Not in ETS.
![Page 75: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/75.jpg)
Come on now.
![Page 76: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/76.jpg)
What happened?
![Page 77: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/77.jpg)
A jiffy bug.
![Page 78: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/78.jpg)
A jiffy bug.
(we think)
![Page 79: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/79.jpg)
Shout out to Miriam Pena for spending two weeks
tracking this down.
![Page 80: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/80.jpg)
Okay, great. We have monitoring and instrumentation.
![Page 81: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/81.jpg)
Now all our problems are solved, right?
![Page 82: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/82.jpg)
Not quite.
![Page 83: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/83.jpg)
Instruments make up for our lack of insight.
![Page 84: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/84.jpg)
Monitoring makes up for our frailty.
![Page 85: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/85.jpg)
Every solution brings its own problems.
![Page 86: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/86.jpg)
Instruments may be misleading.
![Page 87: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/87.jpg)
Instruments may be overwhelming.
![Page 88: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/88.jpg)
Instruments may be
inaccurate.
![Page 89: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/89.jpg)
Instruments may be ignored.
![Page 90: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/90.jpg)
What can be done?
![Page 91: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/91.jpg)
A little paranoia never hurt
anyone.
![Page 92: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/92.jpg)
Use glass displays.
![Page 93: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/93.jpg)
Train.
![Page 94: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/94.jpg)
Keep sight of the main goal.
![Page 95: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/95.jpg)
Have resources you’re willing to sacrifice.
![Page 96: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/96.jpg)
AdRoll is Hiring! :D
![Page 97: Monitoring Complex Systems - Chicago Erlang, 2014](https://reader031.fdocuments.in/reader031/viewer/2022013011/5476d47cb4af9f9c3f8b4572/html5/thumbnails/97.jpg)
Thanks, folks!<3
@bltroutwine