Graph Everything
-
Upload
kunal-kerkar -
Category
Technology
-
view
647 -
download
2
description
Transcript of Graph Everything
![Page 1: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/1.jpg)
Graph Everything
![Page 2: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/2.jpg)
Kunal Kerkar
@tsudot
![Page 3: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/3.jpg)
We enable Phone Calls and SMS Apps to be built with a few lines of code.
a = {‘from’: ‘555-1234’, # calling party, ‘to’ : ‘919611111111, # called party, ’answer_url‘ : ‘http://example.com/answer’}
plivo.make_call(a)
![Page 4: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/4.jpg)
Points to be discussed
1. Data at Plivo2. Graphite3. Carbon Daemons4. Configuring Carbon5. Whisper Database6. The render URL API, Graphene
![Page 5: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/5.jpg)
Components
Tank
![Page 6: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/6.jpg)
![Page 7: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/7.jpg)
zgrep ‘inbound calls’ 20120823-‐plivo.tar.gz | wc -‐l
![Page 8: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/8.jpg)
•Track calls/SMS made every hour•Number of recharges, sign ups•Endpoints registered•WebRTC vs Softphones•Server metrics
What we wanted to do?
![Page 9: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/9.jpg)
Graphite
![Page 10: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/10.jpg)
• Enterprise-‐scale monitoring tool• Written by Chris Davis at Orbitz in 2006 • Opensourced in 2008
Graphite
![Page 11: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/11.jpg)
1.Store numeric time-‐series data2.Render graphs of this data on demand
What does it do?
![Page 12: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/12.jpg)
1.carbon -‐ a Twisted daemon that listens for time-‐series data
2.whisper -‐ a simple database library for storing time-‐series data
3.graphite webapp -‐ A django webapp that renders graphs on-‐demand using cairo
Architecture : in a nutshell
![Page 13: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/13.jpg)
• 1 daemon, carbon-‐cache.py • Listen for time-‐series data • Use a common set of protocols
Carbon Daemons
![Page 14: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/14.jpg)
• Accepts metrics • Writes them to disk
carbon.confport (2003/2004/7002) protocols (newline delimited, pickle)transports (TCP/UDP)
storage-‐schemas.confDefines a retention policy for incoming metrics based on regex patterns.
carbon-cache.py
![Page 15: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/15.jpg)
•A name, specified inside square brackets•A regex, specified after “pattern=”•A retention rate line, specified after “retentions=”
storage-schemas.conf
[plivoOutboundCalls]pattern=plivoOutboundCalls$retentions=10s:14d
![Page 16: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/16.jpg)
[plivoOutboundCalls]pattern=plivoOutboundCalls$retentions=10s:14d
• [plivo_outbound_calls] is the name of this rule
• Will match any metric that ends with plivoOutboundCalls. plivo.server.proxy.p1.plivoOutboundCallsplivo.server.proxy.p1.plivoOutboundCalls.tsudot
• The retention line says that each datapoint represents 10 seconds, and we want to keep enough datapoints so that they add up to 14 days of data.
example
![Page 17: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/17.jpg)
Helps is achieving• replication• sharding
Relay Methods • rules• consistent-‐hashing
carbon-relay.py
![Page 18: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/18.jpg)
• Plaintext• Pickle• AMQP
Feeding in your data
![Page 19: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/19.jpg)
<metric path> <metric value> <metric timestamp>
PORT=2003SERVER=gr.plivo.comecho "plivo.OutboundCalls 4 `date +%s`" | nc ${SERVER} ${PORT};
plaintext protocol
![Page 20: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/20.jpg)
•Efficient than the plaintext protocol•Supports sending batches of metrics
payload = pickle.dumps(listOfMetricTuples)header = struct.pack("!L", len(payload))message = header + payload
pickle protocol
![Page 21: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/21.jpg)
•Fixed sized database•Fast, reliable storage of numeric data•Long term retention
Whisper
![Page 22: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/22.jpg)
•Collapse multiple data points•Uses avg by default•Other methods -‐ sum,last,max,min
Whisper : aggregation
![Page 23: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/23.jpg)
•Each datapoint is stored with timestamp•Archives overlap time periods•All time-‐slots take up space
Whisper : efficiency
![Page 24: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/24.jpg)
Graphite at Plivopip install graphite-web
![Page 25: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/25.jpg)
/render?
from=-24hours
&until=now
&target=api1.process_gunicorn._cpu_
&target=api2.process_gunicorn._cpu_
The Render URL API
![Page 26: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/26.jpg)
The Render URL API cpu usage metrics
![Page 27: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/27.jpg)
The Render URL API “json output anyone?”
/render?
from=-24hours
&until=now
&target=django.users._registered_
&target=django.users._webrtc_
&format=json
![Page 28: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/28.jpg)
Graphenedescription = { "Total Notifications": { source: "http://localhost:4567/", GaugeLabel: { parent: "#hero-‐one", title: "Notifications Served", type: "max" } }, "Poll Time": { source: "http://localhost:4567/", GaugeGadget: { parent: "#hero-‐one", title: "P1" } }, "<just an informative label>": { source: "<graphite graph url, add &format=json to querystring>", "<widget type>": { parent: "<which will be placed in this element>", title: "<title>" // ... many other view opts ... } }}
![Page 29: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/29.jpg)
![Page 30: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/30.jpg)
Summary
![Page 31: Graph Everything](https://reader034.fdocuments.in/reader034/viewer/2022051323/547be996b4af9f9b158b508a/html5/thumbnails/31.jpg)
Thanks!