Post on 05-Dec-2014
description
Monitor all the Things!
KEVIN MCGUIRE - DIRECTOR OF ENGINEERING, PLATFORMOCTOBER 25/2013
Wednesday, November 6, 13
The evolution of monitoring
Wednesday, November 6, 13
The evolution of monitoring
Wednesday, November 6, 13
The evolution of monitoring
Wednesday, November 6, 13
The evolution of monitoring
Wednesday, November 6, 13
Your end user experience depends on more
Wednesday, November 6, 13
A lot more
Wednesday, November 6, 13
You can’t manage what you can’t monitor
Wednesday, November 6, 13
You can’t manage what you can’t monitor
Wednesday, November 6, 13
You can’t manage what you can’t monitor
Wednesday, November 6, 13
Solution?
Wednesday, November 6, 13
MONITOR ALL THE THINGS!
Solution?
http://hyperboleandahalf.blogspot.com/
Wednesday, November 6, 13
Platform gives complete visibility
Wednesday, November 6, 13
Platform gives complete visibility
Wednesday, November 6, 13
Platform gives complete visibility
Wednesday, November 6, 13
Plugins for, well, everything!68 69 published plugins29 different publishersToo big to fit in presentation!
Wednesday, November 6, 13
We got databases
Wednesday, November 6, 13
We got yer NoSQL stores
Wednesday, November 6, 13
Yup, caching too
Wednesday, November 6, 13
Queuing, web server, load blance, ...
Wednesday, November 6, 13
And my personal favorite:
Wednesday, November 6, 13
And my personal favorite:
You go Yuri Yasiyarov, wherever you are!
Wednesday, November 6, 13
Integrated, 1st class experience
Wednesday, November 6, 13
Integrated, 1st class experience
Wednesday, November 6, 13
Integrated, 1st class experience
Wednesday, November 6, 13
Integrated, 1st class experience
Wednesday, November 6, 13
Integrated, 1st class experience
Wednesday, November 6, 13
Customized chartsMySQL plugin has MySQL’y stuff
Wednesday, November 6, 13
Install is a snapEasy as...
1
Wednesday, November 6, 13
Install is a snapEasy as...
1 2[ {
"name" : "Localhost","host" : "localhost","user" : "USER_NAME_HERE","passwd" : "USER_PASSWD_HERE"
},]
licenseKey=blahblahblah12345
Wednesday, November 6, 13
Install is a snapEasy as...
1 2 3[ {
"name" : "Localhost","host" : "localhost","user" : "USER_NAME_HERE","passwd" : "USER_PASSWD_HERE"
},]
java -jar newrelic_mysql_plugin.jar
licenseKey=blahblahblah12345
Wednesday, November 6, 13
Independent runtime
+ GUID +
RPM
MySQL UI Memcached UI
VarnishUI
Infrastructure
MySQLServer 2
MemcachedServer
MySQLServer 1
Platform Agents
VarnishAgent
Memcached Agent
MySQLAgent...
VarnishServer
...
Externally monitored
New Relic MetricStore
Wednesday, November 6, 13
SaaS
+ GUID +
Customer BData
Customer BData
Customer BData
RPM
SaaS UI
Memcached UI VarnishUI
SaaS Service
Customer B Customer XCustomer A
...HTTP API
New Relic MetricStore
SaaS ‘agent code’
Customer Xlicense key
Wednesday, November 6, 13
Don’t see one you like?
Built 9 prior to launch,now at 14! Built the Redis Plugin in a
couple hours, and used it to solve performance issues the same day
Built 6! Built 4! Built 3!
Wednesday, November 6, 13
Simple APIs + SDKs = Easy to write
Wednesday, November 6, 13
Simple APIs + SDKs = Easy to write
•SDKs do all the “heavy lifting”•Java (primary), Ruby•Runs polling cycle•Connecting to New Relic •Aggregation
Wednesday, November 6, 13
Simple APIs + SDKs = Easy to write
•SDKs do all the “heavy lifting”•Java (primary), Ruby•Runs polling cycle•Connecting to New Relic •Aggregation
•HTTP API•SaaS•Non Java/Ruby
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Unique identifier (GUID), agent version
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Query metrics from system
Unique identifier (GUID), agent version
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Query metrics from system
Unique identifier (GUID), agent version
Value
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Query metrics from system
Unique identifier (GUID), agent version
Identifier Value
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Query metrics from system
Unique identifier (GUID), agent version
Identifier Units Value
Wednesday, November 6, 13
Anatomy of a plugin agent(yes, it’s really that simple)public class MySQLAgent extends Agent { public MySQLAgent() { super("com.mycompany.mysql", "1.0.0"); }
private Map<String, Number> gatherMetrics() { Map<String, Number> metrics = MySQL.runSQL(getConnection(), "SHOW GLOBAL STATUS"); return metrics; }
public void pollCycle() { Map<String, Number> metrics = gatherMetrics();
reportMetric("network/bytes_reads", "bytes/second", metrics.get("bytes_reads")); reportMetric("network/bytes_writes", "bytes/second", metrics.get("bytes_writes")); reportMetric("queries/com_select", "selects/second", metrics.get("com_select")); reportMetric("queries/com_insert", "inserts/second", metrics.get("com_insert")); reportMetric("queries/slow_queries", "queries/second", metrics.get("slow_queries")); reportMetric("connections/connected", "connections", metrics.get("connections_connected")); reportMetric("innodb/buffer_pool_pages_flushed", "pages", metrics.get("innodb_buffer_pool_pages_flushed")); reportMetric("innodb/buffer_pool_pages_dirty", "pages", metrics.get("innodb_buffer_pool_pages_dirty")); // ... }}
Query metrics from system
Send to NR
Unique identifier (GUID), agent version
Identifier Units Value
Wednesday, November 6, 13
Easy to build UIs
Wednesday, November 6, 13
Custom DashboardsFast and easy UI creation
Wednesday, November 6, 13
Optionally publishLists in Plugin Central, makes available the UIs
Wednesday, November 6, 13
Install UI = run the agent
+ GUID +
RPM
MySQL UI Memcached UI
VarnishUI
Infrastructure
MySQLServer 2
MemcachedServer
MySQLServer 1
Platform Agents
VarnishAgent
Memcached Agent
MySQLAgent...
VarnishServer
...
Externally monitored
New Relic MetricStore
Wednesday, November 6, 13
Install UI = run the agent
+ GUID +
RPM
MySQL UI Memcached UI
VarnishUI
Infrastructure
MySQLServer 2
MemcachedServer
MySQLServer 1
Platform Agents
VarnishAgent
Memcached Agent
MySQLAgent...
VarnishServer
...
Externally monitored
New Relic MetricStore
Wednesday, November 6, 13
“But Kevin, how much will all this cost?”
Wednesday, November 6, 13
“But Kevin, how much will all this cost?”
Free!!Wednesday, November 6, 13
New! Plugin community reviewsHelp improve plugins, give feedback
Wednesday, November 6, 13
New! Private publishingDistribute within multiple accounts, closed Betas
Wednesday, November 6, 13
In progress: Deployment recipesSimplified configuration and deployment across your data center
Wednesday, November 6, 13
Get involved!
✓ Review plugins
✓ Contribute to the plugins • Many are open source• Write one!
✓ Contribute to the SDKs
✓ Help us with Chef• Try out our recipes• Try authoring recipes
Wednesday, November 6, 13
Thank you!
Wednesday, November 6, 13