C* Summit 2013: Optimizing the Public Cloud for Cost and Scalability with Cassandra - The MetricsHub...
-
Upload
planet-cassandra -
Category
Technology
-
view
910 -
download
0
description
Transcript of C* Summit 2013: Optimizing the Public Cloud for Cost and Scalability with Cassandra - The MetricsHub...
Optimizing the Public Cloud for Cost and Scalability with Cassandra
#CASSANDRA13
Charles LamannaSenior Development Lead @clamanna
Ricardo VillalobosSenior Cloud Architect@ricvilla
MetricsHubkeep services up and running for the lowest possible cost
#CASSANDRA13
Live Status
Cost Awareness
Alerts and Notifications
Actions and Scaling
$
#CASSANDRA13
#CASSANDRA13
#CASSANDRA13
growth2000+ customers in 6 months
#CASSANDRA13
10/18/2012 12/7/2012 1/26/2013 3/17/2013 5/6/2013 6/25/20130
500
1000
1500
2000
2500
Number of MetricsHub Customers
#CASSANDRA13
10/18/2012 12/7/2012 1/26/2013 3/17/2013 5/6/2013 6/25/20130
1000
2000
3000
4000
5000
6000
7000
8000
9000
Number of VMs Monitored by MetricsHub
#CASSANDRA13
10/18/2012 12/7/2012 1/26/2013 3/17/2013 5/6/2013 6/25/20130
1
2
3
4
5
6
7
8 Number of Metricshub Employees
#CASSANDRA13
storing data200M data points per hour
#CASSANDRA13
Planning for huge data ingestion rates• MetricsHub requires high scale, real-time data: • 1,000 data points per minute per VM• 12 data points per endpoint per minute• 500+ data points per storage account per hour
• Need to aggregate, analyze and take actions based on this data stream (in near real-time)
• Must be cheap, scalable and reliable
#CASSANDRA13
Looked at Redis…• Perform aggregation in memory (using INCR and other
native operations)
• Flush aggregate data from Redis to persistent storage at a regular interval
• Is fast, powerful and a good OSS community
#CASSANDRA13
… but it was fragile, and expensive for this use case
• RAM/Memory in the public cloud is *expensive* (but storage is *cheap*)
• Flushing the data requires complex coordination
• If we did not flush quickly enough – out of memory!
#CASSANDRA13
Looked at SQL…• Create tables for different time windows and
granularities
• Roll over from table-to-table (and drop entire tables when the data expires)
• Update in place (for counters, min, max, etc.) in a reliable way
#CASSANDRA13
… but SQL did not fit• Higher write than read volume pushed boundaries of
the servers
• Requires complex sharding after just a few dozen new customers
• Is possible, but not worth the operational cost
#CASSANDRA13
Then we tried Cassandra (and never went back)
• Scales fluidly • Grows horizontally – double the nodes, double capacity• Add / remove capacity / nodes with no downtime
• Highly available• No single point of failure• Replication factor (i.e. hot copies) is just a config switch
#CASSANDRA13
… and by the way
• Little-to-none operations cost• New nodes take minutes to setup• Nodes just keep running for months on end
• “Aggregate on write” – no jobs required!• Atomic distributed counters make it easy to do
aggregates on write
• …and a nice kicker: has *great* perf / COGS in Azure
#CASSANDRA13
architecture68 virtual machines (PAAS and IAAS)
#CASSANDRA13
#CASSANDRA13
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Web API Web Role
(8 instances)
End User Web Browsers
Monitored Customer Resources
(e.g. websites; SQL databases)
Monitored Virtual Machines
Endpoints Replicated datain multiple
datacenters
ClientsPaaS
IaaS
Services
Avoiding state
• Application logic / code all lives on stateless machines
• Keeps it simple: decreases human operations cost
• Use Azure PAAS offerings (Web and Worker roles)
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Web API Web Role
(8 instances)
Endpoints Replicated datain multiple
datacenters
#CASSANDRA13
PaaS
Windows Azure Cloud Services (PAAS)
• Scale horizontally (grew from 1 to 30+ instances)
• Managed by the platform (patched; coordinated recycling; failover; etc.)
• 1 click deployment from Visual Studio (with automatic load balancer swaps)
Web Role Worker Role
#CASSANDRA13
Table Storage
SQL Database
Blob storage
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Web API Web Role
(8 instances)
Endpoints Replicated datain multiple
datacenters
Jobs Worker RoleRuns recurring tasks to pull, generate and analyze data
Jobs are synchronized and scheduled using Windows Azure Tables and Queues
Jobs Worker Role (24 instances)
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Endpoints Replicated datain multiple
datacenters
Web API Role
RESTful endpoint for saving and reading custom metrics.
Highly concurrent, secure & scalable.
Web API Web Role
(8 instances)
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Cassandra VM Cluster
(32 XL instances)
Web API Web Role
(8 instances)
Endpoints Replicated datain multiple
datacenters
Portal Web Role
Interface for our customers – shows trends, charts and issues. Portal Web
Role (3 instances)
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Web API Web Role
(8 instances)
Endpoints Replicated datain multiple
datacenters
Maintains all state for metrics / time series data.
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Cassandra Cluster
Windows Azure Virtual Machines (IaaS)
Management Portal
Scripting (Windows, Linux and Mac)
REST API
Starting Select Image and VM Size New Disk Persisted in Storage
Boot VM from New Disk
32 nodes, 8 “pods” of 4 nodes
……..
……….
Exposed via a single endpoint (port 9160)
9160
9160 Exposed via a single endpoint (port 9161)
Exposing the pods• Each pod of 4 nodes
has a single load balanced endpoint
• Clients (on our stateless roles) treats the endpoint as a pool
• Blacklists and skips an endpoint if it starts producing a lot of errors
#CASSANDRA13
Where does the data go?
• Data files are on 8 mounted network backed disks (*not* ephemeral disks)
• Data disks are geo-replicated (3 copies local; 1 remote) for “free” DR
• Azure data disks offer great throughput (VMs end up CPU bound)
#CASSANDRA13
Our Column Families (CQL 3)
CREATE TABLE oneminute (
rk text, ck text, cnt counter, sum counter, PRIMARY KEY (rk, ck)
);
#CASSANDRA13
Updating values…Realtime “average” values at any granularity, for any time window
updateoneminute/tenminute/oneday
setsum = sum + {sample_value},cnt = cnt + 1
where rk = '{customer_name}' and ck = '{metric_path}'
#CASSANDRA13
Reading values…
*ONE* round trip to fetch a metric over time (e.g. CPU over past week)
select * from oneminutewhere rk = ‘{customer_name}' and ck < '{metric_path_start}' and ck >= '{metric_path_end}‘order by ck desc;
#CASSANDRA13
What’s next?
• Windows Azure Virtual Networks to connect / secure all of our resources
(PAAS + IAAS + Services)• Expand Cassandra cluster across
datacenter boundaries for improved availability• Integrate with more off-the-shelf Azure
components to reduce operational overhead
#CASSANDRA13
#CASSANDRA13
Global Physical Infrastructureservers/network/datacenters
automated
elastic
managed resources
usage based
REST API + OTHER SERVICES
compute data management networking
SQL database
noSQL databasewebsites blob connect
virtual network
traffic manager
cloud services VMs
#CASSANDRA13