Scaling MySQL in Amazon Web Services
-
Upload
laine-campbell -
Category
Technology
-
view
862 -
download
0
description
Transcript of Scaling MySQL in Amazon Web Services
![Page 1: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/1.jpg)
Scaling MySQL in AWS
All Your Base Conf, Oxford, UK
Laine Campbell, Co-Founder
October 17th, 2014
![Page 2: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/2.jpg)
Who am I?
DB Architect, Entrepreneur and super hero…Good hair, not overly clever, female pronouns
![Page 3: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/3.jpg)
Who am I?
Humor - Self effacing, self aggrandizing, dry…Cajun, rogueish, and I destroy everything I touch...
![Page 4: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/4.jpg)
Who am I?
When in doubt, laugh
![Page 5: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/5.jpg)
Who am I?
Happy Belated Ada Lovelace Day
![Page 6: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/6.jpg)
Agenda
● Amazon options for implementation● MySQL scaling patterns● Resiliency● Round it out● War Stories
![Page 7: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/7.jpg)
RDS and EC2/MySQL
A love story...
![Page 8: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/8.jpg)
Amazon RDS (DBaaS)
Basic Operations Managed
Ease of Deployment
Supports Scaling via Replication
Resilient via Replication, EBS RAID, Multi-AZ, Multi-Region
![Page 9: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/9.jpg)
What is Multi-AZ?
Automatic Failover, spread across availability zones
Significantly reduces impact of operations such as:● Backups● Replica Builds● Patching
![Page 10: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/10.jpg)
Managed Operations
● Backups and Recovery ● Provisioning ● Patching● Auto Failover● Replication
![Page 11: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/11.jpg)
What does it cost?
● Instance tax: 35% - 37%● Provisioned IOPS: 15% ● Lack of transparency - can increase downtime
![Page 12: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/12.jpg)
What does it cost?
● Multi AZ Master - db.r3.8xlarge, 1 yr reserved○ $17,312
● 3 replicas - b.r3.8xlarge, 1 yr reserved○ $25,968
● Provisioned IOPS - general purpose (new)○ 1 TB dataset = $6,144 (5 copies)
● Total Cost RDS = $49,424● Off of RDS = $34,726
![Page 13: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/13.jpg)
Cost Thoughts
● RDS costs $14,698/yr ○ DBA costs $144K/yr, = $108 hour (time off, productivity,
retention/churn)○ Equals 136 hours of DBA time (3.5 weeks)
● Automating via EC2 is a one time job, RDS tax is ALWAYS○ 5 clusters costs you 680 hours of DBA time (17 weeks)○ 5 clusters, 3 years = 2,040 hours, or 51 weeks○ What can your DBA do with an extra 51 weeks?
![Page 14: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/14.jpg)
Other Costs
● Lock-in:○ In 5.6, you can replicate out, making this moot.○ You have to automate once out, you could have spent
this in the beginning.● Lack of Visibility:
○ Dtrace, TCPDump, Top, VMStat, etc…● Lack of Control:
○ Data Security, Shared Environments, Backups????○ Restarts due to exploits, etc...
![Page 15: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/15.jpg)
Amazon EC2, Roll Your OwnBuild your own automation:● Provisioning, with replication● Configuration management● Backup and recovery● Instance registration
Other DBaaS Options such as Tesora Trove● Community, free● Enterprise starts at $25,000 for 50 instances● don’t forget RDS, 5 clusters, 1 years = $73,440
![Page 16: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/16.jpg)
Choosing RDS vs EC2
![Page 17: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/17.jpg)
Why use RDS?
● Legacy apps that cannot use 5.6, and can accept < 99.65% SLAs
● Low volume/traffic applications for companies who choose to not have their own operations expertise
![Page 18: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/18.jpg)
Why use EC2?
● Want MariaDB or XtraDB Variants● Want more flexibility in multi-region setups (before
5.6)● Want portability to other clouds● High performance and scale needs that require
access to the OS, and to the FULL DB instance
![Page 19: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/19.jpg)
Storage Options in AWSType RDS EC2 Persistent Max
IOPSMax Throughput
Cost
Pure SSD No Yes No 100,000 390 MBps Free w/Instance Cost
EBS General Purpose SSD
Yes Yes Yes 3,000 128 MBps $.10/GB/month
EBS PIOPS SSD Yes Yes Yes 4,000 128 MBps $.125/GB/month + $.065/PIOPS/month
EBS Magnetic Yes Yes Yes 40-200 40-90 MBps $.05/GB/month + $.05/million IOPS
![Page 20: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/20.jpg)
Scaling MySQL at AWS
![Page 21: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/21.jpg)
My Definition of Scaling
● Capacity is elastic and automated● Performance stays reasonably consistent● Availability scales● Resiliency scales● Operational visibility scales● Backup and recovery scales
![Page 22: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/22.jpg)
MySQL Workload Scaling
● Break out workloads to their own clusters○ To facilitate sharding large data-sets horizontally○ To segregate specific workload characteristics
● Evaluate each workload’s read/write needsTotal dataset size and growthData change delta (updates/deletes)
![Page 23: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/23.jpg)
MySQL Workload Scaling
User Login/Profile1 TB (1MM User)20,000 iops peak10% write90% read
User Content5 TB500,000 iops peak1% write99% read
Site Metadata5 GB500 iops peak25% write75% readShard Candidate
Shard Candidate
![Page 24: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/24.jpg)
MySQL Workload Scaling
Determine sharding size based on constraints○ AWS Write IO○ Replication Limits○ Tolerance for large numbers of systems○ Budget
![Page 25: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/25.jpg)
MySQL Workload Scaling
Sharding Topology:● Schema:Shard relationship 1:1● Instance:Schema relationship 1:N● Host:Instance relationship 1:1
Host 1
Instance 1
Shard 1 Shard 2
![Page 26: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/26.jpg)
MySQL Workload Scaling
User Profile Data - 10 shards, hashable○ Per Shard: 2,000 (500MB) iops, 200 wps (50MB), 100 GB storage
■ Two replication threads/shards per cluster○ SSD General Purpose EBS, 2 300 GB Volumes Striped (
■ 1,800 IOPS, 6000 burst, 800 MBps○ 3,600 rps (900 MBps) requires 2 replicas (500 MBps), +1 for
redundancy○ 1 master, 1 failover, 3 replicas = 25 hosts, r3.2x lg
■ Memory = 61 GB > active dataset■ Assumes read/write splitting
○ Total Cost = $48,600 instances + $18,000 storage = $66,600
![Page 27: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/27.jpg)
![Page 28: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/28.jpg)
MySQL Workload Scaling
Keep Schema:Shard relationship 1:1Change Schema:Instance relationship from 1:N to 1:1Change Instance:Host relationship from 1:1 to 1:N
Host 1
Instance 1
Shard 1
Instance 2
Shard 2
![Page 29: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/29.jpg)
![Page 30: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/30.jpg)
MySQL Workload Scaling
Summary○ Final Constraint is Write IOPS○ Sharding eliminates Constraint○ There are ways to reduce those reads:
■ Caching○ There are ways to reduce those writes:
■ Throttling concurrency with queuing and loose coupling to keep write IO down
■ Compression! Native or application based○ Moving storage to ephemeral SSD saves $$$$
■ If you truly can leverage backup and recovery with small datasets
![Page 31: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/31.jpg)
MySQL Workload Scaling
Master
Replica Replica
Failover
Shard 1
Master
Replica Replica
Failover
Shard 2
![Page 32: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/32.jpg)
Resiliency Layers
● Sharding○ Shard N = y% of traffic, where y = 1/Shards*100○ Aka with 64 shards, one shard lost = 1.56%
● EBS Snapshots○ Rapid redeployment of failed nodes (kill, rebuild)
![Page 33: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/33.jpg)
Type of Change EC2 RDS Master(Non Multi-AZ)
RDS Master(Multi-AZ)
RDS Replica
Instance resize up/down
Rolling Migrations Moderate Downtime
Minimal Downtime
Moderate Downtime (take out of service)
EBS <-> PIOPS Severe Perf impact Severe Perf impact Minor Perf impact Severe Perf Impact (take out of service)
PIOPS Amount Change
Minor Perf impact Minor Perf impact Minor Perf impact Severe Perf Impact (take out of service)
Disk Space Change (add)
Severe Perf impact Severe Perf impact Minor Perf impact Severe Perf Impact (take out of service)
Disk Space Change (reduce)
Rolling Migrations Severe Downtime(promote from replica)
Minimal Downtime
Moderate Downtime (take out of service)
![Page 34: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/34.jpg)
Power Up - Resiliency via Geography
![Page 35: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/35.jpg)
MySQL Workload Scaling
S1 Master
Replica Replica
US East 1
S2 Failover
During failover, spawn replicas
US West 1
AZ 2 AZ 3
AZ 1
Future Replica
Future Replica
AZ 2 AZ 3
Nginx Nginx
![Page 36: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/36.jpg)
Cluster Management and Failover
● Roll your own…. ○ Config Mgmt○ Automation Scripting○ HAProxy
● Spend some dough○ RDS○ Continuent Tungsten○ Scalearc
● Bleeding edge but clotting○ Galera, via MariaDB or XtraDB Cluster
● Bleeding edge and fresh○ Openstack, Trove and Tesora
![Page 37: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/37.jpg)
Rounding it Out: Operational Visibility
● Monitoring and Alerting: Sensu (not Nagios)● Time Series Trending: Graphite or OpenTSDB● Graphing of Data: Grafana● Log Aggregation and Management: Logstash or Splunk● Application Monitoring: New Relic or AppDynamics
![Page 38: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/38.jpg)
Rounding it Out: Backup and Recovery
● Sharding keeps systems small and agile○ Snapshots for tactical kill and build○ S3 for longer-term○ Glacier forever○ Offsite for Legal
![Page 39: Scaling MySQL in Amazon Web Services](https://reader033.fdocuments.in/reader033/viewer/2022061204/547e86e0b47959c0508b4b88/html5/thumbnails/39.jpg)
Questions and Follow up
○ [email protected]○ www.pythian.com○ www.linkedin.com/lainecampbell○ www.adafoundation.org