Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will...
Transcript of Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will...
![Page 1: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/1.jpg)
Scaling
Marty WeinerKrypton
Yashh NelapatiGotham City
Friday, July 27, 12
![Page 2: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/2.jpg)
Pinterest is . . .An online pinboard to organize and
share what inspires you.
Scaling Pinterest
Friday, July 27, 12
![Page 3: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/3.jpg)
Friday, July 27, 12
![Page 4: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/4.jpg)
Friday, July 27, 12
![Page 5: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/5.jpg)
Friday, July 27, 12
![Page 6: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/6.jpg)
Relationships
Scaling Pinterest
Marty WeinerGrayskull, Eternia
Friday, July 27, 12
![Page 7: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/7.jpg)
Relationships
Scaling Pinterest
Marty WeinerGrayskull, Eternia
Yashh NelapatiGotham City
Friday, July 27, 12
![Page 8: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/8.jpg)
Mar 2010 Jan 2011 Jan 2012
Scaling PinterestScaling Pinterest
Page Views / Day
Mar 2010 Jan 2011 Jan 2012 May 2012
Friday, July 27, 12
![Page 9: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/9.jpg)
Mar 2010 Jan 2011 Jan 2012
Scaling PinterestScaling Pinterest
· RackSpace
· 1 small Web Engine
· 1 small MySQL DB
· 1 Engineer
Page Views / Day
Mar 2010 Jan 2011 Jan 2012 May 2012
Friday, July 27, 12
![Page 10: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/10.jpg)
Scaling Pinterest
Page Views / Day
Mar 2010 Jan 2011 Jan 2012 May 2012
Friday, July 27, 12
![Page 11: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/11.jpg)
Scaling Pinterest
Page Views / Day
Mar 2010 Jan 2011 Jan 2012 May 2012
· Amazon EC2 + S3 + CloudFront
· 1 NGinX, 4 Web Engines
· 1 MySQL DB + 1 Read Slave
· 1 Task Queue + 2 Task Processors
· 1 MongoDB
· 2 Engineers
Friday, July 27, 12
![Page 12: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/12.jpg)
Mar 2010 Jan 2011 Jan 2012
Scaling PinterestScaling Pinterest
Page Views / Day
Mar 2010 Jan 2011 Jan 2012 May 2012
Friday, July 27, 12
![Page 13: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/13.jpg)
Mar 2010 Jan 2011 Jan 2012
Scaling PinterestScaling Pinterest
Page Views / Day
Mar 2010 Jan 2011 Jan 2012 May 2012
· Amazon EC2 + S3 + CloudFront· 2 NGinX, 16 Web Engines + 2 API Engines· 5 Functionally Sharded MySQL DB + 9 read slaves· 4 Cassandra Nodes· 15 Membase Nodes (3 separate clusters)· 8 Memcache Nodes· 10 Redis Nodes· 3 Task Routers + 4 Task Processors· 4 Elastic Search Nodes· 3 Mongo Clusters· 3 Engineers
Friday, July 27, 12
![Page 14: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/14.jpg)
Lesson Learned #1It will fail. Keep it simple.
Scaling Pinterest
Friday, July 27, 12
![Page 15: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/15.jpg)
Page Views / Day
Scaling Pinterest
Mar 2010 Jan 2011 Jan 2012 May 2012
Friday, July 27, 12
![Page 16: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/16.jpg)
Page Views / Day
Scaling Pinterest
Mar 2010 Jan 2011 Jan 2012 May 2012
· Amazon EC2 + S3 + Akamai, ELB
· 90 Web Engines + 50 API Engines
· 66 MySQL DBs (m1.xlarge) + 1 slave each
· 59 Redis Instances
· 51 Memcache Instances
· 1 Redis Task Manager + 25 Task Processors
· Sharded Solr
· 6 Engineers
Friday, July 27, 12
![Page 17: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/17.jpg)
Page Views / Day
Scaling Pinterest
Mar 2010 Jan 2011 Jan 2012 May 2012
Friday, July 27, 12
![Page 18: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/18.jpg)
Page Views / Day
Scaling Pinterest
Mar 2010 Jan 2011 Jan 2012 May 2012
· Amazon EC2 + S3 + Edge Cast, ELB
· 135 Web Engines + 75 API Engines
· 80 MySQL DBs (m1.xlarge) + 1 slave each
· 110 Redis Instances
· 60 Memcache Instances
· 2 Redis Task Manager + 60 Task Processors
· Sharded Solr
· 25 Engineers
Friday, July 27, 12
![Page 19: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/19.jpg)
Why Amazon EC2/S3?
· Very good reliability, reporting, and support
· Very good peripherals, such as managed cache,
DB, load balancing, DNS, map reduce, and more...
· New instances ready in seconds
Scaling Pinterest
Friday, July 27, 12
![Page 20: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/20.jpg)
Why Amazon EC2/S3?
· Very good reliability, reporting, and support
· Very good peripherals, such as managed cache,
DB, load balancing, DNS, map reduce, and more...
· New instances ready in seconds
Scaling Pinterest
· Con: Limited choice
Friday, July 27, 12
![Page 21: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/21.jpg)
Why Amazon EC2/S3?
· Very good reliability, reporting, and support
· Very good peripherals, such as managed cache,
DB, load balancing, DNS, map reduce, and more...
· New instances ready in seconds
Scaling Pinterest
· Con: Limited choice· Pro: Limited choice
Friday, July 27, 12
![Page 22: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/22.jpg)
· Extremely mature
· Well known and well liked
· Rarely catastrophic loss of data
· Response time to request rate increases linearly
· Very good software support - XtraBackup, Innotop, Maatkit
· Solid active community
· Very good support from Percona
· Free
Scaling Pinterest
Why MySQL?
Friday, July 27, 12
![Page 23: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/23.jpg)
Why Memcache?
· Extremely mature
· Very good performance
· Well known and well liked
· Never crashes, and few failure modes
· Free
Scaling Pinterest
Friday, July 27, 12
![Page 24: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/24.jpg)
Why Redis?· Variety of convenient data structures
· Has persistence and replication
· Well known and well liked
· Consistently good performance
· Few failure modes
· Free
Scaling Pinterest
Friday, July 27, 12
![Page 25: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/25.jpg)
Clustering vs
Sharding
Scaling Pinterest
Friday, July 27, 12
![Page 26: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/26.jpg)
Scaling Pinterest
Clustering
Sharding
· Data distributed automatically
· Data can move
· Rebalances to distribute capacity
· Nodes communicate with each other
Friday, July 27, 12
![Page 27: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/27.jpg)
Scaling Pinterest
Clustering
Sharding
· Data distributed manually
· Data does not move
· Split data to distribute load
· Nodes are not aware of each other
Friday, July 27, 12
![Page 28: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/28.jpg)
Why Clustering?· Examples: Cassandra, MemBase, HBase, Riak
· Automatically scale your datastore
· Easy to set up
· Spatially distribute and colocate your data
· High availability
· Load balancing
· No single point of failure
Scaling Pinterest
Friday, July 27, 12
![Page 29: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/29.jpg)
Scaling Pinterest
What could possibly go wrong?
source: thereifixedit.com
Friday, July 27, 12
![Page 30: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/30.jpg)
Why Not Clustering?
· Still fairly young
· Fundamentally complicated
· Less community support
· Fewer engineers with working knowledge
· Difficult and scary upgrade mechanisms
· And, yes, there is a single point of failure. A BIG one.
Scaling Pinterest
Friday, July 27, 12
![Page 31: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/31.jpg)
Scaling Pinterest
Clustering Single Point of Failure
Friday, July 27, 12
![Page 32: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/32.jpg)
Scaling Pinterest
Clustering Single Point of Failure
Friday, July 27, 12
![Page 33: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/33.jpg)
Scaling Pinterest
Clustering Single Point of Failure
Friday, July 27, 12
![Page 34: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/34.jpg)
Scaling Pinterest
Clustering Single Point of Failure
Friday, July 27, 12
![Page 35: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/35.jpg)
Cluster ManagementAlgorithm
Scaling Pinterest
Clustering Single Point of Failure
Friday, July 27, 12
![Page 36: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/36.jpg)
Cluster Manager
· Same complex code replicated over all nodes
· Failure modes:
· Data rebalance breaks
· Data corruption across all nodes
· Improper balancing that cannot be fixed (easily)
· Data authority failure
Scaling Pinterest
Friday, July 27, 12
![Page 37: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/37.jpg)
Lesson Learned #2Clustering is scary.
Scaling Pinterest
Friday, July 27, 12
![Page 38: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/38.jpg)
Why Sharding?
· Can split your databases to add more capacity
· Spatially distribute and colocate your data
· High availability
· Load balancing
· Algorithm for placing data is very simple
· ID generation is simplistic
Scaling Pinterest
Friday, July 27, 12
![Page 39: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/39.jpg)
When to shard?
· Sharding makes schema design harder
· Solidify site design and backend architecture
· Remove all joins and complex queries, add cache
· Functionally shard as much as possible
· Still growing? Shard.
Scaling Pinterest
Friday, July 27, 12
![Page 40: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/40.jpg)
Our Transition1 DB + Foreign Keys + Joins
1 DB + Denormalized + Cache
Several functionally sharded DBs + Read slaves + Cache
1 DB + Read slaves + Cache
ID sharded DBs + Backup slaves + Cache
Scaling Pinterest
Friday, July 27, 12
![Page 41: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/41.jpg)
Watch out for...
Scaling Pinterest
· Cannot perform most JOINS
· No transaction capabilities
· Extra effort to maintain unique constraints
· Schema changes requires more planning
· Single report requires running same query on all
shards
Friday, July 27, 12
![Page 42: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/42.jpg)
How we sharded
Scaling Pinterest
Friday, July 27, 12
![Page 43: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/43.jpg)
Sharded Server Topology
Initially, 8 physical servers, each with 512 DBs
Scaling Pinterest
db00001db00002
.......db00512
db00513db00514
.......db01024
db03584db03585
.......db04096
db03072db03073
.......db03583
Friday, July 27, 12
![Page 44: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/44.jpg)
High Availability
Multi Master replicationScaling Pinterest
db00001db00002
.......db00512
db00513db00514
.......db01024
db03584db03585
.......db04096
db03072db03073
.......db03583
Friday, July 27, 12
![Page 45: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/45.jpg)
Increased load on DB?
To increase capacity, a server is replicated and the new replica becomes responsible for some DBs
Scaling Pinterest
db00001db00002
.......db00512
db00001db00002
.......db00256
db00257db00258
.......db00512
Friday, July 27, 12
![Page 46: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/46.jpg)
ID Structure
· A lookup data structure has physical server to shard
ID range (cached by each app server process)
· Shard ID denotes which shard
· Type denotes object type (e.g., pins)
· Local ID denotes position in table
Shard ID Local ID
64 bits
Scaling Pinterest
Type
Friday, July 27, 12
![Page 47: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/47.jpg)
Why not an ID service?
· It is a single point of failure
· Extra look up to compute a UUID
Scaling Pinterest
Friday, July 27, 12
![Page 48: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/48.jpg)
Lookup Structure
Scaling Pinterest
sharddb003a
{“sharddb001a”: ( 1, 512), “sharddb002b”: ( 513, 1024), “sharddb003a”: (1025, 1536), ... “sharddb008b”: (3585, 4096)}
DB01025 users
users
user_has_boards
boards
1 ser-data
2 ser-data
3 ser-data
Friday, July 27, 12
![Page 49: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/49.jpg)
· New users are randomly distributed across shards
· Boards, pins, etc. try to be collocated with user
· Local ID’s are assigned by auto-increment
· Enough ID space for 65536 shards, but only first
4096 opened initially. Can expand horizontally.
Scaling Pinterest
ID Structure
Friday, July 27, 12
![Page 50: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/50.jpg)
Objects and Mappings· Object tables (e.g., pin, board, user, comment)
· Local ID MySQL blob (JSON / Serialized thrift)
· Mapping tables (e.g., user has boards, pin has likes)
· Full ID Full ID (+ timestamp)
· Naming schema is noun_verb_noun
· Queries are PK or index lookups (no joins)
· Data DOES NOT MOVE
· All tables exist on all shards
· No schema changes required (index = new table)Scaling Pinterest
Friday, July 27, 12
![Page 51: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/51.jpg)
Loading a Page
· Rendering user profile
· Most of these calls will be a cache hit
· Omitting offset/limits and mapping sequence id sort
SELECT body FROM users WHERE id=<local_user_id>SELECT board_id FROM user_has_boards WHERE user_id=<user_id>SELECT body FROM boards WHERE id IN (<board_ids>)SELECT pin_id FROM board_has_pins WHERE board_id=<board_id>SELECT body FROM pins WHERE id IN (pin_ids)
Scaling Pinterest
Friday, July 27, 12
![Page 52: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/52.jpg)
Scripting· Must get old data into your shiny new shard
· 500M pins, 1.6B follower rows, etc
· Build a scripting farm
· Spawn more workers and complete the task faster
· Pyres - based on Github’s Resque queue
Scaling Pinterest
Friday, July 27, 12
![Page 53: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/53.jpg)
Caching· Redis lists to cache mappings
· lpush user:82363:pins 7233494
· lrange user:82363:pins 0 49
· Use Memcache to cache objects
· Shard caches based upon pools
· Better stats
· Easier to scale and manage
Scaling Pinterest
Friday, July 27, 12
![Page 54: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/54.jpg)
Caching with decorators
def user_get_many(self, user_ids):
cursor = get_mysql_conn().cursor
cursor.execute(“SELECT * FROM users WHERE id in (%s)”, user_ids)
return cursor.fetchall()
Scaling Pinterest
Friday, July 27, 12
![Page 55: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/55.jpg)
Caching with decorators
def user_get_many(self, user_ids):
cursor = get_mysql_conn().cursor
cursor.execute(“SELECT * FROM users WHERE id in (%s)”, user_ids)
return cursor.fetchall()
Scaling Pinterest
@mc_objects(USER_MC_CONN, format = “user:%d”, version=1, serialization=simplejson expire_time=0)
Friday, July 27, 12
![Page 56: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/56.jpg)
Caching with decorators
def get_board_pins(self, board_id, offset, limit):
cursor = get_mysql_conn().cursor
cursor.execute(“SELECT pin_id FROM bp WHERE id =%s OFFSET
%d LIMIT %d”, board_id, offset, limit)
return cursor.fetchall()
Scaling Pinterest
Friday, July 27, 12
![Page 57: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/57.jpg)
Caching with decorators
def get_board_pins(self, board_id, offset, limit):
cursor = get_mysql_conn().cursor
cursor.execute(“SELECT pin_id FROM bp WHERE id =%s OFFSET
%d LIMIT %d”, board_id, offset, limit)
return cursor.fetchall()
Scaling Pinterest
@paged_list(BOARD_REDIS_CONN, format = “b:p:%d”, version=1, expire_time=24*60*60)
Friday, July 27, 12
![Page 58: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/58.jpg)
Current problems
· Service Based Architecture
· Connection limits
· Isolation of functionality
· Isolation of access (security)
· Scaling the Team
Scaling Pinterest
Friday, July 27, 12
![Page 59: Scaling - Meetupfiles.meetup.com/107604/MySQL _Meetup_July_2012-Scaling...Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, July 27, 12 Page Views / Day Scaling](https://reader033.fdocuments.in/reader033/viewer/2022060310/5f0a97e27e708231d42c653a/html5/thumbnails/59.jpg)
Lesson Learned #3Keep it fun.
Scaling Pinterest
Friday, July 27, 12