LiveJournal: Behind The Scenes - USENIX ¢†â€™ mod_perl/php internal...

download LiveJournal: Behind The Scenes - USENIX ¢†â€™ mod_perl/php internal caching memory waste (address space

of 115

  • date post

    08-Oct-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of LiveJournal: Behind The Scenes - USENIX ¢†â€™ mod_perl/php internal...

  • http://danga.com/words/

    LiveJournal: Behind The Scenes Scaling Storytime

    June 2007 USENIX

    Brad Fitzpatrick brad@danga.com

    danga.com / livejournal.com / sixapart.com

    This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/1.0/ or send a letter to

    Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

    1

  • http://danga.com/words/

    The plan...

     Refer to previous presentations for more details...  http://danga.com/words/

     Questions anytime! Yell. Interrupt.  Part 0:

    − show where talk will end up  Part I:

    − What is LiveJournal? Quick history. − LJ’s scaling history

     Part II: − explain all our software, − explain all the moving parts

    2

  • http://danga.com/words/

    LiveJournal Backend: Today (Roughly.)

    User DB Cluster 1 uc1a uc1b

    User DB Cluster 2 uc2a uc2b

    User DB Cluster 3 uc3a uc3b

    User DB Cluster N ucNa ucNb

    Job Queues (xN) jqNa jqNb

    Memcached

    mc4

    mc3

    mc2

    mcN

    ...

    mc1

    mod_perl

    web4

    web3

    web2

    webN

    ...

    web1

    BIG-IP

    bigip2 bigip1 perlbal (httpd/proxy)

    proxy4

    proxy3

    proxy2

    proxy5

    proxy1

    Global Database

    slave1

    master_a master_b

    slave2 ... slave5

    MogileFS Database

    mog_a mog_b

    Mogile Trackers tracker3tracker1

    Mogile Storage Nodes

    ... sto2 sto8

    sto1

    net.

    djabberd djabberd djabberd

    gearmand gearmand1 gearmandN

    “workers” gearwrkN theschwkN

    slave1 slaveN 3

  • http://danga.com/words/

    LiveJournal Overview

     college hobby project, Apr 1999  4-in-1:

    − blogging − forums − social-networking (“friends”) − aggregator: “friends page”

    − “friends” can be external RSS/Atom  10M+ accounts  Open Source!

    − server, − infrastructure, − original clients,

    4

  • http://danga.com/words/

     memcached − distributed caching

     MogileFS − distributed filesystem

     Perlbal − HTTP load balancer, web

    server, swiss-army knife  gearman

    − LB/HA/coalescing low- latency function call “router”

     TheSchwartz − reliable, async job

    dispatch system

     djabberd − the super-extensible

    everything-is-a-plugin mod_perl/qpsmtpd/ Eclipse of XMPP/Jabber servers

     .....  OpenID

     federated identity protocol

    Stuff we've built... (all production, open source)

    5

  • http://danga.com/words/

    “Uh, why?”

     NIH? (Not Invented Here?)  Are we reinventing the wheel?

    6

  • http://danga.com/words/

    Yes.

     We build wheels. − ... when existing suck, − ... or don’t exist.

    7

  • http://danga.com/words/

    Yes.

     We build wheels. − ... when existing suck, − ... or don’t exist.

    7

  • http://danga.com/words/

    Yes.

     We build wheels. − ... when existing suck, − ... or don’t exist.

    7

  • http://danga.com/words/

    Yes.

     We build wheels. − ... when existing suck, − ... or don’t exist.

    (yes, arguably tires. sshh..)

    7

  • http://danga.com/words/

    Part I Quick Scaling History

    8

  • http://danga.com/words/

    Quick Scaling History

     1 server to hundreds...

     you can do all this with just 1 server! − then you’re ready for tons of servers, without pain − don’t repeat our scaling mistakes

    9

  • http://danga.com/words/

    Terminology

     Scaling: − NOT: “How fast?” − But: “When you add twice as many servers, are you

    twice as fast (or have twice the capacity)?”  Fast still matters,

    − 2x faster: 50 servers instead of 100...  that’s some good money

    − but that’s not what scaling is.

    10

  • http://danga.com/words/

    Terminology

     “Cluster” − varying definitions... basically: − making a bunch of computers work together for

    some purpose − what purpose?

     load balancing (LB),  high availablility (HA)

     Load Balancing?  High Availability?  Venn Diagram time!

    − I love Venn Diagrams

    11

  • http://danga.com/words/

    LB vs. HA

    Load Balancing High Availability

    12

  • http://danga.com/words/

    LB vs. HA

    Load Balancing

    High Availability

    http reverse proxy, wackamole,

    ...

    round-robin DNS, data partitioning, ....

    LVS heartbeat, cold/warm/hot spare,

    ...

    13

  • http://danga.com/words/

    Favorite Venn Diagram

    Times When I’m Truly Happy

    Times When I’m Wearing Pants

    14

  • http://danga.com/words/

    One Server

     Simple:

    mysql

    apache

    15

  • http://danga.com/words/

    Two Servers

    mysql

    apache

    16

  • http://danga.com/words/

    Two Servers - Problems

     Two single points of failure!  No hot or cold spares  Site gets slow again.

    − CPU-bound on web node − need more web nodes...

    17

  • http://danga.com/words/

    Four Servers

     3 webs, 1 db  Now we need to load-balance!

     LVS, mod_backhand, whackamole, BIG-IP, Alteon, pound, Perlbal, etc, etc..

    − ...

    18

  • http://danga.com/words/

    Four Servers - Problems

     Now I/O bound...  ... how to use another database?

    19

  • http://danga.com/words/

    Five Servers introducing MySQL replication

     We buy a new DB  MySQL replication  Writes to DB (master)  Reads from both

    20

  • http://danga.com/words/

    More Servers

    Chaos!

    21

  • http://danga.com/words/

    Where we're at....

    mod_perl

    web4

    web3

    web2

    web12

    ...

    web1

    BIG-IP

    bigip2 bigip1

    mod_proxy

    proxy3

    proxy2

    proxy1

    Global Database

    slave1 slave2 ... slave6

    master

    net.

    22

  • http://danga.com/words/

    Problems with Architecture or,

    “This don't scale...”

     DB master is SPOF  Adding slaves doesn't scale

    well... − only spreads reads, not writes!

    200 writes/s 200 write/s

    500 reads/s 250 reads/s

    200 write/s

    250 reads/s

    23

  • http://danga.com/words/

    Eventually...

     databases eventual only writing

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    400 write/s

    3 reads/s

    400 write/s

    3 r/s

    24

  • http://danga.com/words/

    Spreading Writes

     Our database machines already did RAID  We did backups  So why put user data on 6+ slave machines?

    (~12+ disks) − overkill redundancy − wasting time writing everywhere!

    25

  • http://danga.com/words/

    Partition your data!

     Spread your databases out, into “roles” − roles that you never need to join between

     different users  or accept you'll have to join in app

     Each user assigned to a numbered HA cluster  Each cluster has multiple machines

    − writes self-contained in cluster (writing to 2-3 machines, not 6)

    26

  • http://danga.com/words/

    User Clusters

    27

  • http://danga.com/words/

    User Clusters

    SELECT userid, clusterid FROM user WHERE user='bob'

    27

  • http://danga.com/words/

    User Clusters

    SELECT userid, clusterid FROM user WHERE user='bob'

    userid: 839 clusterid: 2

    27

  • http://danga.com/words/

    User Clusters

    SELECT userid, clusterid FROM user WHERE user='bob'

    userid: 839 clusterid: 2

    SELECT .... FROM ... WHERE userid=839 ...

    27

  • http://danga.com/words/

    User Clusters

    SELECT userid, clusterid FROM user WHERE user='bob'

    userid: 839 clusterid: 2

    SELECT .... FROM ... WHERE userid=839 ...

    OMG i like totally hate my parents they just dont understand me and i h8 the world omg lol rofl *! :^- ^^;

    add me as a friend!!!

    27

  • http://danga.com/words/

    Details

     per-user numberspaces − don't use AUTO_INCREMENT − PRIMARY KEY (user_i