Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 1

0-1 Million in 46 DaysScaling a Facebook Application in Rails

Ikai Lan

Linkedin

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 2

Hi! I’m Ikai Lan

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 3

What this talk is not

One-size-fits all solution to scalingproblems

Example source code

An in-depth discussion of Ruby onRails

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 4

What this talk is

A look into the challenges we facedwith scaling a rapid growth webapplication

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 5

This is a detective story.

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 6

Rather than telling you bestpractices

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 7

We’ll talk about how we came todefine them.

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 8

Bumper Sticker?

Investigation of the Facebookplatform

Investigation of Ruby on Rails as aserious development platform

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 9

Reid Hoffman’s wager

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 10

1 million users in 45 days!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 11

Personal Badging

Create and display badges

Share badges

Find badges

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 12

= Bumper Sticker!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 13

Success!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 14

… 1 million in 46 days

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 15

Trouble in Paradise

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 16

Throw hardware at it!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 17

More hardware!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 18

This is not working.

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 19

Current state of affairs

7 million adds

20 million pageviews

3 million timeouts a day

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 20

Strategy

Analyze MySQL slow queries log

Application log

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 21

D’oh! No logging!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 22

Logging -> Syslog

Rails does not buffer writes tofilesystem

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 23

Clue 1: Slaves ran several hoursbehind Master

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 24

Clue 2: Updates to add count

One by one updates

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 25

Counter Cache

SQL Updates not batched

Very inefficient!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 26

Result: less slow queries

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 27

Lesson 1: Buffereverything!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 28

Still had lots of timeouts!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 29

Clue 3: 80% of traffic to staticcontent

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 30

Next attempt: edge caching!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 31

Edge caching using fb:ref

Facebook mechanism for storingdata

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 32

Fb:ref on the load balancer!

F5 BigIP set to conditionally respondto static URIs with fb:ref

Background process to constantlyupdate fb:ref in Facebook cache

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 33

Result: Static pages =perfect

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 34

Lesson 2: Edge cachewhen possible

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 35

But we were still servingmillions of timeouts!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 36

Clue 4: Most CPU time acceptingstickers

Request taking up to 10 secondsduring peak

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 37

Slow database writes

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 38

1st attempt: Background processwrites

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 39

Result: Timeouts droppedsignificantly!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 40

Success … ?

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 41

No.

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 42

Application would notseem responsive to useractions and seem out ofsync.

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 43

2nd attempt: Two stepwrite

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 44

Step 1: Synchronouswrite to MemCache

Step 2: Asynchronous write to

database (queue)

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 45

Result: Success!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 46

Lesson 3: Useasynchronous processingwhen possible

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 47

Lesson 3.1: Keeping datain sync is not trivial!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 48

Lesson 3.2: Lack ofconcurrency in Railspresents problem earlier

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 49

Lesson 3.3: Very fewproblems can be reducedto a linear equation

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 50

QED: Rails can scale.

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 51

Latest stats

16 million installs

30 million pageviews a day

Average response time of <80ms

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 52

Scaling has a price!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 53

Complexity

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 54

No new functionality!

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 55

Key takeaways

Make long running tasksasynchronous

Use edge caching

Buffer long operations when possible

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 56

The bigger picture

Language choice is less importantthan architecture

Be creative – brute force doesn’twork

Be ready to be wrong

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 57

Questions?

Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin

Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 58

Thanks for coming!

ilan@linkedin.com

http://www.linkedin.com/in/ikailan