Scaling the RailsHow a little quiz achieved Olympic Glory

Josh Schwartzman, Newzwag

Nov 19, 2008

Get Ready For:

• Database Enhancements & Gotchas

• Hardware Basics

• Load Testing

• Rails Backends & Web Servers

From Humble Beginnings:

30,000 concurrent users 9 million pageviews/hour

(on limited hardware)

[question]

[answer]

FindYour

BottlenecksMySQL

Analyze Your Logs

• Look for slowest requests

• Verbose Logging

• Detailed DB logging (a small script)Completed in 0.06763 (14 reqs/sec) | Rendering: 0.01377 (20%) | DB: 0.00587 (8%) 2.6k, 54 rows, 7 queries | 200 OK [http://url]

Database Enhancements

• Index all of the keys you are going to use to find your data

• Turn on debugging and check the slowest queries

• Use EXPLAIN on your SQL statements

(Reads)

(Some MySQL-specific enhancements)

• Minimize your # of queries

• Use Eager Loading

Database Enhancements(Reads)

• Only save the data that has changed

• Rails 2.1: 2x faster on some actions, due to:

• Partial Updates (automatic performance)

• Dirty Tracking

Database Enhancements(Writes)

• Check your model validations

Database Enhancements(Writes)

• Transactions w/InnoDB

Database Enhancements

*40x if you don’t use validations

• Bulk Writes with AR-Extensions

• 8-12x faster on InnoDB

• 8-40x faster on MyISAM*

• Master/Slave Replication

• Master DB for writes with replication to read slaves

• Distributes load to multiple machines

(Writes)

Useful Plugins

• ActiveRecord Extensions

• Bulk writes

• Dr. Nic’s magic_multi_connections

• Offload reads/writes to another DB

• Single-Write Multi-Read (SWMR)

• Our plugin to write to one DB & use multiple read slaves

Hardware

• Cache everything, so lots of RAM

• Make your reads/writes fast

• RAID helps:

RAID 5 - faster reads RAID 10 - faster writes

Hardware

• Minimize hits to your Rails Stack

• Cache images or use a fast web server

• Move all your routes.rb routing logic to dedicated hardware (load balancer)

Hardware

• Keep App & DB on separate servers

• Infrastructure specific to it’s purpose

• Easy to isolate & debug bottlenecks:

Graph from Cacti

Load Testing

• Test the path a user will take

• One Tool: JMeter

• Session and full hardware stack testing

• jmeter-server

• Distributed testing

• Test from any computer @ ~1500 threads per box

Load Testing

• You’ll always find new problems under load

• We discovered issues with lots of new users: checking login twice

• Simple queries (especially joins) became slow

• Tweaking how often we wrote scores to the database, and how many at a time

Load Testing• The type and number of web servers*

is critical. Quick comparison:

Mongrel•Stable•RAM & CPU heavy•Slowest per action

Ebb•Some bugs•RAM & CPU light•Fastest per action (uses C for heavy lifting)

Thin•Almost as stable as Mongrel (it uses Mongrel’s parser)•Lightest hardware load•Almost as fast as Ebb

*Yeah, you still need a web server like Lighttpd or Nginx to handle your static files and proxy to your Rails backends

Load Testing#

of q

uest

ions

1 2 3 4 5 6 7 8 9 10minutes

Gruff graphs from our custom user actions

Using 18 Mongrels

Load Testing

949 questions / minute: 40% faster than mongrel!

1 2 3 4 5 6 7 8 9 10

# o

f que

stio

ns

minutes

Using 18 Ebbs

When You Need a Little Help

• After all of that performance work...

• Servers were crashing spontaneously

• Nothing was apparent in the logs

• Sleepless nights...

We Found Stability in GodSimple monitoring for Rails apps:

What Was the Problem?

• Post requests on older browsers (Firefox 1.5, Navigator 7)

• Killed our backends, and as each failed, it passed on the request to another server

• Solution: switch from Nginx to Lighttpd

A Summary

• Achieved by adding plugins, custom Memcached methods, optimizing the DB, changing to ebbs, and numerous other code tweaks (and a slight hardware upgrade)

a 25x performance gain

• 200 (scaling) 5000 simultaneous users

Any Questions:

Josh SchwartzmanLead Developer, Newzwag

josh@newzwag.com

Resources•Active Record Extensions: http://www.continuousthinking.com/are/activerecord-extensions-0-0-5

•Detailed DB Logging: http://gurge.com/blog/2006/11/09/rails-sql-logging-improvements/

•Ebb: http://ebb.rubyforge.org/

•God (Monitoring): http://god.rubyforge.org/

•Gruff Graphs: http://nubyonrails.com/pages/gruff

•Jmeter: http://jakarta.apache.org/jmeter/

•Magic Multi Connections: http://magicmodels.rubyforge.org/magic_multi_connections/

•Seattle.rb’s Memcache Client: http://github.com/fiveruns/memcache-client/tree/master

•Thin: http://code.macournoyer.com/thin/