Games For The Masses Scaling Rails To The Extreme

GAMES FOR THE MASSESSCALING RAILS TO THE EXTREME

EuRuKo 2011 in Berlin

Who is that guy?

Jesper Richter-‐Reichhelm

@jrirei

Berlin, Germany

Head of Engineering @ wooga

Wooga develops social games

Wooga has dedicated game teams

PHPSQLCloud

RubySQL + Redis

Cloud

RubyRedis

Bare Metal

Both games are similar in their setup

Flash client Ruby backend

Both games have similar problems

High traffic

• 200+ million h=p requests / day

High DB throughput

• 100.000 DB operaDons / second

Hardest problems to solve

• DB migraDons, DB failover, DB performance, ...

What’s best?

Big cloud vs. bare metal

Hybrid (SQL + Redis) vs. Redis only

In-‐memory vs. DB hierarchy

Conclusion: So what’s best?


Big cloud: Amazon EC2

• new server in 5 minutes

• automated deployment

• load based setup


Big cloud: Amazon EC2

• new server in 5 minutes


• load based setup

Bare metal: Hetzner

• new server in 2-‐14 days


What kind of app server are available?

XL CPU instance

• 8 virtual cores

• 8 GB RAM

Bare metal servers are more powerful

XL CPU instance

• 8 virtual cores

• 8 GB RAM

EQ10 server

• 12 “processing threads”

• 24 GB RAM

Bare metal servers are more powerful

XL CPU instance

• 8 virtual cores

• 8 GB RAM

=> 390,-‐ € / month

EQ10 server

• 12 “processing threads”

• 24 GB RAM

=> 200,-‐ € / month

And how is the network?

min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms

Want a fast network? Get your own!

min ping: 0.2 -‐ 0.3 ms

max ping: 7.8 -‐ 59 ms

avg ping: 0.8 -‐ 4.4 ms

min ping: 0.2 ms

max ping: 0.3 ms

avg ping: 0.2 ms

Want a fast network? Get your own!

min ping: 0.2 -‐ 0.3 ms

max ping: 7.8 -‐ 59 ms

avg ping: 0.8 -‐ 4.4 ms

22 DB calls per hfp calls

min ping: 0.2 ms

max ping: 0.3 ms

avg ping: 0.2 ms

saved 13 -‐ 92 ms latency

What’s befer? Depends on your needs!

Big cloud is great if you

• don’t know your load

• scale up / down quickly





Bare metal is great if you

• want to save money

• need stable performance





Say: “Don’t worry”

Bare metal is great if you

• want to save money

• need stable performance

Say: “Think ahead”

What’s best?





Hybrid (SQL + Redis) vs. Redis

Hybrid (SQL + Redis)

• started with MySQL

• hit a wall due to IO limits

• introduced Redis later

Redis only

If MySQL is a truck, ...

Fast enough for reads

Can store on disk

Robust replicakon

h=p://www.flickr.com/photos/erix/245657047/

http://www.flickr.com/photos/erix/245657047/


If MySQL is a truck, Redis is a Ferrari

Fast enough for reads

Can store on disk

Robust replicakon

Super fast reads/writes

Out of memory => dead

Fragile replicakon

h=p://www.flickr.com/photos/erix/245657047/








Redis only






MySQL has its hard limits

Redis only

Works fine if done right

How many tables to store a user in?

22 MySQL tables

7 Redis hashes


22 MySQL tables

7 Redis hashes

1 Redis Hash


22 MySQL tables

7 Redis hashes

Easy to implement

Faster for small changes

1 Redis Hash

Less top level keys

Export and import is easy

What’s best?





Redis storage type

Redis works in-‐memory

Dumps slow down master

• Dumping on slaves only

Redis storage type




Redis used vm

Dumps to disk impossible

• Own system to store JSON files to disk

Redis storage type




Redis used vm

Dumps to disk impossible

• Own system to store JSON files to disk

Redis became a cache only

Redis storage type




Redis now uses diskstore

Redis storage type





• Experimental branch

• Stores async. to disk

Redis storage type





• Experimental branch

• Stores async. to disk

Redis became a DB again

There are many inackve users

hourly

daily

monthly

lifetime

0M 2M 4M 6M

6,0M

3,0M

0,4M

0,1Mhourly

daily

monthly

lifetime

0M 8M 16M 24M

20,8M

8,0M

1,3M

0,3M

Not all users play at the same day

In-‐memory

100% of users in RAM

DB hierarchy

• Redis in-‐memory

• Redis diskstore on disk

3% of users in RAM

What’s best?





So, what didn’t work?

Big cloud


In-‐memory “only”

Bare metal

Redis only

DB hierarchy

Let’s dig a bit deeper

1,300,000 daily users

5000 hfp calls/s peak

100 ms response kme

35-‐70 servers

400,000 daily users

2500 hfp calls/s peak

10 ms response kme

11 servers

Hoskng costs / DAU compared

HosDng

5x higher

There are 2 main reasons for higher cost

Databases Other

DB: 20x higher

Other: 3x higher

There are 2 main reasons for higher cost

Databases Other

DB: 20x higher(both RAM and cost)

Other: 3x higher

No DB hierarchy!


Big cloud



Bare metal

Redis only

DB hierarchy


Big cloud



Bare metal

Redis only

DB hierarchy

depends ...


Big cloud



Bare metal

Redis only

DB hierarchy

depends ...

never

again!


Big cloud



Bare metal

Redis only

DB hierarchy

depends ...

never, ever,

ever again!

Next game will do things befer

Start in cloud and then go metal?

Redis only

DB hierarchy

BTW, wooga is hiring!

Q & A

slideshare.net/wooga

Jesper Richter-‐Reichhelm@jrirei

wooga.com/jobs

BTW, wooga is hiring!

Thank you

slideshare.net/wooga

Jesper Richter-‐Reichhelm@jrirei

wooga.com/jobs

Games For The Masses Scaling Rails To The Extreme

Documents

Transcript of Games For The Masses Scaling Rails To The Extreme