At Scale With Style (Erlang User Conference 2012)
-
Upload
wooga -
Category
Technology
-
view
3.795 -
download
2
description
Transcript of At Scale With Style (Erlang User Conference 2012)
![Page 1: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/1.jpg)
AT SCALE WITH STYLEErlang <3 <3 <3 Ruby
Mar$n Rehfeld, @klickmich
![Page 2: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/2.jpg)
At Scale With Style
How we roll
Server architectures
How to innovate
Mind the limits
![Page 3: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/3.jpg)
![Page 4: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/4.jpg)
Typical game architecture
Client BackendHTTPAPI
![Page 5: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/5.jpg)
Typical game architecture
State Changes
ValidaHon
Persistence
Backend
![Page 6: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/6.jpg)
The scale is interesHng
14 billion requests / month
>100,000 DB operaHons / second
>50,000 DB updates / second
![Page 7: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/7.jpg)
Wooga’s approach to development
Small independent teams for each game
Team gets to choose tools and technologiesSame team also does ops a,er going live
Culture of sharingLook around what is there, pick/adapt exis;ng solu;ons, but take ownership for what you include
![Page 8: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/8.jpg)
ExisHng backends – Technology landscape
![Page 9: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/9.jpg)
At Scale With Style
How we roll
Server architectures
How to innovate
Mind the limits
![Page 10: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/10.jpg)
Most games use stateless applicaHon servers
Server Database
![Page 11: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/11.jpg)
And then there’s sharding
app app app app app app app app app
lb
MySQL
MySQL
slave slave
![Page 12: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/12.jpg)
More app servers, more sharding
app app app app appapp app
app app app app app app app app app
appapp
lb
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
slave slave slave slaveslave slave slave slave
![Page 13: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/13.jpg)
Wait, seriously?!
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
lb lb
redis redis redis redis redisMySQL
MySQL
MySQL
MySQL
MySQL
slave slave slave slave slaveslave slave slave slave slave
![Page 14: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/14.jpg)
Find the flaw
“Stateless applicaHon servers guarantee one thing:The data is never
where you need it!”Paolo Negri, Developer @ Wooga
![Page 15: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/15.jpg)
Strong session pa`ern
User starts playing
many transformaHonsof the same set of data
User stops playing
![Page 16: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/16.jpg)
Stateful servers and DBs
Server Database
One Game Session
![Page 17: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/17.jpg)
Stateful game server
One process per acHve user gaming session
... holds the current state and is the only one that can modify it (strong encapsulaHon)
... handles all API calls for the given user one a,er the other (concurrency control through actor model)
... loads the game state from storage and writes it back periodically and on process termina;on(;meout = user stopped playing)
![Page 18: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/18.jpg)
The DB is no longer the bo`leneck
0
7.500
15.000
22.500
30.000
DB operations / second
Stateless Stateful
700
![Page 19: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/19.jpg)
Magic Land uses Erlang
Details:Awesome presentaHon onthe Magic Land game serverby @knuHn & @hungryblank
h`p://www.slideshare.net/wooga/from-‐0-‐to-‐1000000-‐daily-‐users-‐with-‐erlang
![Page 20: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/20.jpg)
At Scale With Style
How we roll
Server architectures
How to innovate
Mind the limits
![Page 21: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/21.jpg)
![Page 22: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/22.jpg)
Erlang & Ruby
Erlang is greatConcurrent, robustGreat for opera;on
Ruby is greatConcise, expressive, flexibleGreat for development
![Page 23: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/23.jpg)
![Page 24: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/24.jpg)
Bringing two worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender
![Page 25: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/25.jpg)
Do you know Mongrel2?
➡ We chose the same queue setup & message format
Web Server that hooksup applicaHons via
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender
![Page 26: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/26.jpg)
![Page 27: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/27.jpg)
ConnecHng the dots
Mongrel2 h`p://mongrel2.org/ protocol & ZeroMQ setup
Erlang: h`ps://github.com/hungryblank/emongrel2
Ruby
rack-‐mongrel2 fork hKps://github.com/khiltd/khi-‐rack-‐mongrel2rack protocol hKp://rack.rubyforge.orgSinatra hKp://www.sinatrarb.com/
➡ essenHally we are speaking HTTP over ZeroMQand can hook up any Rack-‐based Ruby web framework
![Page 28: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/28.jpg)
app.game_action '/:actor/fruit_tree/self/shake', :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do
x, y = params[:x], params[:y] fruit_trees[x, y].shake
end
Example controller in Ruby
DSL-‐like definiHon of game acHon
Skinny as controllers should be
![Page 29: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/29.jpg)
Example model in Ruby
class FruitTree < Tree
property :last_shake_time, :type => Integer, :default => 0 property :collectable_fruit_count, :type => Integer, :default => 0
def shake raise G8::Error::Validation, "no fruit!" unless carries_fruit?
session.user.xp += 1 session.user.energy -= 1 self.last_shake_time = game_time self.collectable_fruit_count = config.fruit_count end
end
Easily unit testable
Minimal amount of code
![Page 30: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/30.jpg)
Game state
Game state is split in mulHple partsuser, map, fruit_trees etc.
Erlang does not care about contentSerialized Ruby objects
Erlang does know mapping of state parts to URLs
![Page 31: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/31.jpg)
app.game_action '/:actor/fruit_tree/self/shake', :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do
x, y = params[:x], params[:y] fruit_trees[x, y].shakeend
Looking back at the game acHon
Mapping of state parts to game acHonsWorker knows mappingWorker pushes mapping to Erlang on startupErlang can query mapping if needed
![Page 32: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/32.jpg)
NICE!
http://www.flickr.com/photos/aigle_dore/
![Page 33: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/33.jpg)
At Scale With Style
How we roll
Server architectures
How to innovate
Mind the limits
![Page 34: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/34.jpg)
Performance impact for large payloads
0
1000
2000
3000
4000
5000
6000
1 8# of workers
Requ
ests / s
PureErlang
RubyZMQ400 rps
200 kB payload
![Page 35: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/35.jpg)
NOT CPU bound
25%
Fixed bandwith limit @ 300 MB/s
![Page 36: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/36.jpg)
This has happened before
0"
500,000"
1,000,000"
1,500,000"
2,000,000"
Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
![Page 37: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/37.jpg)
![Page 38: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/38.jpg)
shake(Args, State) -> Coords = g8_map:coords(Args), Map = g8_game:get_map(State), Tree = g8_map:find(Coords, Map),
NewTree = g8_fruit_tree:shake(Tree), UserEffects = [incr_xp, decr_energy], NewMap = g8_map:place(Coords, NewTree, Map),
[observable, {state, g8_game:set_map(NewMap,
g8_user:apply(UserEffects, State))}].
Example controller in Erlang
![Page 39: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/39.jpg)
Example model in Erlang
shake(Tree) -> validate_carries_fruit(Tree),
FruitCount = g8_fruit_tree_config:fruit_count(Tree), Tree#fruit_tree{last_shake_time = g8_game:gametime(), collectable_fruit_count = FruitCount}.
![Page 40: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/40.jpg)
✓ YES
![Page 41: At Scale With Style (Erlang User Conference 2012)](https://reader033.fdocuments.in/reader033/viewer/2022052823/55582243d8b42a5e468b5054/html5/thumbnails/41.jpg)
QuesHons?
MarHn Rehfeld@klickmich
slideshare.net/woogawooga.com/jobs