Name : Yan CuiOccupation : Server side dev

Company : gamesysIndustry : Social Gaming

about me

The Game

The World

You!!!

Places

Places

Places

Towns

Towns

Homestead

Homestead

Farming

Crafting

Crafting

Almanac

Almanac

Almanac

thousands

of items!

hundreds of recipes!

more coming each week!!!

Almanac

Almanac

Almanac

Monster Trapping

RightTrap

+ RightBait =

CatchingMonsters!

Monster Trapping

Monster Trapping

Quests

Quests

Quests

Achievements

Achievements

Achievements

In-game Buddies

In-game Buddies

The Community

The Community

In April..

• 200,000 unique players

• 300,000 hours of game play

• 100,000,000 server requests

• 45,000,000 items foraged

• 1,000,000 butterflies caught

• 4,000,000 seeds planted

• 4,000,000 monsters caught

• 5,700,000 items bought

• 24,000,000 items sold

• 280,000 hours of time potion used

• 526,000,000+ km travelled • around the earth 13,000 times...

• Bulk of the game developed by• 1 server developer

• 2 flash developers

• 1 producer and 2 game designers

• Several artists

We have had to find ways to solve

problems effectively and efficiently

The Challenges

• States can be BIG

• Grow with content

Efficiency

• 1000s of items on homestead

• Each item has some state

• Storage units add 1000s more items

• Players like to collect things ;-)

• A lot of states to track on character

• 100s of items in backpack

• 100s of quests/achievements

• Players like to collect things ;-)

• Many actions requires both states at once

• Operations are expensive

• Best players get the worst latency

Stateless Server DatabaseClient

• Stateless server approach+ Easy to scale up and down

+ Scaling can be dynamic based on load

+ Every layer can scale independently

• Stateless server approach– Serialization/deserialization is heavy on CPU

– Heavy load (1:1 read write ratio) on databases

– High bandwidth usage

– High latency

• Stateless server approach– Inefficient

– Expensive

ServerClient

Session 1

Session 2

• Traffic is bursty

• Keep state on server during bursts= Less serialization & deserialization

= Less load on database

= Fewer server and database nodes

=

ServerClient Database

• Stateful server approach

+ 500% improvement on efficiency

+ 60% reduction in average latency

+ Fewer game servers

+ Fewer database nodes

But...

• Server affinity

– Need to ensure server affinity during one burst

– Need to route calls to affined server

• Load balancing

– Some players are more active than others

– Need to avoid hot spots

• Server hogging

– Session lengths vary greatly

– Need to avoid players hogging a server

– Need to be able to scale down!

• Persist state after short inactivity

• Move player to another server after

persistence

• Scaling down

– Need to gracefully drain player states

– Need to be able to stop accepting new players

• Stateful server approach

– Complexity

+ But worthwhile!

+ Others avoid scaling down to reduce complexity

• Progress tied to most actions

• Avoid scripting

• Data driven

Quests & Achievements

Caught a Gnome

EXP Item Gold

Quest Progress

Caught a Gnome

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

New Quest

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

New Quest

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

New QuestAchievement

Progress

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

New QuestAchievement

Progress

Achievement Complete

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

New QuestAchievement

Progress

Achievement Complete

Level Up

Quest Progress

EXP Item Gold

Caught a Gnome

Quest Complete

New QuestAchievement

Progress

Achievement Complete

And that’s not all...

• 100+ actions available in the game

• Most can be tied to quests/achievements

• Many yields rewards

• Triggered from different abstraction layers

• Trapping controller

• Level controller

• ...

• Non-functional requirements

• Analytics

• Partner reporting

• ...

• Message broker pattern

• Something happened, it’s a FACT

• Caught a Gnome

• Got some EXP

• ...

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

Ignore

Process

Process

Process

Process

Ignore

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

EXPItemGold

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

EXPItemGold

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

EXPItemGold

Process

Ignore

Ignore

Ignore

Process

Ignore

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

EXPItemGold

Level Up

Caught Gnome Trapping

Queue

Levelling

Quests

Achievements

Analytics

Partner Reporting

EXPItemGold

Level Up

• Message broker approach

+ Simple

+ Flexible

+ Extensible

• Message broker approach

– Requires large number of different facts

+ F# to the rescue!

• OK for small number of DU cases

• 100+ actions available in the game

• Players love quests

Agility

• Need regular content releases

• Game is heavily data driven

+ No developer involvement for content changes

• Tech team needs to be enablers

• Producers need ability to

• Plan changes weeks in advance

• Quickly see and validate changes before release

• Game designers need ability to

• Work on changes in isolation

• Quickly iterate and test changes in isolation

• QA need ability to

• Quickly switch between different changes to test

• Smoke-test changes in live environment without

affecting real players

• Server team needs ability to

• Version data changes

• Rollback/hotfix data changes with no downtime

We version control our code,

why not our data?

• Git + Git flow

• Servers allow multiple data branches and

switching between them

• What do our players like to do?

• Every action in the game is recorded

Analytics

Feature Analyze

• Analytics is a key decision-making tool

• A/B testing

• 150,000,000+ events recorded in April

• Several GBs of analytics data a day

• Fast growing!

• Google BigQuery

+ Cheap

+ Fast

+ Scalable

+ SQL-like query syntax

• Google BigQuery

– OLAP

– Append-only

• What is our code doing?

Monitoring

• Monitoring

• Not the same as profiling

• Should be always ON

• Difficult to set up in large distributed environment

What to monitor?

• Method execution time

• e.g. IO, CPU intensive

• Method execution count

• e.g. service entry points

• Error count

• ...

• Amazon CloudWatch

+ Cheap

+ Supports alarms and notifications

+ Visualization tool

PostSharp

PostSharp

AmazonCloudWatch

+

=

• Metrics are aggregated at instance level

• Metrics gatherer has high concurrency

requirement

• F# agents to the rescue!

Thank You!

@theburningmonk

Here Be Monstershttp://apps.facebook.com/herebemonsters

AOP and pseudo real-time monitoring with CloudWatchhttp://bit.ly/11GL3SQhttp://bit.ly/10fe98N

Building social games with a .Net stackhttp://bit.ly/ZKtHWb http://bit.ly/10CtyY9

F# Agentshttp://bit.ly/ZmXuEPhttp://bit.ly/17kjbam

F# Deep Diveshttp://bit.ly/XZe10E