Name : Yan CuiOccupation : Server side dev
Company : gamesysIndustry : Social Gaming
about me
The Game
The World
You!!!
Places
Places
Places
Towns
Towns
Homestead
Homestead
Farming
Crafting
Crafting
Almanac
Almanac
Almanac
thousands
of items!
hundreds of recipes!
more coming each week!!!
Almanac
Almanac
Almanac
Monster Trapping
RightTrap
+ RightBait =
CatchingMonsters!
Monster Trapping
Monster Trapping
Quests
Quests
Quests
Achievements
Achievements
Achievements
In-game Buddies
In-game Buddies
The Community
The Community
In April..
• 200,000 unique players
• 300,000 hours of game play
• 100,000,000 server requests
• 45,000,000 items foraged
• 1,000,000 butterflies caught
• 4,000,000 seeds planted
• 4,000,000 monsters caught
• 5,700,000 items bought
• 24,000,000 items sold
• 280,000 hours of time potion used
• 526,000,000+ km travelled • around the earth 13,000 times...
• Bulk of the game developed by• 1 server developer
• 2 flash developers
• 1 producer and 2 game designers
• Several artists
We have had to find ways to solve
problems effectively and efficiently
The Challenges
• States can be BIG
• Grow with content
Efficiency
• 1000s of items on homestead
• Each item has some state
• Storage units add 1000s more items
• Players like to collect things ;-)
• A lot of states to track on character
• 100s of items in backpack
• 100s of quests/achievements
• Players like to collect things ;-)
• Many actions requires both states at once
• Operations are expensive
• Best players get the worst latency
Stateless Server DatabaseClient
• Stateless server approach+ Easy to scale up and down
+ Scaling can be dynamic based on load
+ Every layer can scale independently
• Stateless server approach– Serialization/deserialization is heavy on CPU
– Heavy load (1:1 read write ratio) on databases
– High bandwidth usage
– High latency
• Stateless server approach– Inefficient
– Expensive
ServerClient
Session 1
Session 2
• Traffic is bursty
• Keep state on server during bursts= Less serialization & deserialization
= Less load on database
= Fewer server and database nodes
=
ServerClient Database
• Stateful server approach
+ 500% improvement on efficiency
+ 60% reduction in average latency
+ Fewer game servers
+ Fewer database nodes
But...
• Server affinity
– Need to ensure server affinity during one burst
– Need to route calls to affined server
• Load balancing
– Some players are more active than others
– Need to avoid hot spots
• Server hogging
– Session lengths vary greatly
– Need to avoid players hogging a server
– Need to be able to scale down!
• Persist state after short inactivity
• Move player to another server after
persistence
• Scaling down
– Need to gracefully drain player states
– Need to be able to stop accepting new players
• Stateful server approach
– Complexity
+ But worthwhile!
+ Others avoid scaling down to reduce complexity
• Progress tied to most actions
• Avoid scripting
• Data driven
Quests & Achievements
Caught a Gnome
EXP Item Gold
Quest Progress
Caught a Gnome
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New Quest
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New Quest
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New QuestAchievement
Progress
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New QuestAchievement
Progress
Achievement Complete
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New QuestAchievement
Progress
Achievement Complete
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New QuestAchievement
Progress
Achievement Complete
And that’s not all...
• 100+ actions available in the game
• Most can be tied to quests/achievements
• Many yields rewards
• Triggered from different abstraction layers
• Trapping controller
• Level controller
• ...
• Non-functional requirements
• Analytics
• Partner reporting
• ...
• Message broker pattern
• Something happened, it’s a FACT
• Caught a Gnome
• Got some EXP
• ...
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
Ignore
Process
Process
Process
Process
Ignore
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Process
Ignore
Ignore
Ignore
Process
Ignore
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Level Up
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Level Up
• Message broker approach
+ Simple
+ Flexible
+ Extensible
• Message broker approach
– Requires large number of different facts
+ F# to the rescue!
• OK for small number of DU cases
• 100+ actions available in the game
• Players love quests
Agility
• Need regular content releases
• Game is heavily data driven
+ No developer involvement for content changes
• Tech team needs to be enablers
• Producers need ability to
• Plan changes weeks in advance
• Quickly see and validate changes before release
• Game designers need ability to
• Work on changes in isolation
• Quickly iterate and test changes in isolation
• QA need ability to
• Quickly switch between different changes to test
• Smoke-test changes in live environment without
affecting real players
• Server team needs ability to
• Version data changes
• Rollback/hotfix data changes with no downtime
We version control our code,
why not our data?
• Git + Git flow
• Servers allow multiple data branches and
switching between them
• What do our players like to do?
• Every action in the game is recorded
Analytics
Feature Analyze
• Analytics is a key decision-making tool
• A/B testing
• 150,000,000+ events recorded in April
• Several GBs of analytics data a day
• Fast growing!
• Google BigQuery
+ Cheap
+ Fast
+ Scalable
+ SQL-like query syntax
• Google BigQuery
– OLAP
– Append-only
• What is our code doing?
Monitoring
• Monitoring
• Not the same as profiling
• Should be always ON
• Difficult to set up in large distributed environment
What to monitor?
• Method execution time
• e.g. IO, CPU intensive
• Method execution count
• e.g. service entry points
• Error count
• ...
• Amazon CloudWatch
+ Cheap
+ Supports alarms and notifications
+ Visualization tool
PostSharp
PostSharp
AmazonCloudWatch
+
=
• Metrics are aggregated at instance level
• Metrics gatherer has high concurrency
requirement
• F# agents to the rescue!
Thank You!
@theburningmonk
Here Be Monstershttp://apps.facebook.com/herebemonsters
AOP and pseudo real-time monitoring with CloudWatchhttp://bit.ly/11GL3SQhttp://bit.ly/10fe98N
Building social games with a .Net stackhttp://bit.ly/ZKtHWb http://bit.ly/10CtyY9
F# Agentshttp://bit.ly/ZmXuEPhttp://bit.ly/17kjbam
F# Deep Diveshttp://bit.ly/XZe10E
Top Related