Post on 14-Jul-2015
1
GDC 2015 – Amazon Developer Day
Deploying a Low-Latency Multiplayer Game Globally
Nate Wiger Principal, Gaming Solutions
@nateware nateware@amazon.com
James Gwertzman CEO, PlayFab
@playfabnetwork http://playfab.com
AWS Gaming Solutions
[Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving disconnected players stranded
[Game] has a great online mode – if you can get online, that is
“This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”
AWS Gaming Solutions
How To Be Awesome
1. Game servers near players 2. Everything else as HTTP APIs 3. Data replication = bad 4. Local caches = good 5. Nobody really “plays everywhere”
AWS Gaming Solutions
VPC Subnet
VPC Subnet
Game API Pods
Availability Zone A Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group WEB
VPC Subnet
WEB
JOBS
Cognito
SNS Mobile Push
SES
AWS Gaming Solutions
Region
① Login via HTTP API ② Download Game Assets ③ Matchmaking to Game Server
EC2
Game Flow
EC2
EC2
AWS Gaming Solutions
Region
① Login via HTTP API ② Download Game Assets ③ Matchmaking to Game Server ④ Connect to Server ⑤ Hack Apart Your Friends ⑥ Game Over
Game Flow
EC2
EC2
AWS Gaming Solutions
Region
① Login via HTTP API ② Download Game Assets ③ Matchmaking to Game Server ④ Connect to Server ⑤ Hack Apart Your Friends ⑥ Game Over ⑦ Write via HTTP API
Game Flow
EC2
EC2
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Pods
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
AWS Gaming Solutions
Make. Network. Faster.
• C3/C4 instance family (or R3 for more memory) • In a VPC • HVM AMI – not PV • Enable Enhanced Networking (SR-IOV)
http://bit.ly/ec2-enhanced-net • Linux: Tweak Networking Stack
http://bit.ly/linux-tuning
AWS Gaming Solutions
VPC Private Subnet VPC Private Subnet
RabbitMQ + Elastic Load Balancing
Availability Zone A Availability Zone B
10.1.0.13 10.2.0.16
TCP 5672
rabbitmq-node1 rabbitmq-node2 TCP 4369 & 25672
AWS Gaming Solutions
RabbitMQ + Elastic Load Balancing
• Clustering Tutorial https://www.rabbitmq.com/clustering.html
• Set Queue HA policy to "all" https://www.rabbitmq.com/ha.html
• Create Internal load balancer – Listen Port: TCP 5672
• VPC Security Group – For load balancer: TCP 5672 – Between EC2 nodes: TCP 4369 & 25672
• Set up Client to Heartbeat
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Redis Pub / Sub
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
Auto Scaling group
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
CloudFormation + Chef
Availability Zone A
GAME GAME GAME
Auto Scaling group
AWS Gaming Solutions
Unreal Engine with CloudFormation UnrealServer
unreal_linux.template.json
true
my-bucket-name
Unreal-123.zip
sshkeyname
1.2.3.4/24
MyPass123
AWS Gaming Solutions
VPC Subnet
Server Registration
Availability Zone A Availability Zone B
VPC Subnet
Auto Scaling group WEB WEB
Oregon
Tokyo
VPC Subnet JOBS
AWS Gaming Solutions
Server Registration & Scaling
• HTTPS POST /api/servers/register • Include an HMAC (RFC 2104) • Send Server Status
– Public IP – # Players – Game Modes
• Matchmaking Service – Maintains server list – Removes servers
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
S3 Bucket
Amazon CloudWatch Logs
GAME GAME GAME
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
AmazonKinesis
S3 Bucket
Amazon Redshift
GAME GAME GAME
AWS Gaming Solutions
What is PlayFab?
PlayFab is a complete live game operations platform. Focus on the fun – we’ll do the boring bits.
+
Tools for operating & monetizing live games
Complete set of cloud-based backend services
Eco-system of pre-integrated third-party services
.. and more
+
AWS Gaming Solutions
Some Loadout Stats
• Launched 1/31/14
• Over 1 billion API Calls
• At Peak (Feb 2014):
– 300k DAU
– 120 fixed game servers
– 110 EC2 game servers
AWS Gaming Solutions
PlayFab Core Architecture Tenants
• No scheduled downtime.
• Automatic scaling for load.
• Backend changes should be invisible to developers.
• Use buffering to smooth over spikes or failures.
AWS Gaming Solutions
AWS cloud: PlayFab Web Services
US EU
Hardware Game Servers
Hardware Game Servers
Game Client
Amazon Route 53 (3.playfabapi.com)
Amazon EC2 (API handling)
Matchmaker
Instance Instance
Game Server Monitor
DynamoDB Amazon RDS Amazon S3
Elastic Load Balancing
Amazon Redshift Reports service
Instance Logs
Arc
hite
ctur
e O
verv
iew
Matchmaker (Secondary)
Instance US EU
SA AUS
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
AWS Gaming Solutions
List of PlayFab Web APIs Client Server
Authentication Title-Wide Data Management Shared Group Data Authentication AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser LoginWithSteam GetUserInventory GetUserData RegisterPlayFabUser RedeemCoupon GetUserInternalData SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData
Account Management UnlockContainerItem GetUserStatistics GetAccountInfo StartPurchase UpdateUserData GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData LinkFacebookAccount PurchaseItem UpdateUserStatistics LinkGameCenterAccount Friend List Management Title-Wide Data Management LinkSteamAccount AddFriend GetCatalogItems UnlinkFacebookAccount GetFriendsList GetTitleData UnlinkGameCenterAccount RemoveFriend SetTitleData UnlinkSteamAccount SetFriendTags Player Item Management UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency UpdatePassword RegisterForIOSPushNotification GetUserInventory UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers
Player Data Management Matchmaking APIs SubtractUserVirtualCurrency GetFriendLeaderboard GetCurrentGames Matchmaking APIs GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket GetUserData StartGame Steam-Specific APIs GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement GetUserStatistics AndroidDevicePushNotificationRegistration UpdateUserData ValidateGooglePlayPurchase UpdateUserStatistics Analytics
LogEvent
AWS Gaming Solutions
ELB Logging for Title Tracking
Game Client Amazon Route 53 (3.playfabapi.com)
API Request:
Elastic Load Balancing
Logs
Amazon Redshift Amazon S3 bucket
Reports service
Logs Instance
Logs include title-specific endpoint for analytics
AWS wildcard DNS (*.playfabapi.com)
Log event:
AWS Gaming Solutions
Loadout API Usage Report (Oct 2014)
API NAME TOTAL CALLS
AVERAGE BACKEND TIME (S)
AVERAGE RESPONSE TIME (MS)
AVERAGE REQUEST TIME (MS)
TOTAL REQUEST TIME (M)
AVERAGE RECIEVED
(BYTES)
AVERAGE SENT
(BYTES) TOTAL
ERRORS
adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979
gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290
matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319
matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93
matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400
matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107
gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0
gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
AWS Gaming Solutions
Instance
Making a Singleton Highly Available
Health Check
Auto Scaling group
Auto Scaling service
Instance
Matchmaker (Primary)
Instance
Amazon Route 53)
Elastic Load Balancing
Matchmaker (Secondary)
1. Auto Scaling is set to keep two instances up.
2. Elastic Beanstalk Health Check terminates bad instance.
3. Amazon Route 53 fails over all traffic to secondary server.
4. Auto Scaling detects failure and starts new instance.
AWS Gaming Solutions
Monitoring Service Health
Amazon EC2 (API handling)
Elastic Load Balancing CloudWatch
Health Check
Email Alerts On-call Engineer Mobile Alerts
AWS Gaming Solutions
Handling Spikes Without Losing Data
Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports service Local event
logs Instance Events:
logins and purchases
Reports service
Logs
Customer’s game management dashboard
Analytics
Server Logs
RabbitMQ message
queue
Logs Instance
Logstash service
Logstash
Instance
ElasticSearch Instance
Instance
AWS Gaming Solutions
Storage in DynamoDB, Amazon RDS, Amazon S3, Amazon Redshift
DynamoDB Amazon RDS Amazon S3
Player data: • Accounts • Profile • Inventory • Save data
• Player stats • Player currency
values • Auditing
• Log files • Game replays
Amazon Redshift
• Analytics data
AWS Gaming Solutions
Managing Tables in DynamoDB
Amazon DynamoDB
• Player data: accounts, profile, inventory, save data
User: UserID: Hash TitleID: range key - DisplayName - Linked accounts - …
Custom user data: UserID: Hash TitleID: range key - Key/value 1 - Key/value 2 - …
Inventory: UserID: Hash ItemID: range key - titleID - Catalog Item - …
…
AWS Gaming Solutions
Physical Deployment
AZ 1
AZ 2
PlayFab Web Services
PlayFab Web Services
N. California
AZ 1
AZ 2
PlayFab Web Services
PlayFab Web Services
Oregon
AZ 1
PlayFab Web Services
Beijing
• Original deployment: us-west-1
• Expanded to us-west-2 in July
• Coming soon: cn-north-1
Amazon EC2 Game Server Singapore
Game Client
Instance
AWS Gaming Solutions
Managing Cross-Region Data
Player accounts are shared across all regions:
• Cross-regional wrapper for user.get sets order of databases and regions to check
• user.get then checks for player account data
• High availability, transparent to customers and players
// Returns the first matching player record, searching DynamoDB and SimpleDB
// in both regions in the specified order.
// Updates the player record in the store and region from which it was loaded.
// Creates a player record in the default store and region.
AWS Gaming Solutions
AWS cloud: PlayFab Web Services
US EU
Hardware Game Servers
Hardware Game Servers Game Client
Amazon Route 53 (3.playfabapi.com)
Amazon EC2 (API handling)
Matchmaker
Instance Instance
Game Server Monitor
DynamoDB Amazon RDS Amazon S3
Elastic Load Balancing
Amazon Redshift Reports service
Instance Logs
Arc
hite
ctur
e O
verv
iew
Matchmaker (Secondary)
Instance US EU
SA AUS
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers