GDC 2015 - Low-latency Multiplayer Gaming with AWS

1

GDC 2015 – Amazon Developer Day

Deploying a Low-Latency Multiplayer Game Globally

Nate Wiger Principal, Gaming Solutions

@nateware [email protected]

James Gwertzman CEO, PlayFab

@playfabnetwork http://playfab.com

AWS Gaming Solutions

[Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes

“We are working hard to get everyone online as soon as possible,” [Game Studio] says.

Days later, [Game Studio] leaving disconnected players stranded

[Game] has a great online mode – if you can get online, that is

“This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”

AWS Gaming Solutions http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency


100+ms 100+ms

100+ms 100+ms


How To Be Awesome

1.  Game servers near players 2.  Everything else as HTTP APIs 3.  Data replication = bad 4.  Local caches = good 5.  Nobody really “plays everywhere”


Loosely Coupled Pods

Tokyo

Oregon

Frankfurt

Virginia


VPC Subnet

VPC Subnet

Game API Pods

Availability Zone A Availability Zone B

VPC Subnet

VPC Subnet

Auto Scaling group WEB

VPC Subnet

WEB

JOBS

Cognito

SNS Mobile Push

SES


Region

①  Login via HTTP API ②  Download Game Assets ③  Matchmaking to Game Server

EC2

Game Flow

EC2

EC2


Region

①  Login via HTTP API ②  Download Game Assets ③  Matchmaking to Game Server ④  Connect to Server ⑤  Hack Apart Your Friends ⑥  Game Over

Game Flow

EC2

EC2


Region

①  Login via HTTP API ②  Download Game Assets ③  Matchmaking to Game Server ④  Connect to Server ⑤  Hack Apart Your Friends ⑥  Game Over ⑦  Write via HTTP API

Game Flow

EC2

EC2


VPC Private Subnet

VPC Public Subnet

Game Server Pods


VPC Public Subnet

VPC Private Subnet

GAME GAME GAME GAME GAME GAME


Make. Network. Faster.

•  C3/C4 instance family (or R3 for more memory) •  In a VPC •  HVM AMI – not PV •  Enable Enhanced Networking (SR-IOV)

http://bit.ly/ec2-enhanced-net •  Linux: Tweak Networking Stack

http://bit.ly/linux-tuning


Enhanced Networking (SR-IOV)

Before: Hypervisor

After: Hardware


VPC Private Subnet VPC Private Subnet

RabbitMQ + Elastic Load Balancing


10.1.0.13 10.2.0.16

TCP 5672

rabbitmq-node1 rabbitmq-node2 TCP 4369 & 25672


RabbitMQ + Elastic Load Balancing

•  Clustering Tutorial https://www.rabbitmq.com/clustering.html

•  Set Queue HA policy to "all" https://www.rabbitmq.com/ha.html

•  Create Internal load balancer –  Listen Port: TCP 5672

•  VPC Security Group –  For load balancer: TCP 5672 –  Between EC2 nodes: TCP 4369 & 25672

•  Set up Client to Heartbeat


/etc/rabbitmq/rabbitmq.config 5672

rabbitmq-node1

rabbitmq-node2


VPC Private Subnet

VPC Public Subnet

Redis Pub / Sub


VPC Public Subnet

VPC Private Subnet

GAME GAME GAME GAME GAME GAME

Auto Scaling group


VPC Private Subnet

VPC Public Subnet

CloudFormation + Chef

Availability Zone A

GAME GAME GAME

Auto Scaling group


Unreal Engine with CloudFormation UnrealServer

unreal_linux.template.json

true

my-bucket-name

Unreal-123.zip

sshkeyname

1.2.3.4/24

MyPass123


Where My Servers At?

Tokyo

Oregon

Frankfurt

Virginia

?

?


VPC Subnet

Server Registration


VPC Subnet

Auto Scaling group WEB WEB

Oregon

Tokyo

VPC Subnet JOBS


Server Registration & Scaling

•  HTTPS POST /api/servers/register •  Include an HMAC (RFC 2104) •  Send Server Status

–  Public IP –  # Players –  Game Modes

•  Matchmaking Service –  Maintains server list –  Removes servers


VPC Private Subnet

VPC Public Subnet

Game Server Logs

Availability Zone A

S3 Bucket

Amazon CloudWatch Logs

GAME GAME GAME


VPC Private Subnet

VPC Public Subnet

Game Server Logs

Availability Zone A

AmazonKinesis

S3 Bucket

Amazon Redshift

GAME GAME GAME


Mini Pods Everywhere

China Oregon

Frankfurt

Virginia China

Leaderboard


PlayFab: Loadout Architecture


What is PlayFab?

PlayFab is a complete live game operations platform. Focus on the fun – we’ll do the boring bits.

+

Tools for operating & monetizing live games

Complete set of cloud-based backend services

Eco-system of pre-integrated third-party services

.. and more

+


Some Loadout Stats

•  Launched 1/31/14

•  Over 1 billion API Calls

•  At Peak (Feb 2014):

–  300k DAU

–  120 fixed game servers

–  110 EC2 game servers


PlayFab Core Architecture Tenants

•  No scheduled downtime.

•  Automatic scaling for load.

•  Backend changes should be invisible to developers.

•  Use buffering to smooth over spikes or failures.


AWS cloud: PlayFab Web Services

US EU

Hardware Game Servers


Game Client

Amazon Route 53 (3.playfabapi.com)

Amazon EC2 (API handling)

Matchmaker

Instance Instance

Game Server Monitor

DynamoDB Amazon RDS Amazon S3

Elastic Load Balancing

Amazon Redshift Reports service

Instance Logs

Arc

hite

ctur

e O

verv

iew

Matchmaker (Secondary)

Instance US EU

SA AUS

Amazon EC2 Game Servers





List of PlayFab Web APIs Client Server

Authentication Title-Wide Data Management Shared Group Data Authentication AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser LoginWithSteam GetUserInventory GetUserData RegisterPlayFabUser RedeemCoupon GetUserInternalData SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData

Account Management UnlockContainerItem GetUserStatistics GetAccountInfo StartPurchase UpdateUserData GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData LinkFacebookAccount PurchaseItem UpdateUserStatistics LinkGameCenterAccount Friend List Management Title-Wide Data Management LinkSteamAccount AddFriend GetCatalogItems UnlinkFacebookAccount GetFriendsList GetTitleData UnlinkGameCenterAccount RemoveFriend SetTitleData UnlinkSteamAccount SetFriendTags Player Item Management UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency UpdatePassword RegisterForIOSPushNotification GetUserInventory UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers

Player Data Management Matchmaking APIs SubtractUserVirtualCurrency GetFriendLeaderboard GetCurrentGames Matchmaking APIs GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket GetUserData StartGame Steam-Specific APIs GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement GetUserStatistics AndroidDevicePushNotificationRegistration UpdateUserData ValidateGooglePlayPurchase UpdateUserStatistics Analytics

LogEvent


Example of Web API Call

API Request: Response from Web Services:


ELB Logging for Title Tracking

Game Client Amazon Route 53 (3.playfabapi.com)

API Request:


Logs

Amazon Redshift Amazon S3 bucket

Reports service

Logs Instance

Logs include title-specific endpoint for analytics

AWS wildcard DNS (*.playfabapi.com)

Log event:


Loadout API Usage Report (Oct 2014)

API NAME TOTAL CALLS

AVERAGE BACKEND TIME (S)

AVERAGE RESPONSE TIME (MS)

AVERAGE REQUEST TIME (MS)

TOTAL REQUEST TIME (M)

AVERAGE RECIEVED

(BYTES)

AVERAGE SENT

(BYTES) TOTAL

ERRORS

adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979

gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290

matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319

matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93

matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400

matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107

gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0

gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2


Instance

Making a Singleton Highly Available

Health Check

Auto Scaling group

Auto Scaling service

Instance

Matchmaker (Primary)

Instance

Amazon Route 53)



1. Auto Scaling is set to keep two instances up.

2. Elastic Beanstalk Health Check terminates bad instance.

3. Amazon Route 53 fails over all traffic to secondary server.

4. Auto Scaling detects failure and starts new instance.


Monitoring Service Health


Elastic Load Balancing CloudWatch

Health Check

Email Alerts On-call Engineer Mobile Alerts


Handling Spikes Without Losing Data

Amazon EC2 API Server

Server SSD

Amazon Redshift

Local Reports service Local event

logs Instance Events:

logins and purchases

Reports service

Logs

Customer’s game management dashboard

Analytics

Server Logs

RabbitMQ message

queue

Logs Instance

Logstash service

Logstash

Instance

ElasticSearch Instance

Instance


Storage in DynamoDB, Amazon RDS, Amazon S3, Amazon Redshift


Player data: •  Accounts •  Profile •  Inventory •  Save data

•  Player stats •  Player currency

values •  Auditing

•  Log files •  Game replays

Amazon Redshift

•  Analytics data


Managing Tables in DynamoDB

Amazon DynamoDB

•  Player data: accounts, profile, inventory, save data

User: UserID: Hash TitleID: range key -  DisplayName -  Linked accounts -  …

Custom user data: UserID: Hash TitleID: range key -  Key/value 1 -  Key/value 2 -  …

Inventory: UserID: Hash ItemID: range key -  titleID -  Catalog Item -  …

…


Physical Deployment

AZ 1

AZ 2

PlayFab Web Services


N. California

AZ 1

AZ 2



Oregon

AZ 1


Beijing

•  Original deployment: us-west-1

•  Expanded to us-west-2 in July

•  Coming soon: cn-north-1

Amazon EC2 Game Server Singapore

Game Client

Instance


Managing Cross-Region Data

Player accounts are shared across all regions:

•  Cross-regional wrapper for user.get sets order of databases and regions to check

•  user.get then checks for player account data

•  High availability, transparent to customers and players

// Returns the first matching player record, searching DynamoDB and SimpleDB

// in both regions in the specified order.

// Updates the player record in the store and region from which it was loaded.

// Creates a player record in the default store and region.


AWS cloud: PlayFab Web Services

US EU


Hardware Game Servers Game Client

Amazon Route 53 (3.playfabapi.com)


Matchmaker

Instance Instance

Game Server Monitor



Amazon Redshift Reports service

Instance Logs

Arc

hite

ctur

e O

verv

iew


Instance US EU

SA AUS





52

Nate Wiger Principal, Gaming Solutions

@nateware [email protected]

GDC 2015 - Low-latency Multiplayer Gaming with AWS

Technology

Transcript of GDC 2015 - Low-latency Multiplayer Gaming with AWS