(GAM201) Cloud Gaming Architectures from Mobile to Social to MMO

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Mark Bate, Solutions Architecture, AWS

Jaeman An, Software Engineer, Devsisters Corp.

GAM201

Cloud Gaming ArchitecturesFrom Social to Mobile to MMO

Gratuitous logo slide

Traditional: Rigid AWS: Elastic

Servers

Demand

Capacity

Excess Capacity

Wasted $$

Demand

Unmet Demand

Upset Players

Missed Revenue :(

Scale to what you need, pay for what you use

11 regions

53 edge locations

Continuous expansion

Global is good

Common game back-end concepts

Think in terms of APIs

HTTP + JSON

Get friends, leaderboard

Binary asset data

Multiplayer servers

High availability

Scalability

Core (HA) game back end

ELB

• Choose region

• >=2 Availability Zones

• Amazon EC2 for app

• Elastic Load Balancing

• Amazon RDS database

• Multi-AZ

Region

Scale it way out

ELB

• Amazon S3 for game data

• Assets

• UGC

• Analytics

Region

Scale it way out

ELB


• Assets

• UGC

• Analytics

• ... With Amazon

CloudFront!

Region

CloudFront

CDN

Scale it way out


• Assets

• UGC

• Analytics

• ... with CloudFront!

• Auto Scaling group

• Capacity on demand

• Respond to users

• Automatic healing

ELB

Region

CloudFront

CDN

Scale it way out


• Assets

• UGC

• Analytics

• ... with CloudFront!

• Auto Scaling group

• Capacity on demand

• Respond to users

• Automatic healing

• Amazon ElastiCache

• Memcached

• Redis

ELB

Region

CloudFront

CDN

Writing is painful

• Games are write heavy

• Caching of limited use

• Key value

• Binary structures

• Database = bottleneck

ELB

Region

CloudFront

CDN

Sharding (not fun)

Amazon DynamoDB

• Fully managed

• NoSQL data store

• Provisioned throughput

• Secondary indexes

• PUT/GET keys

• Document support!

ELB

Region

CloudFront

CDN

Example: Leaderboard in DynamoDB

• Hash key = Primary key

• Range key = Sub key

• Range key = Sort key

• Others attributes are

undefined

• So… How to sort based

on top score?

UserID

(hash key)

BoardName

(range key)

TopScore TopScoreDate

"101" "Galaxy Invaders" 5842 "2014-09-15T17:24:31"

"101" "Meteor Blasters" 1000 "2014-10-22T23:18:01"

"101" "Starship X" 24 "2014-08-31T13:14:21"

"102" "Alien Adventure" 192 "2014-07-12T11:07:56"


"103" "Attack Ships" 3 "2014-10-19T01:13:24"


"103" "Meteor Blasters" 723 "2014-10-19T01:14:24"

"103" "Starship X" 42 "2014-07-11T06:53:03"

Leaderboard with secondary indexes

• Create a secondary index!

• Set hash key to BoardName

• Set range key to TopScore

• Project extra attributes as needed

• Can now query by BoardName,

sorted by TopScore

• Handles many common gaming

use cases

BoardName

(hash key)

TopScore

(range key)

UserID

"Alien Adventure" 192 "101"

"Attack Ships" 3 "103"

"Galaxy Invaders" 0 "102"



"Meteor Blasters" 723 "103"

"Meteor Blasters" 1000 "101"

"Starship X" 24 "101"

"Starship X" 42 "103"

UserID

(hash key)

BoardName

(range key)

TopScore TopScoreDate


Documents in DynamoDB

Scalar types: String, Number, Binary, Boolean, Null

Multivalue types: String Set, Number Set, Binary Set

Document types: List, Map

Document content addressing

"name": ”Mark",

"games": ["Megablast","Spacerace"],

"score": {"Megablast" : 123,"Spacerace" : 41

}

"name": {"S": ”Mark"

}"games": {"L": [ { "S": "Megablast" },

{ "S": "Spacerace" } ]},"score": {"M": {"Megablast": { "N": "123" },"Spacerace": { "N": "41" }

}}

"name": {"S": ”Mark"

}"games": {"L": [ { "S": "Megablast" },

{ "S": "Spacerace" } ]},"score": {"M": {"Megablast": { "N": "123" },"Spacerace": { "N": "41" }

}}

document.score.Megablast

Related sessions

DAT204 NoSQL? No Worries: Building Scalable

Applications on AWS NoSQL Services

DAT401 Amazon DynamoDB Deep Dive: Schema Design,

Indexing, JSON, Search, and More

GAM401 Serverless Mobile Game Development with

Amazon Cognito, AWS Lambda, and Amazon

DynamoDB


Cloud Gaming Architectures from Mobile to Social to MMO

Jaeman An (Devsisters Corp.)

18

GAM201

Cloud Gaming Architecturesfrom Mobile to Social to MMO



What to expect from the session

19

How we improved our design

Tips and tricks

Retrospect

How we started



Cookie Run

20



Cookie Run video

21


Cloud Gaming Architectures from Mobile to Social to MMO 22

About Cookie Run

• 70M~ downloads

• 10M DAU

• Top free 1st in 10 countries

• Top free 10th in 38 countries



More about Devsisters and Cookie Run

23



How We Started

24



In early 2013…

Lack of infrastructure, lack of developer, no hope

(1 server developer / 0 system engineers)

Only 1 game in service

Ovenbreak 2

- AWS US East

Cookie Run

- Only 1 person, 1 month left

25



Goal

26

Highly reliable Quality assured

Scalable designAuto configuring

and scaling

Real-time monitoring system

Log system



First design

27

Game server

Operation tool

Monitoring

Java, Spring MVC, MySQL 5.5

Python, Django, Boto

Amazon CloudWatch, Zabbix, Statsd, Graphite



First design

28



After 11 days

29



Design Improvements

30



Design improvements

31

Improving the logging system

Improving the game patch system

Adding global user ranking system

Redesigning the back end




Players send game hearts to each other. Back ends do the bookkeeping

- Plan A: Used MySQL for storing data

Trouble: MySQL can’t keep up; too many rows (100M ~)

- Plan B: Gave unlimited hearts to users! Disabled the feature

Trouble: Not so bad, but need to come up with a better solution

32

Situation



Solution

MySQL → NoSQL (Couchbase)

Use MySQL for game data (shop data, stage data, …)

Use NoSQL for user data (user items, level, coin, …)

33




Before

34



After

35




We need real-time log querying capability

36

Real-time log viewing system based on ELK

Situation

Solution



Before

37



After

38



/Real-time log viewing system

39



Improving the game patch system

40

App Store binary size limit

Some resources need to be downloaded on demand

Wanted to distribute patches without App Store update

Constructed a decent patch system

Based on Amazon S3 and Amazon CloudFront

Situation

Solution



Before

41



After

42




43

Total log size >10 TB; want to analyze all logs

Situation forced us to look for big data solutions

Adopted big data platforms using Amazon EMR or Amazon EC2

Situation

Solution

Eventually migrated to Spark and Spark SQL



Before

44



After

45



Spark

46



Log dashboard

47



Adding global user ranking system

48

Want to introduce global user ranking system

Use ordered set based on skip list using with ElastiCache

…with custom caching and a lot of optimization techniques

Situation

Solution



Before

49



After

50



Tips and Tricks

51



THIS CAN HAPPEN TO YOU

BASED ON THE TRUE STORY OF OUR TEAM

52

WARNING



Auto Scaling gotchas

Frequency: More than 10 times during 2 years

Many users connect to the game simultaneously

• During holiday seasons

• Start of in-game events

• When bulk push notifications are sent

• Or reasons unknown

Booting instances takes several minutes,

which isn’t quick enough to handle spikey loads

We have to predict traffic surges and prepare beforehand

53



Our bulk push system

54




Don’t set minimum instance of 1 or 2

If one machine dies, service fails

Use multiple Availability Zones

Sometimes instance availability of a single AZ can run out

Use multiple AZ with ELB cross-zone balancing

55




Set scale-out(scale-in) policy meticulously

scale-out: +4 when Latency >= 0.1 for 2 minutes

scale-in: -2 when CPUUtilization < 10 for 2 minutes

Sometimes scale-up can be a useful option

56



Chef server failure

Auto Scaling group relying on Chef server is dangerous

Chef server is a single point of failure (SPOF)

May become unresponsive when too many servers start simultaneously

Errors happen in unexpected places!

57



Couchbase storage failure

Hardware problems can occur in EC2 instances

The worst, the most hopeless system failure

Front end API server can crash; that’s OK

But if you are maintaining a database on EC2, this can be a tragedy

It really happens

58



Couchbase storage failure

June 2015

A monumental hell gate in our company history

Server down for 12 consecutive hours because of a disk error in Couchbase

Also, our daily backup script had not worked for 1 week prior to the shutdown

Some data were restored via replication

The other data were restored through adding the lost week’s logs to previously

backed up data

Lesson learned: Replica is necessary. Confirm backups.

59



Overseas network failure

Frequency: More than 5 times over 2 years

This situation has really happened

ISPs cut costs leading to overseas packet loss

Just Call AWS

60



Final Design Review

61



First design

62



Final

63



Future Plans

64



Future plans

Transactional log system (Logstash → Kafka)

High latency / packet loss networks : QUIC

…

Entertain the world!

65


Thank [email protected]

Amazon Cognito

Identity

Providers

Unique

IdentitiesJoe Anna Bob

Any Device

Any Platform

Any AWS

Service

Helps implement security best practicesSecurely access any AWS service from mobile

device; it simplifies the interaction with AWS

Identity and Access Management

Support multiple login providersEasily integrate with major login providers for

authentication, or use your own authentication

system

Unique users vs. devicesManage unique identities; automatically recognize

unique user across devices and platforms

Mobile

AnalyticsS3 DynamoDB Kinesis

Your own

Auth

Amazon Cognito

Synchronize data across devices with Amazon Cognito

Sync game state

across OS, devices State transition

(link multiple accounts)Sync user profiles

across OS, devices, web

Related sessions

GAM401 Serverless Mobile Game Development with

Amazon Cognito, AWS Lambda, and Amazon

DynamoDB

MBL402 Mobile Identity Management and Data

Synchronization Using Amazon Cognito

WRK202 Build a Scalable Mobile App on Serverless,

EventTriggered, BackEnd Logic

Player TwoPress Start

Multiplayer game servers

Region

• API back-end app

• Core session

• Matchmaking

• S3 + CloudFront

• DLC, assets

• Game saves

• UGC

• Public server tier

• Direct client socket

• Scale on players


① Login via API

② Request matchmaking

③ Get game server IP

Region


① Login via API

② Request matchmaking

③ Get game server IP

④ Connect to server

⑤ Pull down assets

⑥ Other players join

Region

Multiregion game servers

Region ARegion B

Related sessions

GAM403 From 0 to 60 Million Player Hours in 400 Billion

Star Systems

GAM404 Evolve: Hunting Monsters in a Low Latency

Multiplayer Game on Amazon EC2

GAM407 Quiplash: The Multiscreen, Multidevice,

Multiplayer Game for 10,000

Wrap it up already

Use Auto Scaling to save money

Amazon CloudFront + Amazon S3 for download and upload

Painful DIYDB? No! Use Amazon DynamoDB

Dynamically manage game servers using the APIs

• Even multiregion!

Thank you!

Mark Bate – [email protected]

Jaeman An – [email protected]

mailto:[email protected]

mailto:[email protected]

Remember to complete

your evaluations!

(GAM201) Cloud Gaming Architectures from Mobile to Social to MMO

Technology

Transcript of (GAM201) Cloud Gaming Architectures from Mobile to Social to MMO