Permanent State in a Serverless

Environment

Presenting users with a consistent world without

persistent clients

Overview

➲Presenting users with a persistent state is easy if you have a

persistent host.

➲With a few techniques it can be done without a server, as

long as clients remain in the game.

➲Primary benefit is the saving on Server hosting costs.

Server based model

➲ Traditional Model : Client/Server

➲ Server runs game logic, deciding when events occur.

➲ Game clients receive incoming events and process them

(deterministically).

➲ Late-joiners can receive a snapshot of the world from the host.

Integrated server

➲ One of the clients acts as the server.

➲ That client drives the others via events.

➲ Processes events themselves.

➲ The problem is that the server “state” is never transmitted, only

the client state.

➲ If the integrated server leaves, what do we do.

Integrated server (cont)

➲ If we make the server logic “stateless” we can move the server

at will.

➲ The server is now just code, that runs on one machine. The

only thing the server code can do is send events.

➲ Server is non-deterministic. Client code deterministic.

Moving to peer-to-peer

➲ This would suffice if only the host needs to interact with the

world, and everyone else just views them.

➲ The real problem comes when we allow the clients to interact

with the world as well. (clients become peers)

➲ Peers actions can overlap, creating race conditions.

➲ To resolve these race conditions we could lock resources using

the host as a token server.

The problem with tokens

➲ Tokens are often touted as a solution

➲ Peer signal requests for tokens

➲ Server grants, or refuses

➲ Peer then can control the object

➲ What if server migrates after the signal?

➲ What if peer leaves after obtaining the token?

➲ We cannot reclaim the token until the peer is definitely gone.

Introducing the reflector

➲ What if the server, merely re-broadcasts the signals, without

any logic.

➲ The outgoing broadcasts are ordered, so all the peers see the

signals in the same order.

➲ Peers smart enough to work out when locks fail, or succeed.

➲ Now we can implement complex behaviors.

➲ Example, can only get into a vehicle if it's empty, or occupied

by a team mate.

The Reflector

➲ The reflector is like a server with no state, nor game specific

logic.

➲ It is a true peer-peer logic, all clients are identical, except for

the automatic reflecting of requests.

➲ Peers must remember the sent signals. If the host changes,

they must resend all signals they have not seen reflected.

➲

Terminology

➲ Peers send signals and handle events.

➲ The reflector reflects the signals. Reflected signals are events,

and are in the same order on all machines.

➲ Signals have a unique identifier formed from machine ID and

count of signals sent.

➲ Events also have a unique ID of the reflector and the event

count. (also known as the event stream position)

The Reflect Method

➲ The reflect method is exposed on all machines.

➲ On any client who is not the reflector, the method does nothing.

➲ On the reflector, the event occurs immediately, with the event

sent as normal to other clients.

➲ Reflect methods must not change state, only generate events.

Ownership analogies

➲ Three common ownership patterns

➲ Unique ownership : no-one else can signal, clients can signal

whenever they like.

➲ Single ownership : State is changed using reflect, effective like

the old “server code” pattern.

➲ No/Shared Ownership: context never changes, peers signal

when they wish to change state. Changes may fail.

Unique ownership pattern

Peer 1

Reflector

Peer 3

Signal

Event

In this example, Peer 1, wants to signal

something about an object they own.

Peer 1 call Signal, which sends a signal

to the reflector.

The reflector calls the reflect method, and

sends the results to all clients,

including themselves.

A use of this pattern might be used for

example to allow a player to select

their own weapon.

Event

Peer 1

Reflector

Peer 3

Event

In this example, something must get

done, once and once only. In this

case we poll for the condition on

each client. Each client calls the

reflect method when required.

Only the reflector will do anything when

reflect is called, and they will send

an event to all clients (including

themselves).

It is important to note, that we cannot

guarantee that there is always a

reflector, but we can guarantee that

one will exist at some point. When a

host does exist, they should call the

reflect, and the event will get done.

This usage pattern also has the lowest

latency (for obvious reasons)

A good us for this pattern might be to

trigger the end of the round.

Event

Single ownership pattern

No ownership pattern

Peer 1

Reflector

Peer 3

Signal

Event

This pattern is identical to the unique

ownership pattern, with BIG caveat.

No ownership means more than one peer

may try and use a resource, creating

a race condition.

It is the job of the reflector to resolve the

race condition, by giving definitive

ordering to the events.

This pattern could be used for example to

determine which player gets a

weapon pick up.

Event

Signal

The context issue

➲ A major problem still exists.

➲ The state on the peers may not be up to date (as defined as

the state on the server).

➲ This means that peers can send signals that are no longer valid

when they are reflected.

➲ Most obvious is referencing an object that has since been

deleted.

Example context problem

Peer 1

Reflector

Peer 3

Delete object

Delete object

An example of context problem

• The object is deleted twice.

• The second delete is valid when

sent, because the first delete has not

arrived yet.

• Could be solved by unique object

identifiers, but...

• This is a generic problem. It occurs

because the knowledge of the state

is perfect, but always lags the

reflector.

• Three common re-occurring causes.

• Game state change. (i.e. BE

transition)

• Level state change. (i.e. end of

round)

• Object state, mainly deletion.

Signal context

➲ Signals are tagged with the event processed last.

➲ Provide a number of context markers. Markers are updated

with the last event the context changed.

➲ Simply compare signal tag with the context marker to see if the

context has changed (how to handle left to user).

➲ Markers can be deleted when all peers send signals with event

ID's greater than the marker.

Determinism barriers

➲ All signals originate from non-deterministic actions (such as

user input)

➲ Reflection translates into a deterministic flow.

➲ Once inside a deterministic event, we may not use the signal

method to trigger more non-deterministic behaviour.

Signal from event

Peer 1 Reflector Peer 3

Signal

Using signals from events problematic

• Event sent to all clients

• We end up with multiple signals.

• Could be solved by detecting

multiple events, but...

• A better way exists.

Signal Signal

Non Deterministic-Signal buffer

➲ A good example of the single ownership pattern.

➲ Signaling from inside event handlers, causes the

event to be stored as pending, on all machines.

➲ Peers use reflect method to cause events which are

pending.

➲ Events are removed from the store when the event is

seen.

➲ Pending buffer is a deterministic to non-deterministic

barrier.

Summary

➲ Non-deterministic code can generate signals, but not change

state.

➲ Deterministic code can change state, but not generate signals.

➲ By splitting code into deterministic and non-deterministic, we

can do almost anything, without the need for a server.

Construction order/destruction order

problems

➲ Events are always owned by a single “Type”, but many types

can listen to those events.

➲ This leads to an ordering problem. If an event constructs an

object then the event must be processed first

➲ If an event destroys an object, then the listeners must be called

first.

➲ This is a property of the event, and can be specified on

creation.

Implementing code

➲ Define event object .It needs pack/unpack and name.

➲ I strongly urge use of PureEvent/IntEvent/etc

➲ Inherit from EventOwner

➲ Define an event handler

➲ Initialise Event handler (with order and determinism flags)

➲ Add ReflectionHandler if required

Adding listeners

➲ Inherit from EventListner

➲ Implement listen function.

➲ (remember the header file only needs the event structure

name, nothing else)

Reflection as translation

➲ We can improve network efficiency by allowing reflection as a

translation stage.

➲ Signal->Event/Events from user defined handlers.

➲ Can also enforce that no invalid context events ever seen

(simplifies event handling, allowing designers to write scripted

code)

Code re-use

➲ The event system provides a general purpose dispatching

mechanism which might be of use to non-network related code.

➲ Dispatcher, with nice-ish auto-registration

➲ Events can be prototyped (not many people seem to realize

this)

➲ Can be used for delayed (deletion order) problems

Host Migration

➲ How to choose the next reflector?

➲ Form voting pools.

➲ Majority vote makes host.

➲ Perhaps a vote-less algorithm can be found.

➲ Ideal would be to use external server (Matchmaking2 lobby

messages?) as host voting reflector.

➲ Without a server, split games are possible, although merging

split games is possible