F# at GameSys
Transcript of F# at GameSys
F# at GameSys!(how we used F# to build games for millions)
Hi, my name is Yan Cui.
agenda
Domain Modelling!Infrastructure!Game Logic!Algorithms!
DSLs!OSS
1MILLION USERS
ACTIVEDAILY
250MILLION DAY
PERREQUEST
2MONTH
TBP E R
secops25,000
we are polyglot
C#
why F#?
time to market
correctness
efficient
tame complexity
Collectables
Wager Size
Special SymbolAvg Wager Size
Web Server call
• Line Win!– X number of matching symbols on adjacent columns!– Positions have to be a ‘line’!– Wild symbols substitute for other symbols!
!• Scatter Win!– X number of matching symbols anywhere!– Triggers bonus game
What symbols should land?!Any special symbol wins?!
Did the player win anything?
What the player’s new average wager?
Should the player receive collectables?
type Symbol = Standard! of string!! ! ! | Wild
type Symbol = Standard! of string!! ! ! | Wild
e.g.! Standard “hat”!! Standard “shoe”!! Standard “bonus”!! …
type Symbol = Standard! of string!! ! ! | Wild
i.e.! Wild
!
!
!
type Win = LineWin of int * Symbol * int!! | ScatterWin of Symbol * int
!
!
!
type Win = LineWin of int * Symbol * int!! | ScatterWin of Symbol * int
e.g.! LineWin (5, Standard “shoe”, 4)
!
!
!
type Win = LineWin of int * Symbol * int!! | ScatterWin of Symbol * int
e.g.! ScatterWin (Standard “bonus”, 3)
type LineNum = int!type Count! = int!!
type Win = LineWin of LineNum * Symbol * Count!! | ScatterWin of Symbol * Count
type LineNum = int!type Count! = int!!
type Win = LineWin of LineNum * Symbol * Count!! | ScatterWin of Symbol * Count
e.g.! LineWin (5, Standard “shoe”, 4)
type LineNum = int!type Count! = int!!
type Win = LineWin of LineNum * Symbol * Count!! | ScatterWin of Symbol * Count
e.g.! ScatterWin (Standard “bonus”, 3)
make invalid states unrepresentable
[<Measure>]!type Pence
e.g.! 42<Pence>!! 153<Pence>!! …
10<Meter> / 2<Second> ! = 5<Meter/Second>!10<Meter> * 2<Second> ! = 20<Meter Second> !10<Meter> + 10<Meter> ! = 20<Meter>!10<Meter> * 10! ! ! = 100<Meter>!10<Meter> * 10<Meter> ! = 100<Meter2>!10<Meter> + 2<Second> ! // error!10<Meter> + 2 ! ! ! // error
10<Meter> / 2<Second> ! = 5<Meter/Second>!10<Meter> * 2<Second> ! = 20<Meter Second> !10<Meter> + 10<Meter> ! = 20<Meter>!10<Meter> * 10! ! ! = 100<Meter>!10<Meter> * 10<Meter> ! = 100<Meter2>!10<Meter> + 2<Second> ! // error!10<Meter> + 2 ! ! ! // error
type Wager = int64<Pence>!type Multiplier = int!!
type Payout = Coins! of Wager!! ! | MultipliedCoins of Multiplier * Wager!! ! | Multi! of Payout list!! ! | BonusGame
type Wager = int64<Pence>!type Multiplier = int!!
type Payout = Coins! of Wager!! ! | MultipliedCoins of Multiplier * Wager!! ! | Multi! of Payout list!! ! | BonusGame
type Wager = int64<Pence>!type Multiplier = int!!
type Payout = Coins! of Wager!! ! | MultipliedCoins of Multiplier * Wager!! ! | Multi! of Payout list!! ! | BonusGame
type State = ! {!
AvgWager! : Wager!SpecialSymbol : Symbol!Collectables : Map<Collectable, Count>!
}
New avg wager Got a Collectable!
A pay line win!
Recap
lightweight syntax for types & hierarchies
great for domain modelling
make invalid states unrepresentable
better correctness
order of magnitude increase in productivity
player states are big
Stateless Server DatabaseClient
1:1 read-write ratio
ServerClient
Session 1
Session 2
ServerClient Database
Elastic Load Balancer
S3
Auto scaling Group
Server A Server B
...
EC2
CloudFront
Stateful Server
The Actor Model
An actor is the fundamental unit of computation which embodies the 3 things!
• Processing!• Storage!• Communication!
that are essential to computation.!!
-Carl Hewitt** http://bit.ly/HoNHbG
The Actor Model
• Everything is an actor!• An actor has a mailbox!• When an actor receives a message it can:!– Create new actors!– Send messages to actors!– Designate how to handle the next message
Stateful Server
• Gatekeeper!– Manages the local list of active workers!– Spawns new workers!
!• Worker!– Manages the states for a player!– Optimistic locking!– Persist state after period of inactivity
Stateful Server
Game Server
Player A
Player B
S3Worker C
Worker B
GatekeeperR
eque
st H
andl
ers
Asynchronous
Stateful Server
Game Server
Worker C
Worker B
Gatekeeper
Worker Aok
Req
uest
Han
dler
sPlayer A
Player B
Asynchronous
S3
Stateful Server
Game Server
S3Worker C
Worker B
Gatekeeper
Worker AR
eque
st H
andl
ers
Player A
Player B
Asynchronous
Stateful Server
Game Server
S3Worker C
Gatekeeper
Worker AR
eque
st H
andl
ers
Player A
Player B
Asynchronous
Stateful Server
Game Server
Worker C
Worker A
Gatekeeper
error
Req
uest
Han
dler
sPlayer A
Player B
S3
Asynchronous
type Agent<‘T> = MailboxProcessor<‘T>!!!!
type Agent<‘T> = MailboxProcessor<‘T>!!type Message = ! | Get of AsyncReplyChannel<…>! | Put of State * Version * AsyncReplyChannel<…>
type Agent<‘T> = MailboxProcessor<‘T>!!type Message = ! | Get of AsyncReplyChannel<…>! | Put of State * Version * AsyncReplyChannel<…>
type Result<‘T> =! | Success! of ’T! | Failure! of Exception!!!!
type Result<‘T> =! | Success! of ’T! | Failure! of Exception!!type GetResult = Result<State * Version>!type PutResult = Result<unit>!
type Agent<‘T> = MailboxProcessor<‘T>!!type Message = ! | Get of AsyncReplyChannel<GetResult>! | Put of State * Version * AsyncReplyChannel<PutResult>
type Agent<‘T> = MailboxProcessor<‘T>!!type Message = ! | Get of AsyncReplyChannel<GetResult>! | Put of State * Version * AsyncReplyChannel<PutResult>
type Worker (playerId) =! let agent = Agent<Message>.Start(fun inbox ->! let state = getCurrentState playerId!! let rec workingState (state, version) = ! async { … }! and closedState () =! async { … }!! workingState (state, 1))
type Worker (playerId) =! let agent = Agent<Message>.Start(fun inbox ->! let state = getCurrentState playerId!! let rec workingState (state, version) = ! async { … }! and closedState () =! async { … }!! workingState (state, 1))
type Worker (playerId) =! let agent = Agent<Message>.Start(fun inbox ->! let state = getCurrentState playerId!! let rec workingState (state, version) = ! async { … }! and closedState () =! async { … }!! workingState (state, 1))
type Worker (playerId) =! let agent = Agent<Message>.Start(fun inbox ->! let state = getCurrentState playerId!! let rec workingState (state, version) = ! async { … }! and closedState () =! async { … }!! workingState (state, 1))
type Worker (playerId) =! let agent = Agent<Message>.Start(fun inbox ->! let state = getCurrentState playerId!! let rec workingState (state, version) = ! async { … }! and closedState () =! async { … }!! workingState (state, 1))
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! | None -> ! do! persist state! return! closedState()! …! }!
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! | None -> ! do! persist state! return! closedState()! …! }!
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! | None -> ! do! persist state! return! closedState()! …! }!
non-blocking I/O
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! | None -> ! do! persist state! return! closedState()! …! }!
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! | None -> ! do! persist state! return! closedState()! …! }!
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! …! | Some(Get(reply)) ->! reply.Reply <| Success(state, version)! return! workingState(state, version)! …! }!
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! …! | Some(Put(newState, v, reply)) when version = v ->! reply.Reply <| Success()! return! workingState(newState, version+1)! …! }
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! …! | Some(Put(_, v, reply)) ->! reply.Reply <| Failure(VersionMismatch(version, v))! return! workingState(state, version)! }
let rec workingState (state, version) = ! async { ! let! msg = inbox.TryReceive(60000)! match msg with! | None ! ! ! ! ! ! -> …! | Some(Get(reply)) ! ! ! ! -> …! | Some(Put(newState, v, reply)) when version = v -> …! | Some(Put(_, v, reply)) ! ! ! -> …! }
and closedState () = ! async { ! let! msg = inbox.Receive()! match msg with! | Get(reply) ->! reply.Reply <| Failure(WorkerStopped)! return! closedState()! | Put(_, _, reply) ->! reply.Reply <| Failure(WorkerStopped)! return! closedState()! }
and closedState () = ! async { ! let! msg = inbox.Receive()! match msg with! | Get(reply) ->! reply.Reply <| Failure(WorkerStopped)! return! closedState()! | Put(_, _, reply) ->! reply.Reply <| Failure(WorkerStopped)! return! closedState()! }
5x efficient improvement
60% latency drop
no databases
90% cost saving
need to ensure server affinity
need to balance load
need to avoid hotspots
• Persist player state after short inactivity!• Move player to another server after persistence
need to gracefully scale down
Recap
Agents!• no locks!• async message passing!
!
Agents!• no locks!• async message passing!• self-contained!• easy to reason with
Agents!• low level!• exceptions kills agents!• primitive error monitoring!• no supervision!• no distribution
MS Orleans
Cricket
akka.Net
Async Workflow!• non-blocking I/O!• no callbacks
Pattern Matching!• clear & concise
Caught a Gnome
EXP Item Gold
Quest Progress
Caught a Gnome
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
New Quest
Quest Complete
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New Quest
Level Up
Quest Progress
New QuestAchievement
Progress
EXP Item Gold
Quest CompleteLevel Up
Caught a Gnome
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New QuestAchievement
Progress
Achievement Complete
Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New QuestAchievement
Progress
Achievement Complete
100+ actions
triggered by different abstraction layers
non-functional requirements
message-broker pattern
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
Ignore
Process
Process
Process
Process
Ignore
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Process
Ignore
Ignore
Ignore
Process
Ignore
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Level Up
Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner Reporting
EXPItemGold
Level Up
need lots of facts
type Fact =! | GotExp! ! of Exp! | GotGold! ! of Gold! | GotItem! ! of Item * Count! | CaughtMonster! of Monster * Bait * Location! | LevelUp! ! of OldLevel * NewLevel! …
type Reward =! | GotExp! ! of Exp! | GotGold! ! of Gold! | GotItem! ! of Item * Count!!
type StateChange =! | LevelUp! ! of OldLevel * NewLevel! …
type Fact =! | StateChange! of StateChange! | Reward! ! of Reward! | Trapping! ! of Trapping! | Fishing! ! of Fishing! …
let process fact =! match fact with! | StateChange(stateChange)! -> …! | Reward(reward)! ! ! -> …! | Trapping(trapping)! ! -> …! …
C# interop
…!var fact = Fact.NewStateChange(!! StateChange.NewLevelUp(oldLvl, newLvl));!…
type IFact = interface end!!
type Reward =! | GotExp! ! of Exp! | GotGold! ! of Gold! | GotItem! ! of Item * Count! interface IFact
type IFact = interface end!!
type Reward =! | GotExp! ! of Exp! | GotGold! ! of Gold! | GotItem! ! of Item * Count! interface IFact
let process (fact : IFact) =! match fact with! | :? StateChange ! as stateChange!-> …! | :? Reward ! ! as reward! ! -> …! | :? Trapping! ! as trapping! -> …! …! | _ -> raise <| NotSupportedFact fact
let process (fact : IFact) =! match fact with! | :? StateChange ! as stateChange!-> …! | :? Reward ! ! as reward! ! -> …! | :? Trapping! ! as trapping! -> …! …! | _ -> raise <| NotSupportedFact fact
simple
flexible
saver
location baitattraction rate
catch rate
auto-tuning trapping stats
Monsterstrength speed
intelligence
Trapstrength speed
technology
Monsterstrength speed
intelligence
Trapstrength speed
technology
Catch Rate %
trial-and-error
genetic algorithms
F#-powered DSLs.!Awesome!
wanna find correlations?
wanna find correlations?
you can DIY it! !
;-)
why was payment service slow last night?
Pain Driven Development
Amazon!! .CloudWatch!! .Selector
github.com/fsprojects/Amazon.CloudWatch.Selector
Find metrics whose 5 min average exceeded
1 second during last 12 hours
cloudWatch.Select( unitIs “milliseconds” + average (>) 1000.0 @ last 12 hours |> intervalOf 5 minutes)
cloudWatch.Select(“ unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes”)
did any cache nodes’ CPU spike
yesterday?
cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes)
cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes)
regex support
Amazing
F# =
usable from anywhere you can run F# code
e.g. F# REPL, executable, ..
Internal DSL
useful for building tools e.g. CLI, …
External DSL
Recap
managed!key-value store
redundancy!9-9s guarantee
great performance
name your throughput
hash key + range key
hash_key_1
hash_key_2
hash_key_3
range_key_1
range_key_2
range_key_1
range_key_2
range_key_1
range_key_2
Hash Key Range Key
Hash Key Range Key
Local Secondary Index
Hash Key Range Key
Local Secondary Index
Global Secondary Index
query:!! given a hash key!! filter on!! ! range key, or!! ! local secondary index
Hash Key Range Key
Local Secondary Index
Global Secondary Index
Who are the TOP 3 players in “Starship X” with a score of at least 1000?
GOAL
Disguise complexity
GOAL
SELECT UserId, TopScore FROM GameScore WHERE GameTitle CONTAINS “Zelda” ORDER DESC!LIMIT !! 3 WITH (NoConsistentRead)
Query
AST
Execution
SELECT * FROM GameScore
Abstract Syntax Tree (AST)
FParsec
select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
S3 Provider
github.com/fsprojects/S3Provider
F# type provider for Amazon S3
compile time validation
no code generation
a long time ago, in a language far far away from .Net - Erlang.
there was an idea to let you pattern match again binary
payloads and work with them at a bit level…
<<SourcePort:16, DestinationPort:16, ! SeqNumber:32,! AckNumber:32, DataOffset:4, ! Reserved:3, Flags:9, WindSize:16,! CheckSum:16, UrgentPointer:16,! Payload/binary>> = SomeBinary
<<SourcePort:16, DestinationPort:16, ! SeqNumber:32,! AckNumber:32, DataOffset:4, ! Reserved:3, Flags:9, WindSize:16,! CheckSum:16, UrgentPointer:16,! Payload/binary>> = SomeBinary
<<SourcePort:16, DestinationPort:16, ! SeqNumber:32,! AckNumber:32, DataOffset:4, ! Reserved:3, Flags:9, WindSize:16,! CheckSum:16, UrgentPointer:16,! Payload/binary>> = SomeBinary
<<SourcePort:16, DestinationPort:16, ! SeqNumber:32,! AckNumber:32, DataOffset:4, ! Reserved:3, Flags:9, WindSize:16,! CheckSum:16, UrgentPointer:16,! Payload/binary>> = SomeBinary
<<SourcePort:16, DestinationPort:16, ! SeqNumber:32,! AckNumber:32, DataOffset:4, ! Reserved:3, Flags:9, WindSize:16,! CheckSum:16, UrgentPointer:16,! Payload/binary>> = SomeBinary
<<SourcePort:16, DestinationPort:16, ! SeqNumber:32,! AckNumber:32, DataOffset:4, ! Reserved:3, Flags:9, WindSize:16,! CheckSum:16, UrgentPointer:16,! Payload/binary>> = SomeBinary
BitSyntax
github.com/theburningmonk/bitsyntax
Working with Stream at bit level
do bitWriter stream {! do! BitWriter.WriteInt16(12345s) ! // source port! do! BitWriter.WriteInt16(12321s)! // destination port! do! BitWriter.WriteInt32(1) ! // sequence number! do! BitWriter.WriteInt32(1) ! // ack number! do! BitWriter.WriteInt32(2, numBits = 4) // data offset! do! BitWriter.WriteInt32(0, numBits = 3) // reserved! do! BitWriter.WriteBool(false) ! // NS! …! do! BitWriter.WriteInt16(42s)! ! // Checksum! do! BitWriter.WriteInt16(1208s)! // Urgent Pointer! do! BitWriter.WriteString("Hello World!") // payload!}
do bitWriter stream {! do! BitWriter.WriteInt16(12345s) ! // source port! do! BitWriter.WriteInt16(12321s)! // destination port! do! BitWriter.WriteInt32(1) ! // sequence number! do! BitWriter.WriteInt32(1) ! // ack number! do! BitWriter.WriteInt32(2, numBits = 4) // data offset! do! BitWriter.WriteInt32(0, numBits = 3) // reserved! do! BitWriter.WriteBool(false) ! // NS! …! do! BitWriter.WriteInt16(42s)! ! // Checksum! do! BitWriter.WriteInt16(1208s)! // Urgent Pointer! do! BitWriter.WriteString("Hello World!") // payload!}
16 bits
do bitWriter stream {! do! BitWriter.WriteInt16(12345s) ! // source port! do! BitWriter.WriteInt16(12321s)! // destination port! do! BitWriter.WriteInt32(1) ! // sequence number! do! BitWriter.WriteInt32(1) ! // ack number! do! BitWriter.WriteInt32(2, numBits = 4) // data offset! do! BitWriter.WriteInt32(0, numBits = 3) // reserved! do! BitWriter.WriteBool(false) ! // NS! …! do! BitWriter.WriteInt16(42s)! ! // Checksum! do! BitWriter.WriteInt16(1208s)! // Urgent Pointer! do! BitWriter.WriteString("Hello World!") // payload!}
32 bits
do bitWriter stream {! do! BitWriter.WriteInt16(12345s) ! // source port! do! BitWriter.WriteInt16(12321s)! // destination port! do! BitWriter.WriteInt32(1) ! // sequence number! do! BitWriter.WriteInt32(1) ! // ack number! do! BitWriter.WriteInt32(2, numBits = 4) // data offset! do! BitWriter.WriteInt32(0, numBits = 3) // reserved! do! BitWriter.WriteBool(false) ! // NS! …! do! BitWriter.WriteInt16(42s)! ! // Checksum! do! BitWriter.WriteInt16(1208s)! // Urgent Pointer! do! BitWriter.WriteString("Hello World!") // payload!}
take ’first’ N bits
bitReader stream {! let! srcPort ! = BitReader.ReadInt32(numBits = 16)! let! destPort ! = BitReader.ReadInt32(numBits = 16)! let! seqNum ! = BitReader.ReadInt32(numBits = 32)! let! ackNum ! = BitReader.ReadInt32(numBits = 32)! let! dataOffset ! = BitReader.ReadInt32(numBits = 4)! let! _ ! = BitReader.ReadByte(numBits = 3)! …! let! payload ! = BitReader.Rest(UTF8.GetString)!! return srcPort, destPort, … payload!}
bitReader stream {! let! srcPort ! = BitReader.ReadInt32(numBits = 16)! let! destPort ! = BitReader.ReadInt32(numBits = 16)! let! seqNum ! = BitReader.ReadInt32(numBits = 32)! let! ackNum ! = BitReader.ReadInt32(numBits = 32)! let! dataOffset ! = BitReader.ReadInt32(numBits = 4)! let! _ ! = BitReader.ReadByte(numBits = 3)! …! let! payload ! = BitReader.Rest(UTF8.GetString)!! return srcPort, destPort, … payload!}
packs into 32 bit int
Byte1!Byte2!00000000!00000000
memory address
Byte1!Byte2!
bitReader stream {! let! srcPort ! = BitReader.ReadInt32(numBits = 16)! let! destPort ! = BitReader.ReadInt32(numBits = 16)! let! seqNum ! = BitReader.ReadInt32(numBits = 32)! let! ackNum ! = BitReader.ReadInt32(numBits = 32)! let! dataOffset ! = BitReader.ReadInt32(numBits = 4)! let! _ ! = BitReader.ReadByte(numBits = 3)! …! let! payload ! = BitReader.Rest(UTF8.GetString)!! return srcPort, destPort, … payload!}
packs into a byte
0 1 0 1 1 0 1 1
…!let! dataOffset = BitReader.ReadInt32(numBits = 4)!…
0 1 0 1 1 0 1 1
…!let! dataOffset = BitReader.ReadInt32(numBits = 4)!…
0 1 1 0 1 1 0 0
<<< 2
0 1 0 1 1 0 1 1
…!let! dataOffset = BitReader.ReadInt32(numBits = 4)!…
0 1 1 0 1 1 0 0
<<< 2
0 0 0 0 0 1 1 0
>>> 4
0 0 0 0 0 1 1 0
…!let! dataOffset = BitReader.ReadInt32(numBits = 4)!…
6uy
…!let! dataOffset = BitReader.ReadInt32(numBits = 4)!…
6uy
6
padding padding padding byte
0 0 0 0 0 1 1 0
1 0 0 1 0 0 0 1
…!let! dataOffset = BitReader.ReadInt32(numBits = 4)!…
1 0 0 0 1 0 1 1
1 0 0 1 0 0 0 1
…!let! dataOffset = BitReader.ReadInt32(numBits = 17)!…
1 0 0 0 1 0 1 1
1 1 0 1 1 0 1 0
0 1 0 0 1 0 0 0
Filbert
github.com/fsprojects/Filbert
BERT serializer and BERT-RPC client
@theburningmonktheburningmonk.comgithub.com/theburningmonk