Launching and Supporting Killzone Shadow Fall
Transcript of Launching and Supporting Killzone Shadow Fall
![Page 1: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/1.jpg)
1
Launching and Supporting Killzone Shadow Fall
Jorrit Rouwé[email protected]
Lead Game Tech, Guerrilla Games
![Page 2: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/2.jpg)
2
Introduction• Killzone Shadow Fall is a First Person Shooter• 24 player team based gameplay• Users can create game modes• PS4 launch title
![Page 3: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/3.jpg)
3
Before Launch
![Page 4: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/4.jpg)
4
Road to Launch• Previous Title: Killzone 3– Closed beta (~100 players within Sony)– Semi public beta (~15K players invited)– Public beta (~25K players concurrent)
• PS4 launch– Limited hardware– Max ~100 player test
![Page 5: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/5.jpg)
5
Development Focus• Telemetry• Load testing• Monitoring• Kill Switches
![Page 6: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/6.jpg)
6
Telemetry – Collecting Game Metrics
Game
Server
TelemetryEvent
Collector
RedshiftTableau
Visualization
Processing
Storage
![Page 7: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/7.jpg)
7
Frame Rate Heat Map – The Terminal
55 FPS: Problem area!
![Page 8: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/8.jpg)
8
SP - Average Deaths Per Player Per Level
![Page 9: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/9.jpg)
9
MP - Death Balance – The Forest
VSA Spawn
HGH Spawn
VSA failing to spawn camp
Objective CObjective B
Objective A
HGH failing to spawn camp
HGH being spawn camped
100%HGH
100%VSA
![Page 10: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/10.jpg)
10
Development Focus• Telemetry• Load testing• Monitoring• Kill Switches
![Page 11: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/11.jpg)
11
Load Testing – What to Test• Functionality– Match making (e.g. day night cycle, featuring game mode)– Score posting– Leaderboards– …
• Failures– Loss of server– Temporary loss of connectivity– PlayStation®Network outage / slow response– Server update
![Page 12: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/12.jpg)
12
Load Testing – Framework
Sim 1..N
Game Server
Sim N+1..2N
PlayStation®Network
![Page 13: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/13.jpg)
13
Load Testing – Framework
Sim 1..N
Game Server
Sim N+1..2N
Mock PlayStation®Network
![Page 14: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/14.jpg)
14
Load Testing – Framework
Sim 1..N
Game Server
Sim N+1..2N
Mock PlayStation®Network
Share state & validation
![Page 15: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/15.jpg)
15
Load Testing – Framework
Sim 1..N
Game Server
Sim N+1..2N
Mock PlayStation®Network
Share state & validation
Telemetry &
Monitoring
BehavioralVerification
![Page 16: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/16.jpg)
16
Development Focus• Telemetry• Load testing• Monitoring• Kill Switches
![Page 17: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/17.jpg)
17
Monitoring• What to monitor– Player count– REST API timings / errors– Database throughput / errors– Connectivity between servers (game servers / PSN)– Low level OS (disc / CPU / memory / network)– Backups
• 24/7 on call support for failures
![Page 18: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/18.jpg)
18
Development Focus• Telemetry• Load testing• Monitoring• Kill Switches
![Page 19: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/19.jpg)
19
Kill Switches• Block apps (killzone.com, public API, live tiles)• Block individual REST APIs (e.g. post score)• Block hard to test things (e.g. game merging)• Block misc. components (e.g. telemetry)
![Page 20: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/20.jpg)
20
After Launch
![Page 21: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/21.jpg)
21
Match Making - What Players Expect• Low ping• Full games• Team balanced– Player count– Skill level
• A choice of game modes• No dropped connections
![Page 22: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/22.jpg)
22
Average Ping Between Regions
![Page 23: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/23.jpg)
23
Average Ping Between Regions - Conclusion• Matchmaking tries to avoid cross region!– Smaller regions / game modes don’t have enough players– Friends may want to play cross region– 98%+ success in large, 80% in smallest region
![Page 24: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/24.jpg)
24
Ping – Cable vs WirelessDropped connectionsPlay Time
WiredWireless
Wireless ± 15 ms slower!Educate players!
![Page 25: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/25.jpg)
25
The Hunt for Server Locations (Sao Paulo)
Sao Paulo
![Page 26: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/26.jpg)
26
The Hunt for Server Locations (Dallas)
![Page 27: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/27.jpg)
27
The Hunt for Server Locations (Istanbul)
![Page 28: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/28.jpg)
28
The Hunt For Server Locations – Conclusion• Geographically close does not mean low ping– Possible packet: US East → US West → US East– Very hard for players to understand– We don’t offer region selection
• Too many regions = too few players per region
![Page 29: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/29.jpg)
29
Match Making - What Players Expect• Low ping• Full games• Team balanced– Player count– Skill level
• A choice of game modes• No dropped connections
![Page 30: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/30.jpg)
30
Joining Full Games
Parties of more than 2 very rare!
8 player party → 8 vs 1 game.Split party?
Join alone = full game
![Page 31: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/31.jpg)
31
Match Making - What Players Expect• Low ping• Full games• Team balanced– Player count– Skill level
• A choice of game modes• No dropped connections
![Page 32: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/32.jpg)
32
Team Balance - ELO Glicko
Divide by time played
Discard not played long enough
![Page 33: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/33.jpg)
33
Team Balance - ELO Glicko
Win
![Page 34: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/34.jpg)
34
Team Balance - ELO Glicko
Loss
![Page 35: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/35.jpg)
35
Team Balance• Skill based separation = too many groups• Player count balance most important• Use ELO skill rating to – Determine team when sizes equal– Shuffle teams at end of round
![Page 36: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/36.jpg)
36
Match Making - What Players Expect• Low ping• Full games• Team balanced– Player count– Skill level
• A choice of game modes• No dropped connections
![Page 37: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/37.jpg)
37
A Choice of Game Modes
![Page 38: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/38.jpg)
38
Match Making - What Players Expect• Low ping• Full games• Team balanced– Player count– Skill level
• A choice of game modes• No dropped connections
![Page 39: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/39.jpg)
39
Errors Per Hour Over Time
PlayStation Plus checkPSN authentication
Buffer increase fix(A/B test)
![Page 40: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/40.jpg)
40
Errors Per Hour Per Router
Technicolor TG587:
UDP socket cannot connect to multiple IP.
![Page 41: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/41.jpg)
41
Conclusion• Had very little downtime• Kill switches used on few occasions• Match making is a hard problem• Telemetry helped a lot
![Page 42: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/42.jpg)
42
Questions?
Also see: ‘The Server Architecture Behind Killzone Shadow Fall’http://jrouwe.nl/serverarchkzsf/
![Page 43: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/43.jpg)
43
Bonus Slides
![Page 44: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/44.jpg)
44
Ping – Educate Players!
Max score not possible on wireless
![Page 45: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/45.jpg)
45
The Hunt for Server Locations (Milan)
![Page 46: Launching and Supporting Killzone Shadow Fall](https://reader035.fdocuments.in/reader035/viewer/2022070522/58eea2191a28aba1368b46ab/html5/thumbnails/46.jpg)
46
The Hunt for Server Locations (Frankfurt)