[NetherRealm Studios] Game Studio Perforce Architecture
-
Upload
perforce -
Category
Technology
-
view
194 -
download
1
description
Transcript of [NetherRealm Studios] Game Studio Perforce Architecture
![Page 1: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/1.jpg)
1
Game Studio Perforce Architecture
Ryan Mensching Senior Network Administrator NetherRealm Studios/WB Games Logo area
![Page 2: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/2.jpg)
2
The Setup
• Established Studio with track record • Mortal Kombat (1992 – 2011) • DC Comics – Injustice (2013), Arkham City Lockdown
• AAA Games on multiple platforms • 360 • PS3 • WiiU • Vita • iOS
![Page 3: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/3.jpg)
3
Historical Perspective
• Team has been making games for 20 plus years. • “Things will get done.” • Outages cost money, ship dates, and sanity. • “Crunch” is a reality. • Multi-Year development cycle, two years • Used other tools in the past, Source Safe
![Page 4: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/4.jpg)
4
Statistics
• Games are Code and Binary Art Assets • Textures • Animations • Cinematics • Lighting
• Sync of current project at head is 20G • Sync of source art assets is 545G
![Page 5: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/5.jpg)
5
Visuals
• 460+ CL’s per day at peek intervals (1000’s of builds) • Some 20G of changes per day at peek intervals
![Page 6: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/6.jpg)
6
Products
• As a studio we have 3 software products that are being produced at any given time. • Engine • Games • Art Tool/Game Editor
![Page 7: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/7.jpg)
7
Challenges
• As IT we had several hurdles to overcome • Aging Hardware • Aging processes • Management woes • Insight into systems • Lack of agility to meet business demand
![Page 8: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/8.jpg)
8
Strategic Goals
![Page 9: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/9.jpg)
9
Strategic Goals
• Game development support is a balancing of immediate gratification and resources.
• IT business driven goals/pillars • Space • Speed • Flexibility • Reliability • Management
![Page 10: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/10.jpg)
10
Space
• Available, can’t be used if its locked up, i.e. thin provisioning
• Abstraction, array layer should not need configuration to allocate space and at volume layer should not require reconfig for volume expansion.
![Page 11: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/11.jpg)
11
Speed
• Entire VM platform, OS, and mass storage. • Performance needs to be maintained in relation
to consumption.
![Page 12: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/12.jpg)
12
Flexibility
• Works in conjunction with other pillars. • Stands alone as a goal in games environment. • Freedom of protocol choices. • Ease of VM and hardware additions.
![Page 13: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/13.jpg)
13
Reliability
• Redundancy should be inherent. • Should not require special plans or procedures. • If the reliability path is to complex issues will
arise.
![Page 14: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/14.jpg)
14
Management
• IT Glue • Proper management makes all the pillars come
together. • Self documenting
![Page 15: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/15.jpg)
15
Solutions How as an IT organization do we accomplish these goals and vision?
![Page 16: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/16.jpg)
16
Solutions
• Solid Layer 2 and 3 core with 10G distribution.
• Previous experience with VM, Storage over IP and infrastructure testing.
![Page 17: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/17.jpg)
17
Research
• Built VM farm with existing hardware and repurposed array for iSCSI.
• Benchmarks were in the top on the Perforce Benchmark DB.
• Ran real world project on systems.
![Page 18: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/18.jpg)
18
![Page 19: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/19.jpg)
19
Storage Platform
• Key building block of infrastructure design. • iSCSI management proved not agile enough,
LUN management chores. Redundancy requires complex configurations.
• NFS provided equal or better performance.
![Page 20: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/20.jpg)
20
NFS
• True thin provisioning, space consumed and given back.
• No LUN changes. • Volumes size can be adjusted on the fly. • Strong VMware support. • NetApp engine
• Snapshots • Native NFS among other protocols
• Key to P4 revision file storage design.
![Page 21: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/21.jpg)
21
Virtualization
• VM hardware needs to be “wide and fast”. • As much RAM as possible to improve VM operations
and better disk caching. • Cores should have highest clock speed possible. • Perforce uses 2 vCPUs. • Cisco UCS
• Hardware abstraction • Native Unified Fabric, 10G • VM Passthrough
![Page 22: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/22.jpg)
22
OS
• RHEL for all Perforce servers and many applications.
• Linux provides best usage of resources for VM’s and disk caching.
• Current project Perforce servers have 32GB of RAM.
• Able to achieve 440+Mb sustained transfer to p4 clients.
![Page 23: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/23.jpg)
23
Reliability
• Redundancy at array level with NetApp SnapMirror. • Volume and VM level snapshots.
• Provides ability to single file restore directly back to depots/servers.
• UCS blades multi-homed to core; dual chassis uplinks, dual fabric uplinks, dual core switches.
• VMware HA clustering • Perforce uses proxies for load balancing and
checkpoints for integrity. • Moving to replicas.
![Page 24: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/24.jpg)
24
Management Tools
• NetApp OnCommand • UCS Manager • VMware
• Plugins, SMVI
• Active monitoring of P4 services and disk health.
![Page 25: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/25.jpg)
25
![Page 26: [NetherRealm Studios] Game Studio Perforce Architecture](https://reader034.fdocuments.in/reader034/viewer/2022052619/555d704bd8b42a0b778b516d/html5/thumbnails/26.jpg)
26
Conclusion
• Perforce is very capable of scaling to speed and data intense development.
• Requires investments in engineering and infrastructure.
• Each environment is different, but can share philosophies.
• Abstraction, abstraction, abstraction