[CCP Games] Versioning Everything with Perforce
-
Upload
perforce -
Category
Technology
-
view
136 -
download
3
description
Transcript of [CCP Games] Versioning Everything with Perforce
![Page 1: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/1.jpg)
1
Versioning Everything with Perforce
Halldor Fannar CTO CCP
Logo area
![Page 2: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/2.jpg)
2
Facts and figures
Released in 2003 for Windows PC Grown year-on-year to 500,000+ subscribers Record PCU of 63,170 players
Company founded in 1997 Privately held Over 500 employees worldwide
Released 2012 on Sony PS3 (in beta) Free-to-play with microtransactions
![Page 3: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/3.jpg)
3
Connecting Dust and Eve
Play short movie showing DUST & EVE
![Page 4: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/4.jpg)
4
Perforce server is our Definitive Software Library
• Art assets (audio, textures, geometry, shaders, etc) • Built binaries (executables, dynamic libs, static data) • External libraries and tools • Configuration files (settings, ini files, etc) • Text (UI messages, translations, etc)
Branching is used where it makes sense!
![Page 5: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/5.jpg)
5
Versioning art assets
//depot/content/eve //depot/games/branches
model.mb image1.psd image2.psd image3.psd
EVE-‐TRILAMBDA
[file sizes ~100 MB]
model.gr2 image1.dds image2_and_3.dds
[file sizes ~1 MB] settings.red
![Page 6: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/6.jpg)
6
• Ar*sts want technology that is user friendly and provides seamless workflow
• Visual tools p4win & p4v are good but … • Crea*ng/managing client specs is a burden • Selec*ng the right folder/branch to sync is a burden
• Without any support this is how they make it “seamless”: • Create a “god spec” • Sync the root folder
Artists and versioning
![Page 7: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/7.jpg)
7
• The problem isn’t restricted to ar*sts – we have 3rd party tools and libraries that we don’t branch (e.g. //depot/games/sdk)
• Temp*ng to go for a ‘god spec’ in such an environment
• Because syncing from two loca*ons is annoying • Because crea*ng new client specs is annoying
→ We remove those annoying bits.
Shared and branched files
![Page 8: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/8.jpg)
8
Streamlining Perforce user experience
![Page 9: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/9.jpg)
9
Stream-like branching model
MAIN
EVE-‐DX11
EVE-‐TRILAMBDA
EVE-‐JAPAN
EVE-‐TRANQUILITY EVE-‐2012-‐R2
EVE-‐2012-‐R2-‐ALBERTO
EVE-‐2012-‐R2-‐BERYL
sandbox staging release
![Page 10: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/10.jpg)
10
Perforce streams are popular on smaller projects
//eveLauncher
![Page 11: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/11.jpg)
11
Perforce streams are popular on smaller projects
//virtualGoodsService
![Page 12: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/12.jpg)
12
No one size fits all
![Page 13: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/13.jpg)
13
• Treat all your configura*on values as code and version them in source control
• Otherwise you don’t really have a Defini*ve So[ware Library
Versioning your configuration
![Page 14: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/14.jpg)
14
Nobody is born a versioning hero – you become one after a freak software accident
![Page 15: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/15.jpg)
15
• Myth: “Built binaries have no place in source control” • Usual reasoning:
• Because it takes up so much space • Because your build process should be perfectly repeatable and you
are then storing redundant informa*on
• Perforce is our Defini*ve So[ware Library (DSL) for development – don’t spread your development so[ware across Perforce, file share folders, [p sites, and a bunch of DVDs
Versioning built binaries
![Page 16: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/16.jpg)
16
Example: Python packages
• Checked into //pythonPackages • Automagically appear on http://pypi • Bi-winning:
• Makes the desired behavior convenient • Offers two deployment models
![Page 17: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/17.jpg)
17
Example: External libraries
• Make Perforce your one stop shop for SDKs • Applying that policy to the reference PNG library, libpng:
Transform with a branch spec: sdk/libpng<-‐sdk_installs/libpng
![Page 18: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/18.jpg)
18
Example: External library requiring more tricks
• Applying our policy to Microso[’s DirectX is more challenging • Crea*ng the “easy to consume” compile *me environment is easy
![Page 19: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/19.jpg)
19
Example: External library requiring more tricks
• Crea*ng the right run*me environment is the tricky part (system lib) • O[en developers will ask you to manually do this:
![Page 20: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/20.jpg)
20
Example: External library requiring more tricks
• Create a bootstrapping sequence for automa*c silent installa*on:
Import failure
Import library that depends on run*me
• Store the silent installer in Perforce for each version of the SDK • Developers can just sync and run! (same flow for our customers)
no
Execute silent installer for run*me
yes
![Page 21: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/21.jpg)
21
Yes, it’s extra work but every developer will thank you Actually, they won’t because they won’t notice a thing
![Page 22: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/22.jpg)
22
• “Sta*c data” covers all game related data that doesn’t change during play and isn’t captured in industry standard formats
• Typically this data describes the construc*on of an object or a game level by referring to external binary files and providing specific sekngs for how they are combined
Versioning static data
![Page 23: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/23.jpg)
23
• 2002: Sta*c data is stored unversioned in an MS SQL DB • 2003-‐2011: Explosive growth and explosive issues with this approach • 2012: Versioning of sta*c data moves over to Perforce
Abridged history of static data at CCP
![Page 24: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/24.jpg)
24
• Syntax for code well understood and helps us with resolving conflicts and merging
• Compiler can most o[en tell us when we botched the resolve or merge • File granularity for code projects is usually reasonable and known
upfront, thus reducing chances of conflic*ng changes • Organiza*on within a code file follows rules and conven*ons that also
reduce chances of conflicts
How does this map to sta9c data?
Code versus Static data
![Page 25: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/25.jpg)
25
• It doesn’t map! • Using a typical text merging tool on XML-‐like data will most o[en end in
tears
Code versus Static data
![Page 26: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/26.jpg)
26
products: - sku : BL394D quantity : 4 description : Basketball price : 450.00 - sku : BL4438H quantity : 1 description : Super Hoop price : 2392.00
• We use YAML (less cru[)
Stable ordering of items (sets rather than lists)
Uniquely iden*fiable objects
Making static data merge-able
![Page 27: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/27.jpg)
27
Custom merge and diff tool
![Page 28: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/28.jpg)
28
• Mul*ple smaller files reduce chances of resolving – but inefficient for the game to load and tough on your file system
• Use automated build steps to transform the data from the authoring/merge-‐able format to an op*mal run*me format (also versioned in Perforce)
• Build process also verifies the referen*al integrity of the data across files and validates that each file is not viola*ng its schema
Solving other static data issues
![Page 29: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/29.jpg)
29
What about dynamic data?
• Of course stored in a DB • No branching support so changes must be backwards
compatible • Schema and stored procedures are versioned in
Perforce • CI servers monitor all changes and can verify
backwards compatibility
![Page 30: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/30.jpg)
30
Catch the problem at the source - not downstream
![Page 31: [CCP Games] Versioning Everything with Perforce](https://reader033.fdocuments.in/reader033/viewer/2022052523/555d6fb5d8b42a687b8b4f84/html5/thumbnails/31.jpg)
31
Versioning everything requires an investment It’s worth it.