Scaling Source Control for the Next Generation of Game Development

Scaling Source Control for NextGen Game Development 1

2007 Perforce User’s Conference

Scaling Source Control for the Next Generation of Game Development

Toby RobertsMike Sundy


Overview• Current Gen vs. NextGen

• Test Methodology

• Test Results

• Lessons Learned

• Future of P4 at EA


Challenges in Game Industry• CurrentGen vs. NextGen

• 9 GB for DVD's for CurrentGen. (Xbox, PS2)

• 30-50 GB for NextGen (Blu-Ray, HD-DVD – XBox 360, PS3) - 3-5x increase in storage capacity

• Asset Volume Explosion

• 50k clientspec grew to 500k clientspec

• Build Farms

• 20-30 machines grew to 100+

• Team Sizes

• Outsourcing


EA Redwood Shores (EARS)P4 Installation Metrics

• 11 Perforce Servers, 1200 users (90+ servers, 4,000+ users at EA)

• 7 million files on one server, 80 GB db.have

• Store all source code and source data in P4

• 1.5 TB on one server, 7 TB total P4 storage

• Avg. client workspace – 80-120GB and 250k-500k files

• 2,500 – 66,000 syncs per day

• Titles include Sims, Godfather, Simpsons, and Spore.


Problems with Windows• 2 GB per process memory limit

• Filehandle leak (crashed up to 6 times per day)

• Performance during large concurrent syncs

• 40 GB sync went from 40 min. to 3+ hours

• NTFS

• Threads and troubleshooting

• Patching


Goals of Project

• Improve stability

• Improve scalability

• Solution that works for our dev environment, infrastructure, and team cultures


Parameters of Project• Dell and STK hardware

• Intel

• RedHat

• Windows Desktops

• 95% of user machines

• Gig network


Test Methodology

• Single operation vs. load

• P4 benchmarking tools

• fstst and locktest

• Homegrown scripts

• Force sync of 39 GB data, 180k files

• p4.exe on Windows


Configuration Overview

• Tested 21 different server configurations.

• Windows vs. Linux

• 32-bit vs. 64-bit

• NTFS v. ext3 v. ReiserFS v. XFS

• 10k vs. 15k drives


db locktest

0

1

2

3

4

5

6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

config

Seco

nds


db 8k read-write

0

100

200

300

400

500

600

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

config

MB

\s


rcs 64k write

0

100

200

300

400

500

600

700

800

900

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

config

MB

\s


rcs 64k read

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

config

MB

\s


1 force sync avg.

0 5 10 15 20 25 30 35 40 45 50

1

3

5

7

9

11

13

15

17

19

21

conf

ig

minutes


3 force syncs avg.

0 10 20 30 40 50 60 70 80

1

3

5

7

9

11

13

15

17

19

21

conf

ig

minutes


5 force syncs avg.

0 10 20 30 40 50 60 70 80

1

3

5

7

9

11

13

15

17

19

21

conf

ig

minutes


% degradation from 1 to 3 syncs

0 0.1 0.2 0.3 0.4 0.5 0.6

1

3

5

7

9

11

13

15

17

19

21

conf

ig

percent


edit loop avg. (176 files)

0 0.2 0.4 0.6 0.8 1 1.2

1

3

5

7

9

11

13

15

17

19

21

conf

ig

seconds


Test Results Summary• DB operations 5-10x faster on 64-bit Linux on XFS than

Windows 32-bit.

• Linux sync performance under load degraded only 5% instead of the 450% we saw with Windows.

• XFS outperforms ext3 and even edges ReiserFS for data syncs from the repository volume.


dm-CommitSubmit

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

win-gf1_1666

win-gf1_1777

linux-gf1_1666co

nfig

seconds


avg. edit

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

win-gf1_1666

win-gf1_1777

linux-gf1_1666co

nfig

seconds


avg. fstat

0 0.1 0.2 0.3 0.4 0.5 0.6

win-gf1_1666

win-gf1_1777

linux-gf1_1666co

nfig

seconds


avg. sync compute phase

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

win-gf1_1666

win-gf1_1777

linux-gf1_1666co

nfig

seconds


Configuration Lessons Learned• OS

• 2.6 kernel was 2 – 4x faster than 2.4 in some tests.

• RAID

• RAID 5 best for RCS due to the high number of syncs/reads.

• DB benefited from RAID 1 or RAID 10.

• Filesystem

• XFS proved to be significantly faster than ext3 and beat Reiser by 30% in some tests.

• Drive Speed

• 20% less degradation with 15k drives during multiple concurrent forced syncs due to higher I/O capacity.


Performance Principles

• Keep your db’s on their own disks

• Proxy heavy sync users such as build farms or remote locations

• Best place to improve hardware performance is memory and disk I/O for db’s

• There is a big difference in filesystem speed


Success Stories• New standard has had a huge impact on our local team

productivity and P4 performance across EA.

• New platform has scaled well for 1,000 user team and could scale to larger teams.

• Stability is great!

• No more memory or file descriptor limits

• Can identify and kill individual threads.

• $5 million saved per year in productivity across EA.


NextGen EA P4 Architecture

• P4 Server setup

• Distributed Development

• Proxy Build Farm (30% gain)

• Case-insensitivity

• Disk to disk mirroring


The Future of Perforce at EA• RAID 50 (2x faster locking than RAID 10)

• SAS SFF drives

• AD2P4

• Multi-Node Cluster

• Perforce + binary delta copy technology

• Workspace mirroring

• Single View of Perforce Servers


Thanks

• Michael Shields

• Brett Taylor

• Kenny Wu


Trivia for Schwag

Questions?

Scaling Source Control for the Next Generation of Game Development

Technology

Transcript of Scaling Source Control for the Next Generation of Game Development