From Windows to Linux: Converting a Distributed Perforce Helix Infrastructure David FoglesongSenior Systems Software EngineerTableau Software

2

About Tableau Software

Tableau Software (NYSE: DATA) helps people see and understand data. Tableau helps anyone quickly analyze, visualize and share information. More than 35,000 customer accounts get rapid results with Tableau in the office and on-the-go. And tens of thousands of people use Tableau Public to share data in their blogs and websites. See how Tableau can help you by downloading the trial at www.tableau.com/trial.

3

Perforce At Tableau

Started with CVS. Converted to Perforce in 2007 (on Windows). Infrastructure has evolved over time: One server > commit

+ replicas > commit + replicas + edge servers. 650K+ changes, 900+ users, commit = 500G DB, edge =

800G DB. 3 main dev offices, several smaller offices.

4

Why Change?

Tableau product change from Win-only to Win/Mac to Win/Mac/mobile (iOS/Android).

Run other parts of build infrastructure (Artifactory, ReviewBoard, OpenGrok) on Linux.

Dev management wants “best” systems = performance, stability, scalability.

Perforce admins prefer Linux.

5

Before

1x commit server 1x RO replica (backup and reporting) 3x fwd replica (at main dev offices) Proxies at smaller dev offices. 1x edge server (for build farm) + RO replica (backup) Brokers in front of everything. All on Windows.

6

Before

7

After

1x commit server 2x RO replicas (backup and reporting) 5x edge servers (3x for offices, 2x for build farm) 5x RO replicas (backups for edge servers) Brokers in front of everything. All on Linux. Most in central data center. Proxies in dev offices.

8

After

9

Edge Servers vs. Forwarding Replicas

Why switch offices to edge servers?• Remote users (Palo Alto) see small lag on commands.• Want to move most user data (db.have) off commit server – goal is to have

minimal (ideally none) connections from human users on commit server. Move to cluster?• Not at this time. Want to preserve option of moving edge server to dev

office if needed.

10

Process: Old vs. New

Old way = checkpoint manipulation• Either via scripts or p4migrate.• Need to fix case inconsistencies in metadata. E.g.,

- //depot/source/foo.c#1 vs. //depot/SOURCE/bar.c#1- User “bob” vs. user “BOB”.

• Transfer and convert archive files.

11

Process: Old vs. New

New way = replication• Create Linux replicas and edge servers connected to Windows commit

server.• Use “verify –t” to transfer and convert archive files.• Once everything but commit server is on Linux, failover commit server to

Linux RO replica.

12

Advantages/Disadvantages

Advantages:• Can do migration in stages.

Disadvantages:• Must run Linux servers with –C1 (case-insensitive) option.

13

Challenges

RHEL7 CVS2P4 Platform specific references

14

Challenges – RHEL7

IT standardized on RHEL7. RHEL7 uses systemd (vs. init scripts) to control services. Worked with John Halbig in support to create service unit

files for p4d (and p4p and p4broker), he created KB with sample: http://answers.perforce.com/articles/KB/10832

15

Challenges – CVS2P4

Original Perforce system created with cvs2p4 script. cvs2p4 works by using CVS change history to create

checkpoint to build DB files. The db entries refer to original CVS (RCS) archive files stored

in special “import” dir. CVS/RCS stores branches in the archive files, with revs like

1.1.1.1, but P4 uses separate files for branches.

16

Challenges – CVS2P4

P4D resolves the CVS 1.1.1.1 branch revs OK, but “verify –t” wouldn’t transfer them.

Solution = Use smbclient + dos2unix to transfer the cvs2p4 import dir.

This brought over ~70K changes, then used “verify –t” to get rest.

17

Challenges – Platform specific entries

Because we need to run in a mixed Win/Linux environment during transition, can’t have platform specific entries anywhere.

Specific examples:• Depot dir location.• Trigger table entries.

18

Challenges – Depot dir location

Win servers had “flat” P4ROOT layout where depot dirs were nested in D:\p4d\ dir. (e.g., D:\p4d\depot)

Short path length was partially to mitigate 260 char Win max path issue.

Didn’t want depots in P4ROOT on Linux, so set “server.depot.root” configurable to put dirs in other location.

19

Challenges – Trigger table entries

Single trigger table shared by all servers. As a result, can’t have:• OS-specific paths in table.• OS-specific binaries in table.• OS-specific references in triggers.

20

Challenges – Trigger table paths

Can’t use OS-specific paths in table. Solution = %serverroot% var + make sure tools (Perl,

Python, etc.) are present in base system path. Old = C:\python27\python.exe C:\bin\trigger.py args

New =python2 %serverroot%/triggers/trigger.py args

21

Challenges – Trigger table binaries

Can’t use OS-specific binaries in table. Solution = Wrappers and/or rename via client. Example: p4auth_ad.exe AD auth trigger. Old =C:\bin\p4auth_ad.exe args

New =%serverroot%/triggers/p4auth_ad.exe args

Linux = p4auth_ad.pl gets synced as p4auth_ad.exe

22

Results

Speed = Syncs, checkpoint/DB rebuild process, p4todb rebuild are all faster on Linux servers.

Load = Linux edge server for build farm handled 200+“sync –f” at one time.

23

Summary

Time• It will take longer than expected.

Effort• It is easier than expected.

Work with support.

24

References

systemd = http://answers.perforce.com/articles/KB/10832 P4Migrate = http://answers.perforce.com/articles/KB/2538 P4Convert =

http://ftp.perforce.com/perforce/tools/p4convert/p4convert.pdf

Thank you!Contact: dfoglesong@tableau.com