The Great Migration: from TeamWare to Mercurial

The Great Migration:from TeamWare to Mercurial

James C. McPherson

Senior Kernel Engineer

Sun Microsystems

The Great Migration: from TeamWare to Mercurial, what we learned along the way

James C. McPherson, Sun Microsystems.© Sun Microsystems 2008. All rights reserved



2

IntroductionNomenclature and HistoryTeamWareReplacement SCM CandidatesMercurialWhat we have learnedReferences



3

Introduction

January 2005Sun releases DTrace under CDDL

14 June 2005Sun releases the bulk of the core Solaris kernel and basic userland under CDDL30000+ files, several million LoC

Source Code Management system?SCCS wrapped in TeamWareIt was Time For A Change(tm)



4

Some nomenclature......

“Solaris” and “Solaris Express” are constructed from several Consolidations:

OS and Networking, aka OS/Net, aka “ON”Admin and Install, aka “AI”JavaJDS (Desktop) and XSunClusterNetWorkStorage (now part of ON), andxVM



5

A bit more nomenclature

Under TeamWare, each Consolidation is housed in The Gate, which is updated to The Clone every night (11pm US/Pacific is the convention)You putback to The GateYou bringover from The Clone

The Gate and Clone filesystems are exported read-only, but TeamWare's metadata directories are exported read-write



6

A picture, please?

The Gateexported r/oTeamWare metadataexported r/w

The Cloneexported r/oTeamWare metadataexported r/w

bringover

putback

Developmentprocess

Updated at 11pmUS/Pacific every night



7

Why not stick with TeamWare?

TeamWare's code was not OpenTeamWare had been EOL'd (End Of Life'd)Getting patches for bugfixes was hardGetting patches for new features was really hardTeamWare was ill suited to globally distributed development, and The operational model did not allow moving Gates outside Sun



8

Development with TeamWare

TeamWare is not just a collection of wrappers around SCCS, it also provides application/scripting triggers, tracks name changes, aggregates operations and even provides checkpointing.Heavy use of NFS – fine when all the developers are in one buildingOver time, engineers wrote their own wrappers around TeamWare, to workaround its shortcomings



9

Other development infrastructure

Developer scripts includenightly – to build your workspacewx (Workspace Extensions) – to manage your workspacebfu (Bonwick-Faulkner Upgrade) webrev – generates webpages for code review



10

What do these scripts do? (1)

nightly – general purpose gate and workspace build script. So-called because it's run every night on the gate source, and used to take all night to run.nightly is generally maintained by ON engineers, and used by several other consolidations as wellnightly has a number of options, including package generation, lint checking, C and Java style checking



11


wx sits atop TeamWarewx helps track checked out files, bug and ARC comments and sanity checks your workspace on requestwx shields the user from most raw SCCS operations



12


bfu, the Bonwick-Faulkner Upgrade, takes cpio archives produced by 'nightly' and blats them onto your system under testCurrently the fastest way of getting a new kernel and consistent minimal userland (commands + libraries) on your system.Gate machines are bfu'd every morning – we “eat our own dogfood”



13


Webrev generates web pages of changes for easy code review



14



15

Gate infrastructure

Scripts written by the gate staff (yes, called gatekeepers... no, there is no keymaster)

nightlybuglist backout – used for reverting bad changeslock-gate / unlock-gate – used when large changes are integratedupdate-flagdays



16

Time For A Change

Both Sun and the OpenSolaris Community agreed that updating the SCM system was requiredOpenSolaris project created to oversee this change: scm-migrationRequirements list generated by the Community and tuned by the projectCandidate SCM systems evaluated against requirements, evaluation reports published on opensolaris.org



17

Major Requirements

Must be Open SourceUnbiased and disconnected distributionNetworked operationInterface stability and completenessStandard operations and transactionsPer-changeset metadata

For the full list, visithttp://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/

http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/



18

Candidate SCM systems

SubVersionMercurialBazaar / Bazaar-NGGitMonotoneSVKTeamWare



19

Evaluations

SubVersion, SVK, Monotone and TeamWare were winnowed out via mailing list discussion (tools-discuss) in the preliminary phase

Bazaar, Git and Mercurial all continued to the next phaseEvaluation reports are available at http://www.opensolaris.org

mailto:[email protected]

http://www.opensolaris.org/



20

Successful SCM candidate

Mercurial



21

Once Mercurial was chosen, the real work could get started

.... and boy, was there a lot of it!



22

Migration tasks

Work divided into four major areas:

Developer-focused toolsChanging source in the gateGate infrastructure toolsEducation



23

Developer-focused tools

Three major scripts needed updating:nightlywxwebrev

Many wx features were ported to Mercurial extensions (thus written in Python) and called “Cadmium”nightly and webrev were enhanced to understand Mercurial as well as TeamWare



24

Why call it Cadmium?

http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm

http://www.csudh.edu/oliver/chemdata/periodic/periodic-1.htm



25

Gate source changes

While managed by TeamWare, all files had embedded SCCS ident strings:

%Z%%M% %I% %E% SMI

Since these are an anachronism with Mercurial, they were removed in files that were under active development at the migration point



26

Gate infrastructure changes

The infrastructure saw a lot of the heavy lifting for the migration, leveraging the changes for the wx to Cdm work

Scripts (ksh88) were re-written in Python as Mercurial hooks

These are still being tweaked, but the majority of changes are bedded down



27

Images by David Marker, used with permission

Gate back-end model



28

New model: general

The Gate

The Clone

hg pull

hg push

Developmentprocess

Updated after everysuccessful hg push(almost immediately)



29

New model: Open and Closed

Gate (Open)

hg pull

hg push

Developmentprocess

Updated after every successful hg push (ie, almost immediately)

Gate (Closed)

Clone (Closed)

Clone (Open)

Separate hg push and hg pull/clonerequired for Closed and Open parts



30

ON Go-live

ON went live with Mercurial on 6 August 2008, when build snv_97 opened:

6733918 Teamware has retired, please welcome your new manager, Mercurial

http://bugs.opensolaris.org/view_bug.do?bug_id=6733918



31

What did we learn?



32

What did we learn? (1)

The OpenSolaris community now knows a lot more about Sun's internal processesSwitching to a new SCM is like a sausage(it's really ugly while it's happening)Corporate firewalls are necessary, but get in the way of migratingThe longer you've been using the old system, the harder it is to changeThe longer you've been using the old system, the more obscure its foibles



33


Your time estimates are too optimisticPush up to date tools to the community as soon as possibleWe had (and have!) essential developer tools that very few people understoodWe have developer tools so old that the last people to understand them properly have retiredThe better you know SCMs in general, the easier your migration will be



34


The existing SCM dictates workflow implementation, which influences what developers see as gaps needing plugsEngineers don't like change. At least, not too much changeA consistent developer tool experience requires SCM-agnosticismMigrating to a modern, sophisticated SCM is a great opportunity to change lots of niggles that require coordination



35


The tools in use must support both the old and new SCMs

developers need to migrate existing workspacesNon-integrated projects need to change and sync upSustaining organisations need to work with both SCMs

Productivity will dip while people work their way up the learning curve



36

SCM migration is anall-or-nothing change

Your bags must be packed for the journey

Finally....



37

References

http://www.opensolaris.org/os/community/tools/scm

SCM Migration project page http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/ SCM Migration requirements listhttp://www.opensolaris.org/os/community/tools/scm/scm_milestone SCM Migration milestoneshttp://www.opensolaris.org/os/community/tools/scm/on-scm-tools ON consolidation tools

http://www.opensolaris.org/os/community/tools/scm

http://www.opensolaris.org/os/community/tools/scm/dscmreqdoc/

http://www.opensolaris.org/os/community/tools/scm/scm_milestone

http://www.opensolaris.org/os/community/tools/scm/on-scm-tools



38

Further reading

http://opensolaris.org/os/community/on/os_dev_process/

OpenSolaris development process

http://opensolaris.org/os/community/on/devref_toc

OpenSolaris ON developer's reference guide

An extended version of this presentation will appear at http://blogs.sun.com/jmcp

http://opensolaris.org/os/community/on/os_dev_process/

http://opensolaris.org/os/community/on/devref_toc

http://blogs.sun.com/jmcp



39

What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?What Questions Do You Have?

WhatQuestionsDo You Have?

The Great Migration:from TeamWare to Mercurial

James C. McPherson

Senior Kernel Engineer

Sun Microsystems

[email protected] Great Migration: from TeamWare to Mercurial, what we learned along the way


mailto:[email protected]

The Great Migration: from TeamWare to Mercurial

Technology

Transcript of The Great Migration: from TeamWare to Mercurial