Tracking FreeBSD in a Commercial Setting

M. Warner LoshCisco Systems

Broomfield, COimp@freebsd.org

Abstract

The FreeBSD project publishes two lines of source code:current and stable. All changes must first be committedto current and then are merged into stable. Commer-cial organizations wishing to use FreeBSD in their prod-ucts must be aware of this policy. Four different strate-gies have developed for tracking FreeBSD over time. Acompany can choose to run only unmodified release ver-sions of FreeBSD. A company may choose to importFreeBSD’s sources once and then never merge newerversions. A company can choose to import each newstable branch as it is created, adding its own changesto that branch, as well as integrating new versions fromFreeBSD from time to time. A company can trackFreeBSD’s current branch, adding to it their changesas well as newer FreeBSD changes. Which method acompany chooses depends on the needs of the com-pany. These methods are explored in detail, and theiradvantages and disadvantages are discussed. TrackingFreeBSD’s ports and packages is not discussed.

1 Problem Statement

Companies building products based upon FreeBSD havemany choices in how to use the projects sources andbinaries. The choices range from using unmodifiedbinaries from FreeBSD’s releases, to tracking modifyFreeBSD heavily and tracking FreeBSD’s evolution ina merged tree. Some companies may only need to main-tain a stable version of FreeBSD with more bug fixesor customizations than the FreeBSD project wishes toplace in that branch. Some companies also wish tocontribute some subset of their changes back to theFreeBSD project.

FreeBSD provides an excellent base technology withwhich to base products. It is a proven leader in per-formance, reliability and scalability. The technology

also offers a very business friendly license that allowscompanies to pick and choose which changes they wishto contribute to the community rather than forcing allchanges to be contributed back, or attaching other unde-sirable license conditions to the code.

However, the FreeBSD project does not focus on inte-gration of its technology into customized commercialproducts. Instead, the project focuses on producing agood, reliable, fast and scalable operating system andassociated packages. The project maintains two lines ofdevelopment. A current branch, where the main devel-opment of the project takes place, and a stable branchwhich is managed for stability and reliability. While theproject maintains documentation on the system, includ-ing its development model, relatively little guidance hasbeen given to companies in how to integrate FreeBSDinto their products with a minimum of trouble.

Developing a sensible strategy to deal with both theseportions of FreeBSD requires careful planning and anal-ysis. FreeBSD’s lack of guidelines to companies leavesit up to them to develop a strategy. FreeBSD’s devel-opment model differs from some of the other Free andOpen Source projects. People familiar with those sys-tems often discover that methods that were well suitedto them may not work as well with FreeBSD’s develop-ment model. These two issues cause many companiesto make poor decisions without understanding the prob-lems that lie in their future.

Very little formal guidance exists for companies wishingto integrate FreeBSD into their products. Some emailthreads can be located via a Google search that couldhelp companies, but many of them are full of contradic-tory information, and it is very disorganized. While theinformation about the FreeBSD development process isin the FreeBSD handbook, the implications of that pro-cess for companies integrating FreeBSD into their prod-ucts are not discussed.

27

2 FreeBSD Branching

The FreeBSD development model strikes a balance be-tween the needs of the developers and the needs of itsusers. Developers prefer to have one set of sources thatthey can change arbitrarily and not have to worry aboutthe consequences. Users prefer to have a stable systemthat is compatible with the prior systems. These twodesires are incompatible and can cause friction betweendevelopers and users.

FreeBSD answers the need of both groups by providingtwo versions of its code. The project maintains a mainline for its developers, called “current.” This branch con-tains all the latest code, but all that code might not beready for end users. All changes to FreeBSD are re-quired to be first committed to the current branch. Thequality of the current branch varies from extremely sta-ble to almost unusable over time. The developers try tokeep it towards the stable end of the spectrum, but mis-takes happen.

To provide a stable system users can use, FreeBSD alsomaintains a stable version of the OS. Every few yearsthe current version is branched and that branch becomesthe new stable version. This branch is called either“stable” or “RELENG X” where X is the major ver-sion number for that branch. Stable branches are welltested before they are released. Once released, only welltested patches from the current branch are allowed to bemerged into the branch. Once a stable branch is cre-ated, its ABI and API are never changed in an incom-patible manner, which allows users to upgrade to newerreleases that are made from the stable branch with rel-ative ease. Stable branches tend to have a lifetime ofabout 2-6 years.

An even more stable version of FreeBSD is availablethan the stable branch. For each release made off a sta-ble branch, a release branch is also created. The onlychanges that go into these release branches are secu-rity fixes and extremely important bug fixes. These aredesigned for users that wish to run a specific release,but still have high priority bugs fixed and available in atimely fashion. Since release branches are targeted onlyat end users and have so few changes, the rest of thispaper will treat them as a release.

Figure 3 tries to show the relationships between the dif-ferent branches over time. It shows what should havetheoretically happened if FreeBSD had a major releaseevery two years. The horizontal axis is time (in years).The vertical axis is the amount of change, in arbitrary

units. Vertical arrows point to the theoretical releasepoints (with the release number under the arrow whenthe name fits). After about three years, the branches stopbeing used in favor of newer releases.

Figure 4 shows data from the FreeBSD project sinceearly 1997.1 There are many features of this graph thatdiffer from the idealized graph. The two that are mostrelevant are that major branches live beyond the threeyear idealized vision and that the timing of the releasebranches isn’t completely regular. These points will beimportant later in deciding which method fits the com-pany’s needs the best.

The FreeBSD ports system (which is used to gener-ate the packages that appear in FreeBSD’s releases) isnot branched at all. Instead, it supports both the cur-rent branch, as well as the active stable branches of theproject. For each release, the tree is tagged so that it canbe reproduced in the future if necessary. These policiesare different than the main source tree. Tracking of theports tree is not addressed further in this paper becauseits model is different and the author has fewer examplesfrom which to draw advise and conclusions from.

3 Branching Choices

There are a wide range of companies using FreeBSD intheir products today. On the simplest end, companiesload FreeBSD onto boxes that they ship. On the mostcomplex end, companies modify FreeBSD extensivelyto make it fit their needs. Over the years four differentapproaches to tracking FreeBSD have evolved.

The simplest method involves using the stock FreeBSDreleases unmodified. Companies doing this grabFreeBSD at its release points and make no changes tothe software and just configure th system and install thepackages that their customers need. Typically no sourcesare tracked and only binary packages from FreeBSD’sweb pages are used.

The next simplest method involves grabbing a releaseof FreeBSD and using that as a basis for their product.FreeBSD is effectively forked at this point as the com-pany makes whatever modifications are necessary fortheir product. No thought is given to upgrades or con-tributing bug fixes back into the community.

1The author used fairly simple scripts to extract this data from thecommit logs, whose format changed in 1997. Some flaws exist in thedata, but they do not affect the shape of the graph.

28

Companies often setup repositories of FreeBSD stablebranches. In this model, the tip of a stable branch (orthe latest release point) is imported into some SCM. Thecompany will then make fixes and improvements to itsprivate branch. The company will import newer versionsof FreeBSD on this stable branch from time to time. Bet-ter run companies will try to contribute their fixes backinto FreeBSD to simplify their upgrade path.

The most complicated method involves mirroring theFreeBSD development process. The company will im-port the latest version of the FreeBSD developmentbranch. They will setup automated scripts for pullingin newer versions. They will make their changes toFreeBSD in this mainline of development. Rather thanusing FreeBSD’s stable branches, the company will de-cide when and where to branch its version. Oncebranched, it will control what fixes are merged into itsbranch.

3.1 Stock FreeBSD

The most widespread use of FreeBSD involves thismethod. In this method, the company grabs the binariesfrom a FreeBSD web site or commercial vendor and usesthem as built. They layer packages on top of FreeBSD,typically a mix of stock packages from the release andtheir own additional scripts or programs. The focus ofthese companies is to have a system that they can deployand use for a particular purpose.

Customization of the system is typically tracked in somekind of source code management (SCM) system. Thesecustomizations include the /etc/rc.conf file (whichcontrols most of the global settings for the system), aswell as configuration files and other data used by thesystem. Some of these companies will also compile cus-tomized kernel configurations. These files can typicallybe tracked in any SCM as the demands on the SCM aremodest.

These companies typically upgrade only when they needto do so. Once they find a stable version they stick withit until they need something from a newer version. Thiscould be support for newer hardware (drivers or archi-tectures), or application level features such as threadingsupport. Often times they will track newer security re-leases with services such as FreeBSD update and/or por-tupgrade in package mode.

FreeBSD meets the needs of these companies fairly well.They don’t require additional features or bug fixes not

in the current releases. They don’t need to optimizeFreeBSD for any given platform beyond what the stan-dard system tunables provide for them. The main ad-vantage for these companies is that FreeBSD is a dropin solution. There’s very little overhead necessary to gettheir machines and applications running and FreeBSD’sstandard install tools can be used to create images fortheir products (if they even need separate images at all).Some of these companies participate in the communityand contribute to the community in many ways. Someof these companies do not. The choice is up to the indi-vidual company and its needs, sensitivities and desires.

3.2 Grab and Go

Another easy way to use FreeBSD sources is the graband go method. In this method, the companies grabFreeBSD at some version and then never upgradeFreeBSD. No attempts to track FreeBSD or pull bugfixes in from FreeBSD are made. The company grabs thesource and starts hacking. They layer in their own buildand packaging system often times. Sometimes they portto a new architecture. FreeBSD typically is the base for amore extensive application of appliance which the com-pany has total control over.

There are a few advantages to this method. The companycan concentrate on making their product work withoutthe distractions introduced when software versions arerolled. The company manages its risk by doing ev-erything themselves. The company can keep any in-formation about what they are doing from being in-ferred by competitors looking at their bug submissions toFreeBSD. The company’s employees are not distractedby interactions with the FreeBSD community. Withoutthese distractions, it is believed that this method allowsa company to bring its product to market more quickly.

However, there are many disadvantages to this method.The biggest problem is that companies using this methodoften find it difficult to get support for the community.Most of the active members in the community havemoved on to newer versions of the software, so are un-able to help out with problems in older versions. Manyof the bug fixes in newer versions of the software are dif-ficult to back port because they depend on other changesto the software that aren’t present in the older versionsof the software. Often times, interaction with the com-munity on problems for recent releases of the softwarecan save tremendous amounts of time for the company’semployees because they can leverage the knowledge ofothers who have had similar problems.

29

Companies often times think they are in total control ofthe hardware platform, but in reality this is a mistakenassumption. Hardware platforms are made of up chipsthat one buys from manufacturers. These chips go ob-solete at an alarming rate sometimes, forcing changesto the underlying hardware to even be able to continueto build it. These new chips often times require newchanges to the software. Just as often, others in the com-munity have used the newer parts and have migrated thenecessary changes into FreeBSD. So often times com-panies that go down this path are forced to redo workthat has already been done in the community when theirsupplies tell them that they will no longer be able to givethem a certain chip, and no replacements from other ven-dors exist.

Some companies have managed to start out with thismethod and later transition to one of the other methodsdescribed in this paper. One is even rumored to haverecently completed the jump from FreeBSD 2.1.6 (re-leased in 1996) to FreeBSD 6.2 and are now using thestable branch tracking method described below. Othertimes, the outcome isn’t so good and the product is mi-grated to another system, or the product is killed.

3.3 Stable Branch Tracking

One nice feature of FreeBSD’s stable branches is theirstability. One can typically count on them to build andnot have critical problems. The stable branch trackingstrategy takes advantage of this feature.

The first major release on a branch is imported intoa private SCM for the company to use. The sourcesare imported using the ’vendor branch’ facility of theSCM. This facility allows one to keep a pristine copyof the sources from FreeBSD separate from the mod-ified sources for the company. This separation allowsdevelopers to produce patches between the two. Thesepatches can be used to determine which changes shouldbe contributed back to the FreeBSD tree. In addition,by importing into a vendor branch and merging intothe company’s private branch, the company can upgradeversions of FreeBSD at any time. They can pull eithera whole new FreeBSD tree, or individual files that havethe fixes they need. The company can choose when toroll forward the basis of their tree, and the branchingfeatures of most SCMs make this procedure relativelyeasy. As new stable branches of FreeBSD become avail-able, this process can be repeated for them in a separatemodule or directory in the SCM.

The big advantage to this approach is the underlying na-ture of the stable branch itself. The FreeBSD projecthas policies and practices that ensure that the branchwill be stable, especially near releases off of that branch.The ability to “cherry pick” fixes from newer versions ofFreeBSD without affecting the rest of the branch helpsto mitigate risks associated with upgrading. In addition,by using the vendor branch feature, these changes willnot interfere with future imports of a more complete sys-tem when it is appropriate to do so. Since the ABI andAPI are also frozen for the entire branch, one can grabfixes and changes from newer versions without worryingabout breaking applications under development withinthe company. The isolation of major releases into sepa-rate modules in the SCM allows a company that has sev-eral products built on FreeBSD to selectively upgradethem to newer versions as market conditions warrant.

There are a few disadvantages for this approach. First, tofully leverage the FreeBSD community, it is desirable topush back bug fixes to the community in a timely fash-ion. When this isn’t done, as is often the case when dead-lines are tight, the chore up upgrading increases becauseone must bring forward all of the changes to the system.Second, if the company makes extensive changes thataren’t merged back into FreeBSD and want to migrateto the next major version, they will need to redo theirchanges after the next major branch is created. If theyare in an area of FreeBSD that has changed between thetwo branches, this can take quite a bit of time and effort.

�

COMPANY 6� � �

RELENG 6�6.0

�6.1

�6.2

Figure 1: Code Flow between FreeBSD RELENG 6 andCompany’s Version

Figure 1 shows this graphically. This figure shows anidealized flow of patches into the company tree and backto FreeBSD. It also neglects to picture the required tripthrough FreeBSD current required for all patches to becommitted to stable branches. The number of changes tothe branches are also abstracted out, unlike Figure 1 and2 presented above. The arrows pointing to the RELENGbranch represent FreeBSD releases from that branch.The arrows from the RELENG branch to the COM-PANY branch represent merges of code from FreeBSDinto the company’s repository. The arrows from COM-PANY to RELENG represent patches that have success-fully been contributed back into FreeBSD and have beenmerged into FreeBSD’s RELENG tree.

30

3.4 Own Branching

One way to keep current in FreeBSD is to trackFreeBSD’s main development branched called “cur-rent.” Many developers do this in the FreeBSD per-force tree and it works well for them. This method fol-lows that practice, but also adds stable branches, akin toFreeBSD’s stable branches in concept, but not trackingany specific FreeBSD release.

The company would import FreeBSD’s current codeas its starting point for its FreeBSD development ef-forts. They would start making changes to their currentbranch. In addition, source code pulls from FreeBSD’scurrent branch would be frequent to keep the company’scurrent branch close to FreeBSD’s current branch. Justafter these pulls, the company’s current branch wouldbe exactly FreeBSD’s current branch with only the com-pany’s changes layered on. The company would thenmerge the relevant change from its current tree intoFreeBSD’s current tree by working with the FreeBSDcommunity to produce acceptable patches.

The company would also emulate FreeBSD’s branchingpractices. When the tree is in a good state to branch, pos-sibly driven by delivery schedules for its end products,the company would branch its own stable branch fromtheir current branch. They would merge bug fixes andnew features from their current branch into this stablebranch and build products from this stable branch.

The main advantage of this approach is that it is eas-ier to keep current with FreeBSD than the stable branchtracking approach. To generate patches, a simple diff(3)between the FreeBSD sources and the company sourceswill generate the patches. As patches are merged withFreeBSD, the next pull will automatically include thosechanges and the delta between the company’s sourcesand FreeBSD’s will drop. By controlling the branch-ing times, there’s no need to wait for FreeBSD to cre-ate new a stable branch, so the company can drive re-leased schedules more easily than companies trackingstable branches.

The main disadvantage of this approach is that thecompany loses the work done by the FreeBSD com-munity to keep its stable branches stable and useful.Since there is no connection between the company’sstable tree and FreeBSD’s stable tree, improvements toFreeBSD’s stable branch aren’t automatically reflectedin the company’s stable branch. An engineer will needto watch changes going into either the current branchfrom FreeBSD, or into FreeBSD’s stable tree and man-

ually pull them into their own stable branch. Typically,there are on the order of 100-200 commits to a FreeBSDstable branch a month, so this load can be quite large. Inaddition, except around the time a new branch is cut inFreeBSD, FreeBSD’s current branch may have periodsof instability and it can be quite difficult to know when agood time to branch might be as many of the stability orquality problems that are in FreeBSD’s current branchoften lay undiscovered for months or years because itdoesn’t get the intensity of testing that a FreeBSD stablebranch receives.

��

��

��

��

��

��

��

��

��

��

��

��

��

����

FreeBSD Current

Company Current

Company Stable

Figure 2: Relationship between FreeBSD current andcompany branches

Figure 2 shows this graphically. This figure shows anidealized flow of patches into the company tree andback to FreeBSD. The two parallel current branchesare shown diagonally, with the company’s custom sta-ble branch shown horizontally, much like Figures 1 and2 presented above. No FreeBSD release points are in-cluded, since they are largely irrelevant to the method.The exact delta between the two current branches is alsoabstracted out, as this will ebb and flow over time andneedlessly complicates the graph. The arrows representchanges being merged from one branch to another, ei-ther between the two current branches, or from the com-pany’s current branch to its stable branch.

4 Acknowledgments

I would like to thank the crew at Timing Solutions: BenMesander, John Hein, Patrick Schweiger, Steve Passe,Marc Butler, Matthew Phillips, and Barb Dean for theirinsight and implementation of the ’Stable Branch Track-ing’ method described in this paper. We deployed itacross 4 major versions of FreeBSD.

I would like to thank Julian Elischer for the many con-versations that we have had about development method.He provided much of the input into the ’Own Branching’section.

31

0

20000

40000

60000

80000

100000

1998 2000 2002 2004 2006 2008

Idealized 2 year release cycle

RELENG_3

RELENG_4

RELENG_5

RELENG_6

RELENG_7

HEAD

3.0 3.5 End of life

4.0 4.5 End of life

5.0 5.5 End of life

6.0 6.5 End of life

7.0

Figure 3: Idealized branching model

0

20000

40000

60000

80000

100000

120000

1998 2000 2002 2004 2006 2008

Cumulative Commits For FreeBSD Branches

RELENG_3

RELENG_4

RELENG_5

RELENG_6

RELENG_7HEAD

3.03.2 3.4 3.5.1 < 1 commit/yr< 50 commit/yr

4.04.2

4.44.6 4.7 4.8 4.9 4.10 4.11 < 50 commit/yr

5.0Perforceadoption

5.1

5.2 5.3 5.4 5.5 < 100 commit/yr

6.0 6.1 6.2 6.3

7.0

Figure 4: Actual FreeBSD branching history

32