Japan UNIX SocietyJapan UNIX SocietySeptember 18, 2002September 18, 2002

FreeBSD Release Engineering

Murray StokelyFreeBSD Mall, Inc.

murray@FreeBSD.org

OutlineOutline

Introduction Terminology

FreeBSD Development Model

Release Process Overview Code Review

Release Checklist

CVS Operations

Release Building

OutlineOutline

Extensibility

FreeBSD 4.6 / 4.6.2

Future Directions

SCM TerminologySCM Terminology

SCM - Software Configuration Management

Branch -A variant codeline that is separate from the main line of

development.

Merge - The process of integrating tested changes across branches.

Tag - An identifier for the code in an SCM system at a given point

in time.

CVS - A widely used open-source SCM system.

FreeBSD Development ModelFreeBSD Development Model

FreeBSD development continues along two parallel branches; FreeBSD-CURRENT and FreeBSD-STABLE.

FreeBSD-CURRENT is the main trunk of our CVS repository. All new

development should happen here.

FreeBSD-STABLE is the branch from which major releases are made.

Changes enter this branch at a different pace, and with the general

assumption that they have been well tested by our user community

running -CURRENT.

FreeBSD Development ModelFreeBSD Development Model

Recent FreeBSD Branch Points

FreeBSD Development ModelFreeBSD Development Model

Thousands of developers around the world contribute code to FreeBSD.

Only the "committers" have write access to the CVS repository

324 total committers in FreeBSD (includes source, documentation,

and ports committers).

212 active committers to src/ in the last 12 months.

160 active committers to src/sys in the last 12 months.

(active is loosely defined as making a single commit.)

FreeBSD Development ModelFreeBSD Development Model

Users and external developers should submit bug fixes, enhancements, or problem reports with GNATS.

Online submission forms, send-pr command installed by default on

FreeBSD systems, etc..

Open PRs vs TimeOpen PRs vs Time

Release Process OverviewRelease Process Overview

Releases are made from the -STABLE development branch at approximately 4 month intervals.

Reminder / Announcement email sent to developers <45 days>

MFC Sweeps ("Merge from -CURRENT")

Code slush / freeze <30 days>

Release Candidates <15 days>

Eventually a final release

Upcoming release schedule maintained at

http://www.FreeBSD.org/releases/4.7R/schedule.html

Release ChecklistRelease Checklist

Creating the Release Branch

Branch point tag

$ cvs rtag -rRELENG_4 RELENG_4_6_BP src

Creating the branch

$ cvs rtag -b -rRELENG_4_6 _BP RELENG_4_6 src

Bumping up the Version Bumping up the Version NumberNumber

Before the final release can be tagged, built, and released, the following files need to be modified to reflect the correct version of FreeBSD.

doc/${LANG}/books/handbook/mirrors/chapter.sgml

doc/share/sgml/freebsd.dsl

src/Makefile.inc

src/UPDATING

src/gnu/usr.bin/groff/tmac/mdoc.local

src/release/Makefile

Bumping up the Version Bumping up the Version NumberNumber

(continued)

src/release/doc/${LANG}/share/sgml/release.dsl

src/release/doc/share/examples/Makefile.relnotesng

src/release/doc/share/sgml/release.ent

src/share/examples/cvsup/standard-supfile

src/share/misc/bsd-family-tree

src/sys/conf/newvers.sh

src/sys/sys/param.h

Bumping up the Version Bumping up the Version NumberNumber

(continued)

www/${LANG}/releases/*

www/${LANG}/docs.sgml

www/${LANG}/cgi/ports.cgi

Creating the Final Release Creating the Final Release TagTag

When the final release is ready, the following command will create the RELENG_4_6_0_RELEASE tag.

$ cvs rtag -rRELENG_4_6 RELENG_4_6_0_RELEASE

The Documentation and Ports managers are responsible for tagging the

respective trees with the RELEASE_4_6_0 tag.

"Sliding" tags after the fact :

$ cvs tag -d tagname filename

Release BuildingRelease Building

Tools necessary for building a FreeBSD release are available in src/release/

These tools aim to provide a consistent way to build FreeBSD releases.

A complete release can be built with only a single command, including the

creation of ISO images suitable for burning onto CDROM, installation

floppies, and an FTP install directory. This command is aptly named "make

release"

Release BuildingRelease Building

To successfully build a release, you must first populate /usr/obj by running "make world" or simply "make buildworld". The release target requires that several variables be set properly :

CHROOTDIR - The directory to be used as the chroot environment for the

entire release build.

BUILDNAME - The name of the release to be built.

CVSROOT - The location of a CVS repository.

RELEASETAG - The tag corresponding to the sources to build.

"make release""make release"(1/4)(1/4)

There are many other variables that can be used to customize the release build process. Most of these are documented at the top of src/release/Makefile. The exact command used to build FreeBSD 4.6 (x86) was :make release CHROOTDIR=/local3/release \ BUILDNAME=4.6-RELEASE CVSROOT=/host/cvs/usr/home/ncvs RELEASETAG=RELENG_4_6_0_RELEASE

"make release""make release"(2/4)(2/4)

The release makefile can be broken down into several distinct steps.

Creation of a sanitized system environment in a separate directory hierarchy

with "make installworld".

Checkout from CVS of a clean version of the system source, documentation,

and ports into the release build hierarchy.

Population of /etc and /dev in the chrooted environment.

chroot into the release build environment, to make it harder for the outside

world to taint the build.

"make release""make release"(3/4)(3/4)

Steps of "make release" (continued)

"make world" in the chrooted environment.

Build Kerberos-related binaries.

Build 'GENERIC' kernel.

Creation of a staging directory tree where the binary distributions will be

built and packaged.

Build and installation of the documentation toolchain needed to convert the

documentation source (SGML) into the HTML and text documents that will

accompany the release.

"make release""make release"(4/4)(4/4)

Steps of "make release" (continued)

Build and install of the actual documentation (user manuals, release notes,

tutorials, hardware compatibility lists, etc..)

Build of the "crunched" binaries to be used on the installation floppies

Creation of the distribution tarballs of the binaries and sources.

Create the boot media and a fixit floppy.

Create the FTP installation hierarchy.

(optionally) Create ISO images for CDROM media.

Ports and PackagesPorts and Packages

The FreeBSD Ports Collection is a collection of nearly 7,500 third-party software packages available for FreeBSD.

The ports team (portmgr@FreeBSD.org) is responsible for maintaining a

consistent ports tree that can be used to create the binary packages that

accompany a given FreeBSD release.

Ports and PackagesPorts and Packages

In order to provide a consistent set of third-party packages, every port is built in a separate chroot environment, starting with an empty /usr/local and /usr/X11R6.

The requisite dependencies are installed as packages before the build

proceeds.

By starting the package build in a pristine environment, we can assure that the

package metadata (such as required dependencies) are accurate, and so we will

never generate packages that might work on some systems and not on others

depending on what software was previously installed.

Ports Build ClusterPorts Build Cluster

The "Ports Cluster" for the x86 architecture currently consists of a master node (Dual Pentium III 733 Mhz) and 8 slave nodes (Pentium III 800Mhz) to do the actual package builds. With this configuration, a complete package build takes over 24 hours.

The Ports Cluster for the Alpha architecture consists of 7 PWS 500A

machines donated by Compaq. Faster Alpha machines have recently been

donated by Compaq/HP and will soon be available for the ports cluster.

The clusters for both architectures are co-located with the other FreeBSD

Project equipment on Yahoo's campus in Santa Clara, California.

Ports and PackagesPorts and Packages

For FreeBSD 4.4, over 4.1 gigabytes of packages were created. This causes a problem for CDROM distributions because we would obviously like to ship as many packages as possible without making the user insert another disc to satisfy dependencies.

The solution is to put "clusters" of like packages with similar dependencies

onto specific discs.

The package split is performed by Steve Price and the ports team in

coordination with the general wishes of the general user community with

respect to which packages get to appear on the first CD.

FreeBSD CDROMsFreeBSD CDROMs

Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the BSDi/WRS "official" CDROM distributions.

Each of the four discs must contain a README.TXT, CDROM.INF, and

filename.txt file.

The file manifest can be created with :

find . -type f | sed -e 's/\^.\///' | sort >

filename.txt

Disc #1Disc #1

Output from "make release" with the following additions :

Addition of a 'tools' directory to aid would-be new users working in other

operating systems.

XFree86

As many additional software packages as will fit.

Disc #2Disc #2

Also largely created by "make release".

Contains a "live filesystem" that can be used from sysinstall to troubleshoot a

FreeBSD installation.

Contains a compressed copy of the CVS repository in the CVSROOT directory

Contains commercial software demos (such as Perforce, Accelerated-X, etc..)

in the commerce directory.

Discs #3 and 4 simply contain as many additional packages as will fit.

ExtensibilityExtensibility

A release can be customized in many different ways.

If you have access to the staging directory of an existing "make release" build

hierarchy then you may find it easiest to apply patches or additional files as

necessary to the chroot build directory.

rm ${CHROOTDIR}/usr/obj/usr/src/release/release.[48]

Rebuild sysinstall, the kernel, or whatever parts of the system your change

affected.

chroot ${CHROOTDIR} ./mk release.4 release.8

FreeBSD 4.6FreeBSD 4.6

Code slush began on May 1, 2002. After which time all commits to the RELENG_4 branch (-STABLE) had to be approved by re@FreeBSD.org

The first release candidate for the x86 architecture was released on May 17.

2 more release candidates were released leading up to the final release on June

15.

Over 1,100 emails were sent to re@FreeBSD.org in little over a month.

FreeBSD 4.6.2FreeBSD 4.6.2

Unfortunately, there were a number of security problems discovered in OpenSSH and OpenSSL immediately after the release of FreeBSD 4.6.

Many people also reported problems with the ATA subsystem.

The release engineering and security officers decided that there were enough

problems to warrant a rare point release of FreeBSD.

Previously, approach used with 4.1.1 didn't work. A more conservative

approach was needed.

4.6.1 - The "Phantom Release"

Future WorkFuture Work

More documentation

Regression Testing

Better Installation Tools

Graphical User Interface

Message Catalogs for Internationalization.

Additional InformationAdditional Information

Detailed documentation on "make release" is available in release(7). (Also translated to Japanese).

A schedule of upcoming releases is maintained at http://www.FreeBSD

.org/releng

The release engineering paper is now maintained in CVS and the most recent

version is available from http://www.FreeBSD.org/docs.html

If all else fails, just email re@FreeBSD.org and we will be happy to answer

your questions.