Porting HPC Tools to FreeBSD

Brooks DavisThe Aerospace Corporation

El Segundo, CAbrooks@aero.org

Abstract

Since 2001 we have used FreeBSD as a high perfor-mance computing (HPC) cluster operating system. Inthe process we have ported a number of HPC tools in-cluding Ganglia, Globus, Open MPI, and Sun GridEngine. In this paper we discuss the process of port-ing these types of applications and issues encounteredwhile maintaining these tools. In addition to generalissues of porting code from one Unix-like operatingsystem to another, there are several types of portingcommon to many HPC tools which we will explore.Beyond porting, we will discuss how the ports collec-tion aids our use of HPC applications and ways wethink overall integration could be improved.

1 Introduction

At The Aerospace Corporation we have designed,built, and operated a FreeBSD based HPC clustersince 2001[Davis]. Figure 1 shows the Fellowship clus-ter in it’s current form with 352 dual processor nodes.In the process of building and running Fellowship wehave ported a number of open source HPC tools toFreeBSD. In this paper we discuss the process of port-ing them, issues we encountered, and a few pet peevesabout application portability.

We begin with an overview of the porting of threetools: Sun Grid Engine—also known as SGE—a batchjob manager; the Ganglia monitoring system, a clus-ter/grid monitoring tool; and Open MPI, a leading im-plementation of the Message Passing Interface whichis an API for message based parallel programming.After discussing these tools we review a number ofgeneral portability issues and possible remedies forFreeBSD.

We hope to demonstrate that porting HPC tools isstraight-forward and encourage others to join the ef-

c©2010 The Aerospace Corporation.All trademarks, service marks, and trade names are theproperty of their respective owners.

Figure 1: Fellowship Circa February 2007

fort to port more tools. We also want to encouragedevelopers to think of ways to make FreeBSD an eas-ier porting target.

2 Experiences Porting Applications

2.1 Ganglia

The Ganglia monitoring system[Massie]—usually re-ferred to as Ganglia—is a cluster and grid monitor-ing tool that provides current and historical data onthe status of a cluster or collection of systems. Thedata includes CPU count, CPU utilization, mem-ory use, and network I/O among others. Gangliais used on systems ranging from small clusters toPlanetLab[PlanetLab], a globally distributed networkof over 1000 nodes at nearly 500 sites. Beyond HPCapplications Ganglia is used to monitor web farms oreven desktop computing resources.

Ganglia’s architecture consists of monitoring daemons(gmond) on each node that send periodic updates ofvarious metrics to a central collector daemon (gmetad)that stores the results in a set of Round RobinDatabases[RRDtool]. The metrics cover many aspects

of system configuration and load, and may either befixed (per-boot) or vary with time.

Core metric providers are implemenated as functionsin the monitoring daemon that return string or nu-meric values. In early versions of Ganglia the set ofmetrics was fixed at compile time. In more recent ver-sions, it is configured at runtime and pluggable metricproviders are supported including Python modules.

The process we used to port metrics is simple. First,we checked the code for other operating systems todetermine what the metric means. Then we figuredout a way to use standard tools such as top, ps, proc-stat, or sysctl to extract that information from thesystem. Once appropriate tools were identified, weexamined their source code to determine how theyretrieved the information in question and used thatinformation along with man pages to write an appro-priate metric implementation. In many cases it waspossible to copy the code directly from FreeBSD util-ities along with another copyright statement and li-cense block because Ganglia is BSD licensed.

One of the simplest metrics to port was cpu numwhich is the number of CPU cores in a machine. List-ing 1 shows the entire implementation and illustratesthe basics of Ganglia metric implementation. Eachmetric is implemented as a function that returns ag val t structure representing the current value of themetric.

Listing 1: cpu num func()

g v a l tcpu num func (void ){

g v a l t va l ;int ncpu ;s i z e t l en = s izeof ( int ) ;i f ( sysct lbyname ( ”hw . ncpu” , &ncpu ,&len , NULL, 0) == −1 | | ! l en )

ncpu = 1 ;

va l . u int16 = ncpu ;return va l ;

}

The swap total and swap free metrics are more com-plex metrics and cover several interesting cases includ-ing the use of the kvm[FreeBSD-kvm] interface andsupporting multiple interfaces for cross platform sup-port. In porting them we observed that users andadministrators can obtain swap information from theswapinfo(1) command[FreeBSD-pstat]. We dug intothe code and determined that it used kvm to accessthe data if called on a coredump and a sysctl if not.However, when we first ported Ganglia, FreeBSD 4did not support the sysctl interface so we supported

both sysctl and kvm methods because the sysctl in-terface allowed us to run Ganglia without ever beingroot, but the kvm interface was needed since we needto support FreeBSD 4.x. As in Listing 2 our imple-mentation probes at runtime and prefers the sysctlinterface where available.

Listing 2: Excerpts from metric init()g v a l tm e t r i c i n i t (void ){

g v a l t va l ;

mibswap size = MIB SWAPINFO SIZE ;i f ( sysct lnametomib ( ”vm. swap in fo ” ,mibswap , &mibswap size ) == −1) {kd = kvm open (NULL, NULL,NULL, O RDONLY,” me t r i c i n i t ( ) ” ) ;

} else {kd = kvm open ( PATH DEVNULL,NULL, NULL, O RDONLY,” me t r i c i n i t ( ) ” ) ;

use vm swap info = 1 ;}page s i z e = ge tpag e s i z e ( ) ;<...>va l . in t32 = SYNAPSE SUCCESS;return va l ;

}

The core of the swap total implementation shown inListing 3 uses the result of the probe in metric init ()

to determine how to retrieve the amount of swap usedusing sysctl or kvm. In the sysctl case, we retrieveinformation from each individual swap device and to-tal them where the kvm interface provides a directtotal. There are a couple other interesting things ofnote here. First, Ganglia expects swap size to be afloating point number of KiB. Variants of this arecommon in Ganglia due to the desire to representlarge sizes identically on all machines. Second, is theXSWDEV VERSION line. This points out a problemwith version numbers in binary interfaces. They aregood in principle, but clients need to know about thenew version to do anything with it so in practice theydo little to help provide ABI stability. In this case wehave no choice but to reject versions we do not under-stand so existing binaries would necessarily be brokenif this were changed.

Listing 3: Excerpts from swap total func()i f ( use vm swap info ) {

for (n = 0 ; ; ++n) {mibswap [ mibswap size ] = n ;s i z e = s izeof ( xsw ) ;i f ( s y s c t l (mibswap ,mibswap size + 1 , &xsw ,&s i z e , NULL, 0) == −1)

break ;i f ( xsw . xsw vers ion != XSWDEV VERSION)return va l ;totswap += xsw . xsw nblks ;

}} else i f ( kd != NULL) {

n = kvm getswapinfo (kd , swap , 1 , 0 ) ;i f (n < 0 | | swap [ 0 ] . k sw to ta l == 0)

va l . f = 0 ;totswap = swap [ 0 ] . k sw to ta l ;

}va l . f = totswap ∗ ( page s i z e / 1024 ) ;

The Ganglia metrics for memory use were some ofthe hardest to port and some of the ones we areleast satisfied with overall. The problem stems fromthe fact that the different virtual memory systems indifferent operating systems use vastly different setsof accounting buckets. The Ganglia metrics “total”,“buffers”, “cached”, “free”, and “shared” appear to bederived from values that are easy to obtain on Linux.Unfortunately, they do not match any of the mem-ory use statistics provided by FreeBSD. For exam-ple top shows “active”, “inactive”, “wired”, “cache”,“buffers”, and “free” memory, where “free” memoryis always a small number on a system that has beenup and active for some time since the VM system seeslittle need to waste CPU time freeing memory thatmight be used again. As the saying goes, free memoryis wasted memory in the operating system. Even to-tal memory is complicated. For example we have bothhw.physmem which is the actual amount of memory inthe system (modulo 32-bit limits on 32-bit, non-PAEsystem) and hw.realmem, the memory that it is pos-sible to use. Table 1 shows the set of mappings wechose, but this is far from optimal.

Over all Ganglia was a straight forward package toport. Supporting multiple FreeBSD versions requiredsome effort, but most metrics were easy to support.

2.2 Sun Grid Engine

Sun Grid Engine (SGE)[SGE] is a leading open sourcebatch scheduler and resource manager. The othercredible option is Torque, a fork of OpenPBS. Prior

Ganglia Metric sysctl Used in Portmem total hw.physmemmem buffers vfs.bufspacemem cached vm.stats.vm.v cache countmem free vm.stats.vm.v free countmem shared 0

Table 1: Ganglia Metric to sysctl Mappings

to Sun’s purchase of Grid Engine and the subsequentopen source release, we had been attempting to getOpenPBS to work in our environment, but it wasnever stable for us (or many others based on trafficon the on the OpenPBS mailing lists). When SGEwas released as open source in 2001 and Ron Chenstarted a port we leap at the chance to try somethingelse and completed the port.

The first step in porting SGE was figuring out thebuild system. It is a unique and complex systemconsisting of a nearly three-thousand line csh scriptnamed aimk (not be confused with the aimk build toolfrom PVM[PVM]). Within the script, each pair ofoperating system and architecture has a separate con-figuration section. The script invokes a dependencygenerator, a variety of make instances, configure insome cases, and in recent versions Apache Ant. His-torically configure was prerun for each platform andthus configure was not actually run by aimk.

In the process of porting the build system we intro-duced a number of innovations to reduce the complex-ity of adding new platforms. Most of these were re-lated to adding support for multi-platform builds sincewe knew from the start we wanted to support at leastFreeBSD i386 and amd64. We wanted to be able tosupport new architectures with little or no additionalchange to the build system. To that end we added asingle configuration section for FreeBSD architecturesand changed the architecture naming so that FreeB-SDs platform string is always fbsd-arch where arch isthe machine architecture as given by uname -m. Wealso augmented the build system so that for portionsof the source tree that depend on configure output, werun configure if no pregenerated output is available.

Because the build system requires many steps to gen-erate a working system, we also created a FreeBSDport[Ports] early on so we can build the system re-peatably with a reasonable number of command invo-cations. This greatly simplified the process of testingnew features.

One section we have not yet ported is the code to buildthe Java interfaces. Java use started with an interfaceto the DRMAA job submission interface, but now in-cludes a GUI installer. At one point we were able toharvest Java components from the pre-build binaries,but those are no longer available under an appropri-ate license so we have disabled all Java support in theport.

After the build system, we tackled a variety of porta-bility definitions. Early versions of Grid Engine wereported to systems that significantly predate modernversions of the C standard such as C99. As a resultthere are several sets of type definitions for fixed width

integers as well as a number of definitions to handledifferences in handling required to print a number ofstandard Unix typedefs like uid t, gid t, and pid t.For example these are some of the FreeBSD specificdefinitions are shown in Listing 4.

Listing 4: cpu num func()

#define u long32 u in t 32 t#define u id t fmt ”%u”#define g i d t fmt ”%u”#define p id t fmt ”%d”

A fair bit of this code could be simplified today, butdoing so would probably require removing support forsome older systems like Cray and HP-UX and the de-velopers have not been willing to do so.

Like Ganglia, SGE collects a number of metrics fromthe execution daemons on each node and uses thoseresults to make job placement decisions. The imple-mentation is not as neatly divided into metric func-tions, but the basic principles of porting metrics arethe same so we will not cover them here. We foundthe easiest way to find them was to search for LINUX

in the source tree to find all the things Linux had toimplement. The Darwin and NetBSD ports seem tohave searched for FREEBSD to aid their porting.

One place where SGEs metrics differ from Gangliais that SGE wants to track resource use by all theprocesses that make up a job. This is done in thePortable Data Collector sub system. On Irix, there isa mechanism for attaching job IDs to processes andthen querying resource use for the process and all itschildren (as well as sending signals to the group). Thisfeature does not exist on most other platforms so SGEimplements a clever hack to achieve a similar effect.The trick is that they allocate an otherwise unusedgroup on each node and then add it to the group listfor each process. Since the group list is copied to eachchild on fork() and ordinary users can not adjust theirgroup list this provides a tag which can be used todetect the processes that make up a job. When SGEwants to determine the cumulative resource use of ajob, it walks the process table using kvm and totals allresource use. Recent changes to FreeBSD (available in7.3 and 8.0) will allow the use of sysctl to perform thistask, but that will not be a portable option for sometime.

The SGE port took some significant work and stillcould use some more polish, but the individual pieceswere relatively simple to port and the current resultis quite useful.

2.3 Open MPI

Open MPI[Open MPI] is a leading open source im-plementation of the Message Passing Interface[MPI].MPI is the primary interface for building messagepassing parallel application. These are applicationswhere multiple process coordinate their computationswith messages. MPI hides the details of the under-lying network beneath a common API. Open MPI isalso the basis of commercial toolkits such as Sun’sHPC toolkit.

Overal, Open MPI was one of the easiest ports wehave made. The code is highly modular with exten-sive use of autoconf scripts to detect features. It isclear the development team has taken to heart thehistorical need for portability in HPC code, especiallymiddleware. The one feature we needed in the initialport was the backtrac() and backtrac symbols() func-tions glibc implements. Fortunately the devel/libex-ecinfo port already provides that functionality so wesimply added the necessary autoconf bits to do detectit and we had a working port.

One optional feature we have not yet ported is CPUaffinity. The Open MPI team created the PortableLinux Processor Affinity (PLPA) framework to dealwith ABI issues in the Linux CPU affinity systemcalls. We could easily support the same interface with-out the tricks they play in the name of binary com-patibility, but have not done so yet. PLPA has beensuperseded in Open MPI by a combination of its func-tionality and libtopology[libtopology] in a new pack-age called Portable Hardware Locality (hwloc), butporting PLPA may still be useful because it has beenincorporated into recent SGE releases.

3 Porting Issues

3.1 IPv6 Socket Behavior

One of the more annoying porting issues we encoun-tered in Ganglia was with IPv6 support. KAME de-rived IPv6 stacks violate RFC 2553 and disallow IPv4mapped IPv6 sockets from receiving IPv4 data and in-stead require two sockets to be opened. This is donefor relatively well justified security reasons[Metz], butcan be a significant portability issue. When Gan-glia introduced IPv6 support they did so using theApache Portable Runtime[APR] which did not un-derstand this issue. As a result, IPv6 support wasdisabled on FreeBSD for some time. After many dis-cussions, Ganglia was modified to open two sockets as

required under FreeBSD.

Given that code tends to expect this behavior to workand that changes to the RFCs have not been forthcom-ing, we think the BSD default should be reevaluated.

3.2 Signal Handling

Another interesting portability issue we have encoun-tered was with signal handlers. At some point theSGE developers fixed some bugs by adding persistentsignal handlers. Unfortunately, they used the easyto use, but not entirely portable sigset () function.POSIX defines the sigset () function, but FreeBSDdoes not yet implement it. In practice our imple-mentation of signal () is equivalent to sigset (), butPOSIX compliant implementations are not. At thetime we were able to persuade them to switch all in-stances of sigset () or signal () to sigaction () which iswell defined, but has the unfortunate characteristic ofrequiring multiple lines of code to replace each simplesignal () or sigset () call. Since then further sigset ()

calls have been introduced.

The overall state of affairs with respect to signal han-dling is a mess. This quote from Jim Frost[Frost] sumsup the situation:

To make certain that no one could write aneasily portable application, the POSIX com-mittee added yet another signal handling en-vironment which is supposed to be a supersetof BSD and both System-V environments.

The POSIX sigaction () function is well defined, buttoo hard to use. FreeBSD needs to follow NetBSDand implement sigset () and related functions.

3.3 Schema Conflicts

When porting Ganglia one of the more vexing issueswe ran into was matching the Linux derived schemato represent memory use with the available metrics inFreeBSD. This is a common problem in monitoringsystems, some other examples include: Network pro-tocol statistics where different implementations maypresent much more or less information or present thesame information is different ways. Determining thedisk space used or available on disks and wanting todistinguish between local, remote, and pseudo file sys-tems. Programs tend to end up with hard coded listsof local file system types. This scales poorly and failsto answer questions like, “Are UFS file systems on aSAN local or remote?”

As porters there often is little we can do about anexisting schema. The best we can do is encouragestandard and rational schemas where possible. Agood source of well though out schemas is SNMPMIBs[RFC3411].

3.4 Lack of Good Interfaces

One final general portability issue is a lack of good in-terfaces to some operating system data. For examplean SGE port would have been much simpler if therewere a way to track the resource of a process and allits children and to send signals to all of them suchas the Irix jid t . An interface that does not requireaccess to kernel memory is now available, but is notparticularly efficient. Another program that suffersfrom these problems is lsof[Abell] which currently ac-cesses kernel memory directly to retrieve all mannerof file handle related data. As a result it breaks ona regular basis when kernel developers move thingsaround. Creating proper interfaces to these sorts ofdata would be useful, but it is often difficult to de-termine which needs to be exposed when developingfeatures. Additionally, poorly thought out interfacescan lead to maintenance problems down the road asinterfaces like the routing socket demonstrate.

In FreeBSD is adding new interfaces at a steady rateand is developing a more mature understand of inter-face design. Some areas still need work and a numberof interfaces need significant modernization, but theproject is heading in the right direction.

4 Conclusions and Future Work

We have covered our ports of the high performancecomputing tools Ganglia, Sun Grid Engine, and OpenMPI to FreeBSD. In all three cases the basic task ofporting was relatively straight forward. Where we en-countered problems we noted how we resolved ma-jor issues and have discussed ways FreeBSD could bechanged to ease future porting efforts.

We believe that porting HPC tools to FreeBSD issomething more people could do and we encourageothers to jump in and help out. Some tools that wouldbe beneficial to port include: PLPA and its successorhwloc; various PAPI[PAPI] based performance anal-ysis tools; the Eclipse Parallel Tools Platform[PTP];and the ROCKS cluster system[Rocks]. The portingof PAPI based tools could provide a significant com-petitive advantage to FreeBSD as the mainline Linuxkernel and most Linux distributions to not include na-

tive support for hardware performance counters, butFreeBSD ships with hwpmc[FreeBSD-hwpmc] enabledby default.

References

[Abell] Vic Abell, Frequently Asked Questions aboutlsof. ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ. January 18, 2010.

[APR] The Apache Portable Runtime Project.http://apr.apache.org/

[Davis] Brooks Davis, Michael AuYeung, J. MattClark, Craig Lee, James Palko, Mark Thomas,Reflections on Building a High-performanceComputing Cluster Using FreeBSD.Proceedings, AsiaBSDCon 2007.

[FreeBSD-hwpmc] The FreeBSD Project, hwpmc(4)FreeBSD Kernel Interfaces Manual,http://www.freebsd.org/cgi/man.cgi?query=kvm&manpath=FreeBSD+8-currentSeptember 22, 2008.

[FreeBSD-kvm] The FreeBSD Project, kvm(3),FreeBSD Library Functions Manual,http://www.freebsd.org/cgi/man.cgi?query=kvm&manpath=FreeBSD+8-currentJanuary 29, 2004.

[FreeBSD-pstat] The FreeBSD Project, pstat(8),FreeBSD System Manager’s Manual,http://www.freebsd.org/cgi/man.cgi?query=pstat&manpath=FreeBSD+8-currentAugust 20, 2008.

[Frost] Jim Frost UNIX Signals and Process Groups.http://www.cs.ucsb.edu/~almeroth/classes/W99.276/assignment1/signals.htmlAugust 17, 1994.

[libtopology] libtopologyhttp://libtopology.ozlabs.org/

[Massie] Matthew L. Massie and Brent N. Chun andDavid E. Culler, The Ganglia DistributedMonitoring System: Design, ImplementationAnd Experience. Parallel Computing. Volume30. 2003.

[Metz] Craig Metz and Jun-ichiro itojun Hagino,IPv4-Mapped Addresses on the Wire ConsideredHarmful. http://tools.ietf.org/id/draft-itojun-v6ops-v4mapped-harmful-02.txt October 21, 2003.

[MPI] Message Passing Inteface Forumhttp://www.mpi-forum.org/

[Open MPI] Open MPI http://www.open-mpi.org/

[PAPI] PAPI: Performance ApplicationProgramming Interface.http://icl.cs.utk.edu/papi/

[PlanetLab] PlanetLab: An open platform fordeveloping, deploying, and accessingplanetary-scale services.http://www.planet-lab.org/

[Ports] The FreeBSD Collection Portshttp://www.freebsd.org/ports/index.html

[PTP] PTP - Parallel Tools Platform.http://www.eclipse.org/ptp/

[PVM] PVM: Parallel Virtual Machinehttp://www.csm.ornl.gov/pvm/

[RFC3411] D. Harrington, R. Presuhn, B. Wijen, AnArchitecture for Describing Simple NetworkManagement Protocol (SNMP) ManagementFrameworks. RFC 3411. Internet EngineeringTaskforce, Network Working Group. December2002.

[Rocks] Rocks Clusters.http://www.rocksclusters.org

[RRDtool] RRDtoolhttp://oss.oetiker.ch/rrdtool/

[SGE] Sun Grid Engine Projecthttp://gridengine.sunsource.net/

[FreeBSD-cpuaff] The FreeBSD Project,cpuset getaffinity(2), FreeBSD System CallsManual, http://www.freebsd.org/cgi/man.cgi?query=cpuset_getaffinity&manpath=FreeBSD+8-current March 29, 2008.

[FreeBSD-cpuset] The FreeBSD Project, cpuset(2),FreeBSD System Calls Manual,http://www.freebsd.org/cgi/man.cgi?query=cpuset&manpath=FreeBSD+8-currentMarch 29, 2008.