Meeting of the Technical Steering Committee (TSC) Board

Tuesday, October 10th11:00am ET

http://openhpc.community

Meeting  Logistics

• https://www.uberconference.com/jeff_ef

• United  States  :  +1  (510)  224-­9559  (No  PIN  needed).

Antitrust Policy Notice

• Linux Foundation meetings involve participation by industry competitors, and it is the intention of the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws.

• Examples of types of actions that are prohibited at Linux Foundation meetings and in connection with Linux Foundation activities are described in the Linux Foundation Antitrust Policy available at http://www.linuxfoundation.org/antitrust-policy. If you have questions about these matters, please contact your company counsel, or if you are a member of the Linux Foundation, feel free to contact Andrew Updegrove of the firm of Gesmer Updegrove LLP, which provides legal counsel to the Linux Foundation.

3

Agenda

• SC’17 BoF Update- Schedule update: Wednesday, Nov 15: 12:15-13:15

• Linaro Connect - ARM Data Center Day (Sep 28)-multiple references to OpenHPC in talks (Riken, Sandia,

Renato/Linaro)

• Review Cycle #4

• Recipe default update

• PMIx

4

Review cycle #4

• Reviews for Cycle #4 are now complete

• Propose to target mpi4py for v1.3.3 release- baseline start from PR submitted by requester

• Propose to target LIKWID for future release (e.g. v1.3.4)- majority of discussion around LIKWID was on different methods to access H/W counter info

• direct access from user• access via setuid daemon• perf_event backend in kernel

- I believe general consensus for those voting to accept was to highlight use of the daemon• explicitly call out that this is setuid

5

Component  Name #  of  Reviewers

#  of  accepts

#  of  rejects

Avg.  Priority

mpi4py 7 7 0 7.4LIKWID 7 6(3*) 1 5.3

*  w/  changes

Recipe default update for ethernet

• An item we’ve had on the back burner from earlier this year (April) is to update the recipe to be more friendly to ethernet-only environment- think we’ve done a reasonable job calling out which MPI stacks are applicable

to a given interconnect, but some users have been confused about what other items are optional in ethernet-only environment

• Current recipes assume IB environment and call out separate mods if using OPA

• Updated approach targeted for v1.3.3- target ethernet environment and call out IB or OPA via optional stanzas

6

if [[ ${enable_ib} -eq 1 ]];then yum -y groupinstall "InfiniBand Support" yum -y install infinipath-psmsystemctl start rdma

fi...if [[ ${enable_opa} -eq 1 ]];then

yum -y install opa-basic-tools systemctl start rdma

fi

Recipe default update for ethernet

• This approach does necessitate a change in suggesting a default MPI stack

• Recall that we provide a variety of lmod-default variants so an admin can choose what stack is default in their environment

• From 1.3.2 docs:

• MVAPICH2 targets IB/OPA so not appropriate for default in ethernet recipe. Today, avail options are OpenMPI or MPICH

• Go with OpenMPI?

7

Install Guide (v1.3.2): CentOS7.3/x86 64 + Warewulf + SLURM

The llvm compiler toolchains are also provided as a standalone additional compiler family (ie. users caneasily switch between gcc/clang environments), but we do not provide the full complement of downstreamlibrary builds.

[sms]# yum -y install llvm4-compilers-ohpc

4.3 MPI Stacks

For MPI development support, OpenHPC provides pre-packaged builds for a variety of MPI families andtransport layers. Currently available options and their applicability to various network transports aresummarized in Table 1. The command that follows installs a default set of MPI families that providesupport for ethernet and InfiniBand environments. If installing on a system with an Intel® Omni-Pathfabric, use the psm2 variants for OpenMPI and MVAPICH instead (e.g. openmpi-psm2-gnu7-ohpc andmvapich2-psm2-gnu7-ohpc, respectively).

Table 1: Available MPI builds

Ethernet (TCP) InfiniBand Intel® Omni-Path

MPICH XMVAPICH2 XMVAPICH2 (psm2) XOpenMPI X XOpenMPI (psm2) X X X

[sms]# yum -y install openmpi-gnu7-ohpc mvapich2-gnu7-ohpc mpich-gnu7-ohpc

4.4 Performance Tools

OpenHPC provides a variety of open-source tools to aid in application performance analysis (refer to Ap-pendix E for a listing of available packages). This group of tools can be installed as follows:

# Install perf-tools meta-package[sms]# yum -y install ohpc-gnu7-perf-tools

4.5 Setup default development environment

System users often find it convenient to have a default development environment in place so that compilationcan be performed directly for parallel programs requiring MPI. This setup can be conveniently enabled viamodules and the OpenHPC modules environment is pre-configured to load an ohpc module on login (ifpresent). The following package install provides a default environment that enables autotools, the GNUcompiler toolchain, and the MVAPICH2 MPI stack.

[sms]# yum -y install lmod-defaults-gnu7-mvapich2-ohpc

23 Rev: 84312129

Install Guide (v1.3.2): CentOS7.3/x86 64 + Warewulf + SLURM

Tip

If you want to change the default environment from the suggestion above, OpenHPC also provides the GNUcompiler toolchain with the OpenMPI and MPICH stacks:

• lmod-defaults-gnu7-openmpi-ohpc• lmod-defaults-gnu7-mpich-ohpc

4.6 3rd Party Libraries and Tools

OpenHPC provides pre-packaged builds for a number of popular open-source tools and libraries used by HPCapplications and developers. For example, OpenHPC provides builds for FFTW and HDF5 (including serialand parallel I/O support), and the GNU Scientific Library (GSL). Again, multiple builds of each packageare available in the OpenHPC repository to support multiple compiler and MPI family combinations whereappropriate. Note, however, that not all combinatorial permutations may be available for components wherethere are known license incompatibilities. The general naming convention for builds provided by OpenHPCis to append the compiler and MPI family name that the library was built against directly into the packagename. For example, libraries that do not require MPI as part of the build process adopt the following RPMname:

package-<compiler family>-ohpc-<package version>-<release>.rpm

Packages that do require MPI as part of the build expand upon this convention to additionally include theMPI family name as follows:

package-<compiler family>-<mpi family>-ohpc-<package version>-<release>.rpm

To illustrate this further, the command below queries the locally configured repositories to identify all ofthe available PETSc packages that were built with the GNU toolchain. The resulting output that is includedshows that pre-built versions are available for each of the supported MPI families presented in §4.3.

[sms]# yum search petsc-gnu7 ohpcLoaded plugins: fastestmirrorLoading mirror speeds from cached hostfile=========================== N/S matched: petsc-gnu7, ohpc ===========================petsc-gnu7-impi-ohpc.x86_64 : Portable Extensible Toolkit for Scientific Computationpetsc-gnu7-mpich-ohpc.x86_64 : Portable Extensible Toolkit for Scientific Computationpetsc-gnu7-mvapich2-ohpc.x86_64 : Portable Extensible Toolkit for Scientific Computationpetsc-gnu7-openmpi-ohpc.x86_64 : Portable Extensible Toolkit for Scientific Computation

Tip

OpenHPC-provided 3rd party builds are configured to be installed into a common top-level repository so thatthey can be easily exported to desired hosts within the cluster. This common top-level path (/opt/ohpc/pub)was previously configured to be mounted on compute nodes in §3.7.3, so the packages will be immediatelyavailable for use on the cluster after installation on the master host.

For convenience, OpenHPC provides package aliases for these 3rd party libraries and utilities that canbe used to install available libraries for use with the GNU compiler family toolchain. For parallel libraries,aliases are grouped by MPI family toolchain so that administrators can choose a subset should they favor a

24 Rev: 84312129

PMIx

• Another item on the backburner from Review Cycle #3 is the inclusion and usage of PMIx

• Reminder on long term intent: resource managers adopting PMIx over time so we can leverage as unified job launch support library

• SLURM has started to include PMIx in it’s distribution, but prefer external build for use by multiple resource managers in the future

8

PMIxSLURM

PBS

OpenMPI

MPICH

MVAPICH2

Desired End Game

PMIx

• After working on this for current release, have some issues and subtleties to discuss on how we would like to proceed

• Current state:- latest SLURM supports PMIx 1.x (support for linkage against external build

or use distribution included with SLURM)

- latest OpenMPI (3.x) supports native use of PMIx

- PBS committed to supporting PMIx at some point, but not available yet. We continue to use latest PBS Pro open source release (June 2016)

- MPICH and MVAPICH2 do not have direct PMIx support, but have capability to use PMI1 or PMI2 (typically the variants provided by SLURM)• PMIx provides backwards compatibility for PMI1/2 via conversion to PMIx

equivalents• intent has been to leverage these to gain support for MPICH variants

9

PMIx

• The good news:- OpenMPI (3.x) works

• can have single build of OpenMPI that uses PMIx under SLURM, no PMIx under PBS Pro• prun launcher updated to launch via srun --mpi=pmix

- thanks to Adrian, can also have single OpenMPI build support ethernet, IB, and OPA

• The ok news:- can get MPICH builds to use PMI interfaces provided by PMIx to run under SLURM- initial testing showed failures with singleton (ie ./a.out); reported to PMIx developer and

have patch in place to resolve that

- enabling PMI-based job launch with MPICH disables standalone job launch (ie. no mpiexec.hydra). Even if you build/install that by hand, unable to use (PMI provides a standard API for client/server, but does not specify protocol)• hence, cannot have single PMIx-enabled MPICH build for both SLURM/PBS today• we presumably could once we have a PBS release that supports PMIx• added environment variable to resulting MPICH module to indicate pmix support (so prun

can act accordingly):– setenv("OHPC_MPI_LAUNCHERS","pmix")

10

PMIx

• The bad news:- at present, cannot get MVAPICH2 to work using PMI interfaces provided by

PMIX• iterating with PMIx developer, not clear this will work, but maybe....• if it does, will have similar issue of no longer having mpiexec.hydra available for

use with PBS• likely require separate build for PBS

• Other items to consider- to date, our MPI stacks have no formal requirements on the resource manager- if we introduce PMIx in an update, we have to be careful to account for existing

systems that have SLURM/PBS installed- in particular, there is an implied version requirement for OpenMPI and MPICH

to use PMIx (ie. the new build of SLURM with PMIx support enabled is needed)- having a single build of the MPI stack then becomes problematic as it would

pull in resource manager as well

11

PMIx - potential packaging options

• Option #1- leave current MPI build configurations mostly as is (without PMIx), but enable

additional variant(s) for MPIs where it works, e.g. openmpi3-pmix-slurm-gnu7-ohpc, mpich-pmix-slurm-gnu7-ohpc

• Option #2:- move to convention where MPI stacks are RMS specific and add formal version

requirements (w/ PMIx enabled)- e.g. mpich-slurm-gnu7-ohpc, mpich-pbs-gnu7-ohpc- potential gotcha: folks would be unable to use some of the MPI stacks outside of

resource manager (ie. no mpiexec.hydra for use with hostfile/ssh)

• Option #3- hold off for now: wait till PMIx supported by both resource managers

• Option #4- don’t add formal requirements but call out the need to upgrade SLURM in release

notes if desiring to use latest MPI builds

• Thoughts/discussion?

12