Cirrus Documentation - Read the Docs · PDF file17.2 Using LAMMPS on Cirrus ... A typical...

Cirrus DocumentationRelease 1.1

EPCC

May 03, 2018

Cirrus User Guide

1 Introduction 31.1 Acknowledging Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Useful terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Connecting to Cirrus 52.1 Interactive access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 SSH Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Making access more convenient using a SSH Agent . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Data Transfer Guide 113.1 scp command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 rsync command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.3 Performance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 File and Resource Management 134.1 The Cirrus Administration Web Site (SAFE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Checking your CPU-time allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.3 Disk quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.4 File permissions and security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.5 File IO Performance Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.6 Backup policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5 Application Development Environment 175.1 Using the modules environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.2 Available Compiler Suites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.3 Compiling MPI codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.4 Compiler Information and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235.5 Using static linking/libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6 Running Jobs on Cirrus 256.1 Using PBS Pro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.2 Queue Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.3 Output from PBS jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.4 Running Parallel Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.5 Running MPI parallel jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.6 Example parallel MPI job submission scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

i

6.7 Serial Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.8 Job arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346.9 Interactive Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.10 Reservations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7 Singularity Containers 397.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.2 About Singularity Containers (Images) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.3 Using Singularity Images on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407.4 Creating Your Own Singularity Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

8 Using Python 458.1 Accessing central Anaconda Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458.2 Custom Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

9 References and further reading 519.1 Online Documentation and Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519.2 MPI programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519.3 OpenMP programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519.4 Parallel programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.5 Programming languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.6 Programming skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

10 Altair Hyperworks 5310.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5310.2 Using Hyperworks on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5310.3 Running serial Hyperworks jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5310.4 Running parallel Hyperworks jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

11 ANSYS Fluent 5711.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5711.2 Using ANSYS Fluent on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5711.3 Running parallel ANSYS Fluent jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5711.4 Actual Fluent script (“inputfile.fl”): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

12 CASTEP 6112.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6112.2 Using CASTEP on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6112.3 Running parallel CASTEP jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

13 CP2K 6313.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6313.2 Using CP2K on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6313.3 Running parallel CP2K jobs - MPI Only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6313.4 Running Parallel CP2K Jobs - MPI/OpenMP Hybrid Mode . . . . . . . . . . . . . . . . . . . . . . . 64

14 FLACS 6714.1 FLACS-Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6714.2 FLACS-HPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6814.3 Getting help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

15 Gaussian 7115.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7115.2 Using Gaussian on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7115.3 Scratch Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

ii

15.4 Running serial Gaussian jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7215.5 Running parallel Gaussian jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

16 GROMACS 7516.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7516.2 Using GROMACS on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7516.3 Running parallel GROMACS jobs: pure MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7516.4 Running parallel GROMACS jobs: hybrid MPI/OpenMP . . . . . . . . . . . . . . . . . . . . . . . . 76

17 HELYX® 7717.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7717.2 Using HELYX on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7717.3 Running Parallel HELYX Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

18 LAMMPS 7918.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7918.2 Using LAMMPS on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7918.3 Running parallel LAMMPS jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7918.4 Compiling LAMMPS on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

19 MATLAB 8119.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8119.2 Using MATLAB on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8119.3 Running parallel MATLAB jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

20 NAMD 8520.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8520.2 Using NAMD on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8520.3 Running parallel NAMD jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

21 OpenFOAM 8721.1 Available Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8721.2 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8721.3 Using OpenFOAM on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

22 Quantum Espresso (QE) 8922.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8922.2 Using QE on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8922.3 Running parallel QE jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

23 STAR-CCM+ 9123.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9123.2 Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9123.3 Using STAR-CCM+ on Cirrus: Interactive remote GUI Mode . . . . . . . . . . . . . . . . . . . . . 91

24 VASP 9524.1 Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9524.2 Using VASP on Cirrus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9524.3 Running parallel VASP jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

25 Intel MKL: BLAS, LAPACK, ScaLAPACK 9725.1 Intel Compilers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9725.2 GNU Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

26 Debugging using Allinea DDT 9926.1 Debugging runs on the login nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

iii

26.2 Debugging runs on the compute nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9926.3 Memory debugging with DDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10026.4 Getting further help on DDT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

27 Transferring data from INDY to Cirrus 10127.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10127.2 Minimum Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10127.3 Data Transfer via SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

iv

Cirrus Documentation, Release 1.1

Cirrus is a HPC and data science service hosted and run by EPCC at The University of Edinburgh. It is one of theEPSRC Tier-2 National HPC Services.

Cirrus is available to industry, commerce and academic researchers. For information on how to get access to thesystem please see the Cirrus website.

The Cirrus facility is based around an SGI ICE XA system with 10,080 cores. There are 280 compute nodes, eachwith 256 GB of memory, connected together by a single Infiniband fabric. Each node contains two 2.1 GHz, 18-coreIntel Xeon processors, and all nodes access the 406 TiB Lustre file system.

This documentation covers:

• Cirrus User Guide: general information on how to use Cirrus

• Software Applications: notes on using specific software applications on Cirrus

• Software Libraries: notes on compiling against specific libraries on Cirrus. Most libraries work as expected sono additional notes are required however a small number require specific documentation

• Software Tools: Information on using tools such as debuggers and profilers on Cirrus

Information on using the SAFE web interface for managing and reporting on your usage on Cirrus can be found onthe Tier-2 SAFE Documentation

This documentation draws on the Sheffield Iceberg Documentation and the documentation for the ARCHER NationalSupercomputing Service.

Cirrus User Guide 1

http://www.epcc.ed.ac.uk

http://www.ed.ac.uk

http://www.epsrc.ac.uk

http://www.cirrus.ac.uk

http://tier2-safe.readthedocs.io/en/latest/

https://github.com/rcgsheffield/sheffield_hpc

http://www.archer.ac.uk

http://www.archer.ac.uk


2 Cirrus User Guide

CHAPTER 1

Introduction

This guide is designed to be a reference for users of the high-performance computing (HPC) facility: Cirrus. Itprovides all the information needed to access the system, transfer data, manage your resources (disk and computetime), submit jobs, compile programs and manage your environment.

1.1 Acknowledging Cirrus

You should use the following phrase to acknowledge Cirrus in all research outputs that have used the facility:

This work used EPCC’s Cirrus HPC Service (https://www.epcc.ed.ac.uk/cirrus).

You should also tag outputs with the keyword Cirrus whenever possible.

1.2 Hardware

Details of the Cirrus hardware are available on the Cirrus website:

• Cirrus Hardware

1.3 Useful terminology

This is a list of terminology used throughout this guide and its meaning.

kAU Cirrus CPU time is measured in kAU. Each job you run on the service consumes kAUs from your budget. Youcan find out more about kAUs and how to track your usage in the File and Resource Management

3

http://www.cirrus.ac.uk/about/hardware.html


4 Chapter 1. Introduction

CHAPTER 2

Connecting to Cirrus

On the Cirrus system interactive access can be achieved via SSH, either directly from a command line terminal orusing an SSH client. In addition data can be transfered to and from the Cirrus system using scp from the commandline or by using a file transfer client.

This section covers the basic connection methods. The connection procedure is then expanded on and the use of SSHagent is described for ease of access.

2.1 Interactive access

To log into Cirrus you should use the “login.cirrus.ac.uk” address:

ssh [userID]@login.cirrus.ac.uk

2.1.1 Initial passwords

The SAFE web interface is used to provide your initial password for logging onto Cirrus (see the Tier-2 SAFE Docu-mentation for more details on requesting accounts and picking up passwords).

Note: you may now change your password on the Cirrus machine itself using the passwd command. This changewill not be reflected in the SAFE. If you forget your password, you should use the SAFE to request a new one-shotpassword.

2.2 SSH Clients

Interaction with Cirrus is done remotely, over an encrypted communication channel, Secure Shell version 2 (SSH-2).This allows command-line access to one of the login nodes of a Cirrus, from which you can run commands or use acommand-line text editor to edit files. SSH can also be used to run graphical programs such as GUI text editors anddebuggers when used in conjunction with an X client.

5

https://tier2-safe.readthedocs.io



2.2.1 Logging in from Linux and Macs

Linux distributions and OS X each come installed with a terminal application that can be use for SSH access to thelogin nodes. Linux users will have different terminals depending on their distribution and window manager (e.g.GNOME Terminal in GNOME, Konsole in KDE). Consult your Linux distribution’s documentation for details onhow to load a terminal.

OS X users can use the Terminal application, located in the Utilities folder within the Applications folder.

You can use the following command from the terminal window to login into Cirrus:

ssh [email protected]

To allow remote programs, especially graphical applications to control your local display, such as being able to openup a new GUI window (such as for a debugger), use:

ssh -X [email protected]

Some sites recommend using the -Y flag. While this can fix some compatibility issues, the -X flag is more secure.

Current OS X systems do not have an X window system. Users should install the XQuartz package to allow for SSHwith X11 forwarding on OS X systems:

• XQuartz website

2.2.2 Logging in from Windows using MobaXterm

A typical Windows installation will not include a terminal client, though there are various clients available. Werecommend all our Windows users to download and install MobaXterm to access Cirrus. It is very easy to use andincludes an integrated X server with SSH client to run any graphical applications on Cirrus.

You can download MobaXterm Home Edition (Installer Edition) from the following link:

• Install MobaXterm

Double-click the downloaded Microsoft Installer file (.msi), and the Windows wizard will automatically guides youthrough the installation process. Note, you might need to have administrator rights to install on some Windows OS.Also make sure to check whether Windows Firewall hasn’t blocked any features of this program after installation.

Start MobaXterm using, for example, the icon added to the Start menu during the installation process.

If you would like to run any small remote GUI applications, then make sure to use -X option along with the sshcommand (see above) to enable X11 forwarding, which allows you to run graphical clients on your local X server.

2.3 Making access more convenient using a SSH Agent

Using a SSH Agent makes accessing the resources more convenient as you only have to enter your passphrase onceper day to access any remote resource - this can include accessing resources via a chain of SSH sessions.

This approach combines the security of having a passphrase to access remote resources with the convenince of havingpassword-less access. Having this sort of access set up makes it extremely convenient to use client applications toaccess remote resources, for example:

• the Tramp Emacs plugin that allows you to access and edit files on a remote host as if they are local files;

• the Parallel Tools Platform for the Eclipse IDE that allows you to edit your source code on a local Eclipseinstallation and compile and test on a remote host;

6 Chapter 2. Connecting to Cirrus

http://www.xquartz.org/

http://mobaxterm.mobatek.net/download-home-edition.html

http://www.gnu.org/software/tramp/

http://www.eclipse.org/ptp/


Note: this description applies if your local machine is Linux or macOS. The procedure can also be used on Windowsusing the PuTTY SSH terminal with the PuTTYgen key pair generation tool and the Pageant SSH Agent. See thePuTTY documentation for more information on how to use these tools.

Note: not all remote hosts allow connections using a SSH key pair. If you find this method does not work it is worthchecking with the remote site that such connections are allowed.

2.3.1 Setup a SSH key pair protected by a passphrase

Using a terminal (the command line), set up a key pair that contains your e-mail address and enter a passphrase youwill use to unlock the key:

ssh-keygen -t rsa -C "[email protected]"...-bash-4.1$ ssh-keygen -t rsa -C "[email protected]"Generating public/private rsa key pair.Enter file in which to save the key (/Home/user/.ssh/id_rsa): [Enter]Enter passphrase (empty for no passphrase): [Passphrase]Enter same passphrase again: [Passphrase]Your identification has been saved in /Home/user/.ssh/id_rsa.Your public key has been saved in /Home/user/.ssh/id_rsa.pub.The key fingerprint is:03:d4:c4:6d:58:0a:e2:4a:f8:73:9a:e8:e3:07:16:c8 [email protected] key's randomart image is:+--[ RSA 2048]----+| . ...+o++++. || . . . =o.. ||+ . . .......o o ||oE . . ||o = . S ||. +.+ . ||. oo ||. . || .. |+-----------------+

(remember to replace “[email protected]” with your e-mail address).

2.3.2 Copy the public part of the key to the remote host

Using you normal login password, add the public part of your key pair to the “authorized_keys” file on the remote hostyou wish to connect to using the SSH Agent. This can be achieved by appending the contents of the public part of thekey to the remote file:

-bash-4.1$ cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat - >> ~/.ssh/→˓authorized_keys'Password: [Password]

(remember to replace “user” with your username).

Now you can test that your key pair is working correctly by attempting to connect to the remote host and run acommand. You should be asked for your key pair passphase (which you entered when you creasted the key pair)rather than your remote machine password.

2.3. Making access more convenient using a SSH Agent 7

http://the.earth.li/~sgtatham/putty/0.62/htmldoc/

mailto:[email protected]


-bash-4.1$ ssh [email protected] 'date'Enter passphrase for key '/Home/user/.ssh/id_rsa': [Passphrase]Wed May 8 10:36:47 BST 2013


2.3.3 Enabling the SSH Agent

So far we have just replaced the need to enter a password to access a remote host with the need to enter a key pairpassphrase. The next step is to enable an SSH Agent on your local system so that you only have to enter the passphraseonce per day and after that you will be able to access the remote system without entering the passphrase.

Most modern Linux distributions (and macOS) should have ssh-agent running by default. If your system does not thenyou should find the instructions for enabling it in your distribution using Google.

To add the private part of your key pair to the SSH Agent, use the ‘ssh-add’ command (on your local machine), youwill need to enter your passphrase one more time:

-bash-4.1$ ssh-add ~/.ssh/id_rsaEnter passphrase for Home/user.ssh/id_rsa: [Passphrase]Identity added: Home/user.ssh/id_rsa (Home/user.ssh/id_rsa)

Now you can test that you can access the remote host without needing to enter your passphrase:

-bash-4.1$ ssh [email protected] 'date'Warning: Permanently added the RSA host key for IP address '192.62.216.27' to the→˓list of known hosts.Wed May 8 10:42:55 BST 2013


2.3.4 Adding access to other remote machines

If you have more than one remote host that you access regularly, you can simply add the public part of your key pairto the ‘authorized_keys’ file on any hosts you wish to access by repeating step 2 above.

2.3.5 SSH Agent forwarding

Now that you have enabled an SSH Agent to access remote resources you can perform an additional configuration stepthat will allow you to access all hosts that have your public key part uploaded from any host you connect to with theSSH Agent without the need to install the private part of the key pair anywhere except your local machine.

This increases the security of the key pair as the private part is only stored in one place (your local machine) andmakes access more convenient (as you only need to enter your passphrase once on your local machine to enable accessbetween all machines that have the public part of the key pair).

Forwarding is controlled by a configuration file located on your local machine at “.ssh/config”. Each remote site (orgroup of sites) can have an entry in this file which may look something like:

Host cirrusHostName login.cirrus.ac.ukUser userForwardAgent yes




The “Host cirrus” line defines a short name for the entry. In this case, instead of typing “ssh login.cirrus.ac.uk” toaccess the Cirrus login nodes, you could use “ssh cirrus” instead. The remaining lines define the options for the“cirrus” host.

• Hostname login.cirrus.ac.uk - defines the full address of the host

• User username - defines the username to use by default for this host (replace “username” with your ownusername on the remote host)

• ForwardAgent yes - tells SSH to forward the local SSH Agent to the remote host, this is the option thatallows you to store the private part of your key on your local machine only and export the access to remote sites

Now you can use SSH to access Cirrus without needing to enter my username or the full hostname every time:

-bash-4.1$ ssh cirrus 'date'Tue Dec 20 16:48:32 GMT 2016

You can set up as many of these entries as you need in your local configuration file. Other options are available. Seethe ssh_config man page (or man ssh_config on any machine with SSH installed) for a description of the SSHconfiguration file.

2.3. Making access more convenient using a SSH Agent 9

http://linux.die.net/man/5/ssh_config

CHAPTER 3

Data Transfer Guide

This section covers the different ways that you can transfer data on to and off Cirrus.

3.1 scp command

The scp command creates a copy of a file, or if given the -r flag a directory, on a remote machine. Below shows anexample of the command to transfer a single file to Cirrus:

scp [options] source_file [email protected]:[destination]

In the above example, the [destination] is optional, as when left out scp will simply copy the source into theusers home directory.

3.2 rsync command

The rsync command creates a copy of a file, or if given the -r flag a directory, at the given destination, similar toscp above. However, given the -a option rsync can also make exact copies (including permissions), this is referredto as ‘mirroring’. In this case the rsync command is executed with ssh to create the copy of a remote machine. Totransfer files to Cirrus the command should have the form:

rsync [options] -e ssh source [email protected]:[destination]

In the above example, the [destination] is optional, as when left out rsync will simply copy the source into theusers home directory.

3.3 Performance considerations

Cirrus is capable of generating data at a rate far greater than the rate at which this can be downloaded. In practice, it isexpected that only a portion of data generated on Cirrus will be required to be transfered back to users’ local storage -

11


the rest will be, for example, intermediate or checkpoint files required for subsequent runs. However, it is still essentialthat all users try to transfer data to and from Cirrus as efficiently as possible. The most obvious ways to do this are:

1. Only transfer those files that are required for subsequent analysis, visualisation and/or archiving. Do you reallyneed to download those intermediate result or checkpointing files? Probably not.

2. Combine lots of small files into a single tar file, to reduce the overheads associated in initiating data transfers.

3. Compress data before sending it, e.g. using gzip or bzip2.

4. Consider doing any pre- or post-processing calculations on Cirrus. Long running pre- or post- processing cal-culations should be run via the batch queue system, rather than on the login nodes. Such pre- or post-processingcodes could be serial or OpenMP parallel applications running on a single node, though if the amount of data tobe processed is large, an MPI application able to use multiple nodes may be preferable.

Note: that the performance of data transfers between Cirrus and your local institution may differ depending uponwhether the transfer command is run on Cirrus or on your local system.

12 Chapter 3. Data Transfer Guide

CHAPTER 4

File and Resource Management

This section covers some of the tools and technical knowledge that will be key to maximising the usage of the Cirrussystem, such as the online administration tool SAFE and calculating the CPU-time available.

The default file permissions are then outlined, along with a description of changing these permissions to the desiredsetting. This leads on to the sharing of data between users and systems often a vital tool for project groups andcollaboration.

Finally we cover some guidelines for I/O and data archiving on Cirrus.

4.1 The Cirrus Administration Web Site (SAFE)

All users have a login and password on the Cirrus Administration Web Site (also know as the ‘SAFE’): SAFE. Oncelogged into this web site, users can find out much about their usage of the Cirrus system, including:

• Account details - password reset, change contact details

• Project details - project code, start and end dates

• kAU balance - how much time is left in each project you are a member of

• Filesystem details - current usage and quotas

• Reports - generate reports on your usage over a specified period, including individual job records

• Helpdesk - raise queries and track progress of open queries

4.2 Checking your CPU-time allocation

You can view these details by logging into the SAFE (https://www.archer.ac.uk/tier2/).

Use the Login accounts menu to select the user account that you wish to query. The page for the login account willsummarise the resources available to account.

13

https://www.archer.ac.uk/tier2/



You can also generate reports on your usage over a particular period and examine the details of how many CPUhindividual jobs on the system cost. To do this use the Service information menu and selet Report generator.

4.3 Disk quotas

Disk quotas on Cirrus are managed via SAFE

4.4 File permissions and security

By default, each user is a member of the group with the same name as [group_code] in the /lustre/home directorypath, e.g. x01. This allows the user to share files with only members of that group by setting the appropriate groupfile access permissions. As on other UNIX or Linux systems, a user may also be a member of other groups. The listof groups that a user is part of can be determined by running the groups command.

Default Unix file permissions can be specified by the umask command. The default umask value on Cirrus is 22,which provides “group” and “other” read permissions for all files created, and “group” and “other” read and executepermissions for all directories created. This is highly undesirable, as it allows everyone else on the system to access(but at least not modify or delete) every file you create. Thus it is strongly recommended that users change this defaultumask behaviour, by adding the command umask 077 to their $HOME/.profile file. This umask setting onlyallows the user access to any file or directory created. The user can then selectively enable “group” and/or “other”access to particular files or directories if required.

4.4.1 ASCII (or formatted) files

These are the most portable, but can be extremely inefficient to read and write. There is also the problem that ifthe formatting is not done correctly, the data may not be output to full precision (or to the subsequently requiredprecision), resulting in inaccurate results when the data is used. Another common problem with formatted files isFORMAT statements that fail to provide an adequate range to accommodate future requirements, e.g. if we wish tooutput the total number of processors, NPROC, used by the application, the statement:

WRITE (*,'I3') NPROC

will not work correctly if NPROC is greater than 999.

4.4.2 Binary (or unformatted) files

These are much faster to read and write, especially if an entire array is read or written with a single READ or WRITEstatement. However the files produced may not be readable on other systems.

GNU compiler -fconvert=swap compiler option. This compiler option often needs to be used together with asecond option -frecord-marker, which specifies the length of record marker (extra bytes inserted beforeor after the actual data in the binary file) for unformatted files generated on a particular system. To read a binaryfile generated by a big-endian system on Cirrus, use -fconvert=swap -frecord-marker=4. Pleasenote that due to the same ‘length of record marker’ reason, the unformatted files generated by GNU and othercompilers on Cirrus are not compatible. In fact, the same WRITE statements would result in slightly larger fileswith GNU compiler. Therefore it is recommended to use the same compiler for your simulations and relatedpre- and post-processing jobs.

Other options for file formats include:

14 Chapter 4. File and Resource Management



Direct access files Fortran unformatted files with specified record lengths. These may be more portable betweendifferent systems than ordinary (i.e. sequential IO) unformatted files, with significantly better performance thanformatted (or ASCII) files. The “endian” issue will, however, still be a potential problem.

Portable data formats These machine-independent formats for representing scientific data are specifically designedto enable the same data files to be used on a wide variety of different hardware and operating systems. The mostcommon formats are:

• netCDF: http://www.unidata.ucar.edu/software/netcdf/

• HDF: http://www.hdfgroup.org/HDF5/

It is important to note that these portable data formats are evolving standards, so make sure you are aware ofwhich version of the standard/software you are using, and keep up-to-date with any backward-compatibilityimplications of each new release.

4.5 File IO Performance Guidelines

Here are some general guidelines

• Whichever data formats you choose, it is vital that you test that you can access your data correctly on allthe different systems where it is required. This testing should be done as early as possible in the softwaredevelopment or porting process (i.e. before you generate lots of data from expensive production runs), andshould be repeated with every major software upgrade.

• Document the file formats and metadata of your important data files very carefully. The best documentationwill include a copy of the relevant I/O subroutines from your code. Of course, this documentation must be keptup-to-date with any code modifications.

• Use binary (or unformatted) format for files that will only be used on the Intel system, e.g. for checkpointingfiles. This will give the best performance. Binary files may also be suitable for larger output data files, if theycan be read correctly on other systems.

• Most codes will produce some human-readable (i.e. ASCII) files to provide some information on the progressand correctness of the calculation. Plan ahead when choosing format statements to allow for future code usage,e.g. larger problem sizes and processor counts.

• If the data you generate is widely shared within a large community, or if it must be archived for future reference,invest the time and effort to standardise on a suitable portable data format, such as netCDF or HDF.

4.6 Backup policies

There are currently no backups of data on Cirrus as backing up the whole Lustre file system would adversly affectthe performance of write access for simulations. The nature of the Lustre parallel file system means that there is dataresiliance in the case of failures of individual hardware components. However, we strongly advise that you keep copiesof any critical data on different systems.

We are currently investigating options for providing backups of critical data.

4.5. File IO Performance Guidelines 15

http://www.unidata.ucar.edu/software/netcdf/

http://www.hdfgroup.org/HDF5/


16 Chapter 4. File and Resource Management

CHAPTER 5

Application Development Environment

The application development environment on Cirrus is primarily controlled through the modules environment. Byloading and switching modules you control the compilers, libraries and software available.

This means that for compiling on Cirrus you typically set the compiler you wish to use using the appropriate modules,then load all the required library modules (e.g. numerical libraries, IO format libraries).

Additionally, if you are compiling parallel applications using MPI (or SHMEM, etc.) then you will need to load oneof the MPI environments and use the appropriate compiler wrapper scripts.

By default, all users on Cirrus start with no modules loaded.

Basic usage of the module command on Cirrus is covered below. For full documentation please see:

• Linux manual page on modules

Note: The modules provided by the Spack package manager behave differently to those usually encountered in Linuxenvironments. In particular, each module has the versions of dependency libraries hardcoded using RPATH. Moreinformation is provided below. You can identify Spack modules as they have a random string of 7 characters at theend of their name, e.g.: fftw-3.3.5-intel-17.0.2-dxt2dzn.

5.1 Using the modules environment

5.1.1 Information on the available modules

Finding out which modules (and hence which compilers, libraries and software) are available on the system is per-formed using the module avail command:

[user@cirrus-login0 ~]$ module avail...

This will list all the names and versions of the modules available on the service. Not all of them may work in youraccount though due to, for example, licencing restrictions. You will notice that for many modules we have more thanone version, each of which is identified by a version number. One of these versions is the default. As the servicedevelops the default version will change.

17

http://linux.die.net/man/1/module

http://spack.readthedocs.io


You can list all the modules of a particular type by providing an argument to the module avail command. Forexample, to list all available versions of the Intel Compiler type:

[user@cirrus-login0 ~]$ module avail intel-compilers

--------------------------------------- /lustre/sw/modulefiles -----------------------→˓----------------intel-compilers-16/16.0.2.181 intel-compilers-16/16.0.3.210

If you want more info on any of the modules, you can use the module help command:

[user@cirrus-login0 ~]$ module help mpt

----------- Module Specific Help for 'mpt/2.14' -------------------

The SGI Message Passing Toolkit (MPT) is an optimized MPIimplementation for SGI systems and clusters. See theMPI(1) man page and the MPT User's Guide for moreinformation.

The simple module list command will give the names of the modules and their versions you have presently loadedin your envionment:

[user@cirrus-login0 ~]$ module listCurrently Loaded Modulefiles:1) mpt/2.14 3) intel-fc-16/16.0.3.2102) intel-cc-16/16.0.3.210 4) intel-compilers-16/16.0.3.210

5.1.2 Loading, unloading and swapping modules

To load a module to use module add or module load. For example, to load the intel-compilers-17 into thedevelopment environment:

module load intel-compilers-17

This will load the default version of the intel compilers. If you need a specfic version of the module, you can add moreinformation:

module load intel-compilers-17/17.0.2.174

will load version 17.0.2.174 for you, regardless of the default. If you want to clean up, module remove will removea loaded module:

module remove intel-compilers-17

(or module rm intel-compilers-17 or module unload intel-compilers-17) will unload whatever version of intel-compilers-17 (even if it is not the default) you might have loaded. There are many situationsin which you might want to change the presently loaded version to a different one, such as trying the latest versionwhich is not yet the default or using a legacy version to keep compatibility with old data. This can be achieved mosteasily by using “module swap oldmodule newmodule”.

Suppose you have loaded version 16.0.2.181, say, of intel-compilers-16, the following command will change to version16.0.3.210:

module swap intel-compilers-16 intel-compilers-16/16.0.2.181

18 Chapter 5. Application Development Environment


5.1.3 Modules provided by Spack

Care must be taken when using modules provided by Spack as they behave differently from standard Linux modules.

The Spack package management tool is used to manage much of the software and libraries installed on Cirrus. Spackallows us to automatically resolve dependencies and have multiple versions of tested software installed simultaneouslywithout them interfering with each other.

To achieve this, Spack makes use of RPATH to hardcode the paths of dependencies into libraries. This means thatwhen you load a module for a particular library you do not need to load any further modules for dependencies of thatlibrary.

For example, the boost toolkit depends on the MPI, zlib and bzip2 libraries:

[email protected]^[email protected]^[email protected]^[email protected]

Spack arranges things so that if you load the boost module:

module load boost-1.64.0-gcc-6.2.0-pftxg46

then you do not also need to load the bzip2, mpt and zlib modules.

This, however, can lead to behaviour that is unexpected for modules. For example, on Cirrus there are two versions ofzlib available: 1.2.8 and 1.2.10. You may imagine that you can use boost with zlib 1.2.8 with the following commands:

module load zlib-1.2.8-gcc-6.2.0-epathtpmodule load boost-1.64.0-gcc-6.2.0-pftxg46

but this will not work. boost will still use zlib 1.2.10 as the path to this is hrdcoded into boost itself via RPATH. Ifyou wish to use the older version of zlib then you must load it and then compile boost yourself.

If you wish to see what versions of libraries are hardcoded into a particular Spack module then you must use Spackcommands available after loading the spack module, e.g.:

[auser@cirrus-login0 ~]$ module avail boost

------------ /lustre/sw/spack/share/spack/modules/linux-centos7-x86_64 ------------boost-1.63.0-intel-17.0.2-fl25xqn boost-1.64.0-gcc-6.2.0-pftxg46

[auser@cirrus-login0 ~]$ module load spack

[auser@cirrus-login0 ~]$ spack find -dl boost==> 2 installed packages.-- linux-centos7-x86_64 / [email protected] -----------------------------pftxg46 [email protected] ^[email protected] ^[email protected] ^[email protected]

-- linux-centos7-x86_64 / [email protected] --------------------------fl25xqn [email protected] ^[email protected] ^[email protected]

5.1. Using the modules environment 19

http://spack.readthedocs.io


This shows their are two boost modules installed (one for the Intel compilers and one for the GCC compilers), theyboth depend on zlib 1.0.6 and bzip2 1.2.10 and the GCC version also depends on MPI 2.14 (SGI MPT 2.14). Thepaths for these dependencies are hardocoded into the boost RPATH.

5.2 Available Compiler Suites

Note: As Cirrus uses dynamic linking by default you will generally also need to load any modules you used to compileyour code in your job submission script when you run your code.

5.2.1 Intel Compiler Suite

The Intel compiler suite is accessed by loading the intel-compilers-* module. For example:


Once you have loaded the module, the compilers are available as:

• ifort - Fortran

• icc - C

• icpc - C++

C++ with Intel Compilers

Intel compilers rely on GCC C++ headers and libraries to support most recent C++ features. If you are using Intelcompilers to compile C++ on Cirrus you should also load the gcc/6.2.0 module to have access to the correct C++ files:

:: module load gcc/6.2.0

Note: You will also need to load this module in your job submission scripts when running code compiled in this way.

5.2.2 GCC Compiler Suite

The GCC compiler suite is accessed by loading the gcc module. For example:

module load gcc

Once you have loaded the module, the compilers are available as:

• gfortran - Fortran

• gcc - C

• g++ - C++

5.3 Compiling MPI codes

There are two MPI libraries currently available on Cirrus:

• SGI Message Passing Toolkit (MPT)

• Intel MPI



The compilation and run commands are different depending on which of these libraries you choose. Most of theapplications we have compiled on Cirrus have made use of the SGI MPT library and we only use Intel MPI if SGIMPT cannot be used for some reason. If you can use either library it is worthwhile running a few tests to discover ifeither provides a performance advantage for your application.

The following sections discuss each of the MPI library options in turn.

You should also consult the chapter on running jobs through the batch system for examples of how to run jobs compiledagainst the different MPI libraries.

Remember: by default, all compilers produce dynamic executables on Cirrus. This means that you must load thesame modules at runtime (usually in your job submission script) as you have loaded at compile time.

5.3.1 Using SGI MPT

To compile MPI code with SGI MPT, using any compiler, you must first load the “mpt” module.

module load mpt

This makes the compiler wrapper scripts mpicc, mpicxx and mpif90 available to you.

What you do next depends on which compiler (Intel or GCC) you wish to use to compile your code.

Note: We recommend that you use the Intel compiler wherever possible to compile MPI applications as this is themethod officially supported and tested by SGI.

Note: You can always check which compiler the MPI compiler wrapper scripts are using with, for example, mpicc-v or mpif90 -v.

Using Intel Compilers and SGI MPT

Once you have loaded the MPT module you should next load the appropriate intel-compilers module (e.g.intel-compilers-17):


Remember, if you are compiling C++ code, then you will also need to load the gcc/6.2.0 module for the C++ 11headers to be available.

Compilers are then available as

• mpif90 - Fortran with MPI

• mpicc - C with MPI

• mpicxx - C++ with MPI

Note mpicc uses gcc by default:

When compiling C applications you must also specify that mpicc should use the icc compiler with, for example,mpicc -cc=icc. (This is not required for Fortran as the mpif90 compiler automatically uses ifort.) If in doubtuse mpicc -cc=icc -v to see which compiler is actually being called.

Alternatively, you can set the environment variable MPICC_CC=icc to ensure the correct base compiler is used:

export MPICC_CC=icc

5.3. Compiling MPI codes 21


Note mpicxx uses g++ by default:

When compiling C++ applications you must also specify that mpicxx should use the icpc compiler with, for exam-ple, mpicxx -cxx=icpc. (This is not required for Fortran as the mpif90 compiler automatically uses ifort.)If in doubt use mpicxx -cxx=icpc -v to see which compiler is actually being called.

Alternatively, you can set the environment variable MPICXX_CXX=icpc to ensure the correct base compiler is used:

export MPICXX_CXX=icpc

Using GCC Compilers and SGI MPT

Once you have loaded the MPT module you should next load the gcc module:

module load gcc

Compilers are then available as




Note: SGI MPT does not support the syntax use mpi in Fortran applications with the GCC compiler gfortran.You should use the older include "mpif.h" syntax when using GCC compilers with mpif90.

5.3.2 Using Intel MPI

To compile MPI code with Intel MPI, using any compiler, you must first load the “intel-mpi-17” module:

module load intel-mpi-17

This makes the compiler wrapper scripts available to you. The name of the wrapper script depends on the compilersuite you are using. In summary:

Language Intel GCCFortran mpiifort mpif90C++ mpiicpc mpicxxC mpiicc mpicc

Further details on using the different compiler suites with Intel MPI are available in the following sections.

Using Intel Compilers and Intel MPI

Once you have loaded the intel-mpi-17 module you should next load the appropriate intel-compilersmodule (e.g. intel-compilers-17):


Remember, if you are compiling C++ code, then you will also need to load the gcc/6.2.0 module for the C++ 11headers to be available.

MPI compilers are then available as

• mpiifort - Fortran with MPI



• mpiicc - C with MPI

• mpiicpc - C++ with MPI

Note: Intel compilers with Intel MPI use non-standard compiler wrapper script names. If you use the standard namesyou will end up using the GCC compilers.

Using GCC Compilers and Intel MPI

Once you have loaded the intel-mpi-17 module you should next load the gcc module.

module load gcc

MPI compilers are then available as




5.4 Compiler Information and Options

The manual pages for the different compiler suites are available:

GCC Fortran man gfortran , C/C++ man gcc

Intel Fortran man ifort , C/C++ man icc

5.4.1 Useful compiler options

Whilst difference codes will benefit from compiler optimisations in different ways, for reasonable performance onCirrus, at least initially, we suggest the following compiler options:

Intel -O2

GNU -O2 -ftree-vectorize -funroll-loops -ffast-math

When you have a application that you are happy is working correctly and has reasonable performance you may wishto investigate some more aggressive compiler optimisations. Below is a list of some further optimisations that you cantry on your application (Note: these optimisations may result in incorrect output for programs that depend on an exactimplementation of IEEE or ISO rules/specifications for math functions):

Intel -fast

GNU -Ofast -funroll-loops

Vectorisation, which is one of the important compiler optimisations for Cirrus, is enabled by default as follows:

Intel At -O2 and above

GNU At -O3 and above or when using -ftree-vectorize

To promote integer and real variables from four to eight byte precision for FORTRAN codes the following compilerflags can be used:

Intel -real-size 64 -integer-size 64 -xAVX (Sometimes the Intel compiler incorrectly generatesAVX2 instructions if the -real-size 64 or -r8 options are set. Using the -xAVX option prevents this.)

GNU -freal-4-real-8 -finteger-4-integer-8

5.4. Compiler Information and Options 23


5.5 Using static linking/libraries

By default, executables on Cirrus are built using shared/dynamic libraries (that is, libraries which are loaded at run-time as and when needed by the application) when using the wrapper scripts.

An application compiled this way to use shared/dynamic libraries will use the default version of the library installedon the system (just like any other Linux executable), even if the system modules were set differently at compile time.This means that the application may potentially be using slightly different object code each time the application runsas the defaults may change. This is usually the desired behaviour for many applications as any fixes or improvementsto the default linked libraries are used without having to recompile the application, however some users may feel thisis not the desired behaviour for their applications.

Alternatively, applications can be compiled to use static libraries (i.e. all of the object code of referenced librariesare contained in the executable file). This has the advantage that once an executable is created, whenever it is runin the future, it will always use the same object code (within the limit of changing runtime environemnt). However,executables compiled with static libraries have the potential disadvantage that when multiple instances are runningsimultaneously multiple copies of the libraries used are held in memory. This can lead to large amounts of memorybeing used to hold the executable and not application data.

To create an application that uses static libraries you must pass an extra flag during compilation, -Bstatic.

Use the UNIX command ldd exe_file to check whether you are using an executable that depends on sharedlibraries. This utility will also report the shared libraries this executable will use if it has been dynamically linked.


CHAPTER 6

Running Jobs on Cirrus

The Cirrus facility uses PBSPro to schedule jobs.

Writing a submission script is typically the most convenient way to submit your job to the job submission system.Example submission scripts (with explanations) for the most common job types are provided below.

Interactive jobs are also available and can be particularly useful for developing and debugging applications. Moredetails are available below.

Note: There are a number of different queues on Cirrus. If you do not specify a queue your job will be submittedto the default workq which has limits on the maximum job size and total amount of resource that can be used. Seebelow for more information on the different queues and their limits.

If you have any questions on how to run jobs on Cirrus do not hesitate to contact the EPCC Helpdesk.

6.1 Using PBS Pro

You typically interact with PBS by (1) specifying PBS directives in job submission scripts (see examples below) and(2) issuing PBS commands from the login nodes.

There are three key commands used to interact with the PBS on the command line:

• qsub

• qstat

• qdel

Check the PBS man page for more advanced commands:

man pbs

6.1.1 The qsub command

The qsub command submits a job to PBS:

25


qsub job_script.pbs

This will submit your job script “job_script.pbs” to the job-queues. See the sections below for details on how to writejob scripts.

Note: There are a number of different queues on Cirrus. If you do not specify a queue your job will be submittedto the default workq which has limits on the maximum job size and total amount of resource that can be used. Seebelow for more information on the different queues and their limits.

6.1.2 The qstat command

Use the command qstat to view the job queue. For example:

qstat

will list all jobs on Cirrus.

You can view just your jobs by using:

qstat -u $USER

The -a option to qstat provides the output in a more useful format.

To see more information about a queued job, use:

qstat -f $JOBID

This option may be useful when your job fails to enter a running state. The output contains a PBS comment fieldwhich may explain why the job failed to run.

6.1.3 The qdel command

Use this command to delete a job from Cirrus’s job queue. For example:

qdel $JOBID

will remove the job with ID $JOBID from the queue.

6.2 Queue Limits

Queues on Cirrus are designed to enable users to use the system flexibly while retaining fair access for all.

There are two queues available to general users on Cirrus:

• workq (default): This is the default queue that user jobs are submitted to if the -q option to qsub is notspecified. Jobs in this queue can have a maximum walltime of 96 hours (4 days) and a maximum job size of2520 cores (70 nodes). Each project can use a maximum of 2520 cores (70 nodes) summed across all theirrunning jobs at any one time.

• large: Specified by using -q large at submission time. There is no upper limit on job size in this queue butthere is a minimum job size of 2521 cores (71 nodes), a maximum walltime of 48 hours (2 days), each user canhave a maximum of 1 job running at any one time, and a maximum of 4 jobs in the queue (including a runningjob).

26 Chapter 6. Running Jobs on Cirrus


If you try to submit a job that asks for more than the maximum allowed wall time or cores you will see an error similarto:

[user@cirrus-login0 ~]$ qsub submit.pbsqsub: Job violates queue and/or server resource limits

6.3 Output from PBS jobs

PBS produces standard output and standard error for each batch job can be found in files <jobname>.o<Job ID>and <jobname>.e<Job ID> respectively. These files appear in the job’s working directory once your job hascompleted or its maximum allocated time to run (i.e. wall time, see later sections) has ran out.

6.4 Running Parallel Jobs

This section describes how to write job submission scripts specifically for different kinds of parallel jobs on Cirrus.

All parallel job submission scripts require (as a minimum) you to specify three things:

• The number of nodes and cores per node you require via the -l select=[Nodes]:ncpus=72 option.Note ncpus should always be 72, regardless of how many cores you intend to employ. This simply indicatesthat you want to reserve all cores on a node. Each node has 36 physical cores (2x 18-core sockets) and hyper-threads are enabled (2 per core) giving a maximum of 72 cores per node (most users will actually only use amaximum of 36 cores per node for best performance). For example, to select 4 nodes (144 physical cores intotal) you would use -l select=4:ncpus=72.

• The maximum length of time (i.e. walltime) you want the job to run for via the -l walltime=[hh:mm:ss]option. To ensure the minimum wait time for your job, you should specify a walltime as short as possible foryour job (i.e. if your job is going to run for 3 hours, do not specify 12 hours). On average, the longer thewalltime you specify, the longer you will queue for.

• The project code that you want to charge the job to via the -A [project code] option

In addition to these mandatory specifications, there are many other options you can provide to PBS. The followingoptions may be useful:

• By default compute nodes are shared, meaning other jobs may be placed alongside yours if your resource request(with -l select) leaves some cores free. To guarantee exclusive node usage, use the option -l place=excl.

• The name for your job is set using -N My_job. In the examples below the name will be “My_job”, but youcan replace “My_job” with any name you want. The name will be used in various places. In particular it will beused in the queue listing and to generate the name of your output and/or error file(s). Note there is a limit on thesize of the name.

• -q large will specify that you want to submit your job to the large queue for running larger jobs than arepermitted in the standard queue.

6.4.1 Exclusive Node Access

By default on Cirrus, jobs are scheduled to nodes in a non-exclusive way. This means that, by default, you will likelyend up sharing a node with another user. This can lead to variable and/or poor performance as you will be potentiallybe competing with other users for resources such as CPU and memory.

If you are running parallel jobs on Cirrus we strongly recommend that you specify exclusive placement to ensurethat you get the best performance out of the compute nodes. The only case where you may not want to use exclusive

6.3. Output from PBS jobs 27


placement for parallel jobs is when you are using a very small number of cores (e.g. less than half a node, 18 cores).Even then, you may find that the benefits of exclusive placement outweigh the addition costs incurred (as you arecharged for all the cores on a node in exclusive mode).

To make sure your josb have exclusive node access you should add the following PBS option to your jobs:

#PBS -l place=excl

All of our example parallel job submission scripts below specify this option.

Of course, if you are running a serial job then you should not generally specify this option as it would result in youreserving (and being charged for) a full 36 core node when you are only using a single core.

6.5 Running MPI parallel jobs

When you running parallel jobs requiring MPI you will use an MPI launch command to start your executable inparallel. The name and options for this MPI launch command depend on which MPI library you are using: SGI MPT(Message Passing Toolkit) or Intel MPI. We give details below of the commands used in each case and our examplejob submission scripts have examples for both libraries.

Note: If you are using a centrally-installed MPI software package you will need to know which MPI library was usedto compile it so you can use the correct MPI launch command. You can find this information using the moduleshow command. For example:

[auser@cirrus-login0 ~]$ module show vasp-------------------------------------------------------------------/lustre/sw/modulefiles/vasp/5.4.4-intel17-mpt214:

conflict vaspmodule load mptmodule load intel-compilers-17module load intel-cmkl-17module load gcc/6.2.0prepend-path PATH /lustre/home/y07/vasp5/5.4.4-intel17-mpt214/binsetenv VASP5 /lustre/home/y07/vasp5/5.4.4-intel17-mpt214setenv VASP5_VDW_KERNEL /lustre/home/y07/vasp5/5.4.4-intel17-mpt214/→˓vdw_kernal/vdw_kernel.bindat-------------------------------------------------------------------

This shows that VASP was compiled with SGI MPT (from the module load mpt in the output from the command.If a package was compiled with Intel MPI there would be module load intel-mpi-17 in the output instead.

6.5.1 SGI MPT (Message Passing Toolkit)

SGI MPT is accessed at both compile and runtime by loading the mpt module:

module load mpt

SGI MPT: parallel launcher mpiexec_mpt

The SGI MPT parallel launcher on Cirrus is mpiexec_mpt.

Note: this parallel job launcher is only available once you have loaded the mpt module.

A sample MPI launch line using mpiexec_mpt looks like:



mpiexec_mpt -n 72 -ppn 36 ./my_mpi_executable.x arg1 arg2

This will start the parallel executable “my_mpi_executable.x” with arguments “arg1” and “arg2”. The job will bestarted using 72 MPI processes, with 36 MPI processes are placed on each compute node (this would use all thephysical cores on each node). This would require 2 nodes to be requested in the PBS options.

The most important mpiexec_mpt flags are:

-n [total number of MPI processes] Specifies the total number of distributed memory par-allel processes (not including shared-memory threads). For jobs that use all physical cores this willusually be a multiple of 36. The default on Cirrus is 1.

-ppn [parallel processes per node] Specifies the number of distributed memory parallelprocesses per node. There is a choice of 1-36 for physical cores on Cirrus compute nodes (1-72 ifyou are using Hyper-Threading) If you are running with exclusive node usage, the most economicchoice is always to run with “fully-packed” nodes on all physical cores if possible, i.e. -ppn 36. Running “unpacked” or “underpopulated” (i.e. not using all the physical cores on a node) isuseful if you need large amounts of memory per parallel process or you are using more than oneshared-memory thread per parallel process.

Note: mpiexec_mpt only works from within a PBS job submission script.

Please use man mpiexec_mpt query further options. (This is only available once you have loaded the mptmodule.)

SGI MPT: interactive MPI using mpirun

If you want to run short interactive parallel applications (e.g. for debugging) then you can run SGI MPT compiledMPI applications on the login nodes using the mpirun command.

For instance, to run a simple, short 4-way MPI job on the login node, issue the following command (once you haveloaded the appropriate modules):

mpirun -n 4 ./hello_mpi.x

Note: you should not run long, compute- or memory-intensive jobs on the login nodes. Any such processes are liableto termination by the system with no warning.

SGI MPT: running hybrid MPI/OpenMP applications

If you are running hybrid MPI/OpenMP code using SGI MPT you will also often make use of the omplace tool inyour job launcher line. This tool takes the number of threads as the option -nt:

-nt [threads per parallel process] Specifies the number of cores for each parallel pro-cess to use for shared-memory threading. (This is in addition to the OMP_NUM_THREADS envi-ronment variable if you are using OpenMP for your shared memory programming.) The default onCirrus is 1.

Please use man mpiexec_mpt and man omplace to query further options. (Again, these are only available onceyou have loaded the mpt module.)

6.5.2 Intel MPI

Intel MPI is accessed at runtime by loading the intel-mpi-17.

module load intel-mpi-17

6.5. Running MPI parallel jobs 29


Intel MPI: parallel job launcher mpirun

The Intel MPI parallel job launcher on Cirrus is mpirun.

Note: this parallel job launcher is only available once you have loaded the intel-mpi-17 module.

A sample MPI launch line using mpirun looks like:

mpirun -n 72 -ppn 36 ./my_mpi_executable.x arg1 arg2

This will start the parallel executable “my_mpi_executable.x” with arguments “arg1” and “arg2”. The job will bestarted using 72 MPI processes, with 36 MPI processes are placed on each compute node (this would use all thephysical cores on each node). This would require 2 nodes to be requested in the PBS options.

The most important mpirun flags are:

-n [total number of MPI processes] Specifies the total number of distributed memory par-allel processes (not including shared-memory threads). For jobs that use all physical cores this willusually be a multiple of 36. The default on Cirrus is 1.

-ppn [parallel processes per node] Specifies the number of distributed memory parallelprocesses per node. There is a choice of 1-36 for physical cores on Cirrus compute nodes (1-72 ifyou are using Hyper-Threading) If you are running with exclusive node usage, the most economicchoice is always to run with “fully-packed” nodes on all physical cores if possible, i.e. -ppn 36. Running “unpacked” or “underpopulated” (i.e. not using all the physical cores on a node) isuseful if you need large amounts of memory per parallel process or you are using more than oneshared-memory thread per parallel process.

Documentation on using Intel MPI (including mpirun) can be found online at:

• Intel MPI Documentation

Intel MPI: running hybrid MPI/OpenMP applications

If you are running hybrid MPI/OpenMP code using Intel MPI you need to set the I_MPI_PIN_DOMAIN environmentvariable to omp so that MPI tasks are pinned with enough space for OpenMP threads.

For example, in your job submission script you would use:

export I_MPI_PIN_DOMAIN=omp

You can then also use the KMP_AFFINITY enviroment variable to control placement of OpenMP threads. For moreinformation, see:

• Intel OpenMP Thread Affinity Control

Intel MPI: MPI-IO setup

If you wish to use MPI-IO with Intel MPI you must set a couple of additional environment variables in your jobsubmission script to tell the MPI library to use the Lustre file system interface. Specifically, you should add the lines:

export I_MPI_EXTRA_FILESYSTEM=onexport I_MPI_EXTRA_FILESYSTEM_LIST=lustre

after you have loaded the intel-mpi-17 module.

If you fail to set these environment variables you may see errors such as:


https://software.intel.com/en-us/articles/intel-mpi-library-documentation

https://software.intel.com/en-us/articles/openmp-thread-affinity-control


This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. Generic MPICHMessage: File locking failed inADIOI_Set_lock(fd 0,cmd F_SETLKW/7,type F_WRLCK/1,whence 0) with return valueFFFFFFFF and errno 26.- If the file system is NFS, you need to use NFS version 3, ensure that the lockddaemon is running on all the machines, and mount the directory with the 'noac'option (no attribute caching).

- If the file system is LUSTRE, ensure that the directory is mounted with the 'flock'option.

ADIOI_Set_lock:: Function not implementedADIOI_Set_lock:offset 0, length 10application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3

6.6 Example parallel MPI job submission scripts

A subset of example job submssion scripts are included in full below. The full set are available via the following links:

• SGI MPT MPI Job: example_mpi_sgimpt.bash

• Intel MPI Job: example_mpi_impi.bash

• SGI MPT Hybrid MPI/OpenMP Job: example_hybrid_sgimpt.bash

• Intel MPI Hybrid MPI/OpenMP Job: example_hybrid_impi.bash

6.6.1 Example: SGI MPT job submission script for MPI parallel job

A simple MPI job submission script to submit a job using 4 compute nodes (maximum of 144 physical cores) for 20minutes would look like:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N Example_MPI_Job# Select 4 full nodes#PBS -l select=4:ncpus=72# Parallel jobs should always specify exclusive node access#PBS -l place=excl#PBS -l walltime=00:20:00

# Replace [budget code] below with your project code (e.g. t01)#PBS -A [budget code]

# Change to the directory that the job was submitted fromcd $PBS_O_WORKDIR

# Load any required modulesmodule load mptmodule load intel-compilers-17

# Set the number of threads to 1# This prevents any threaded system libraries from automatically# using threading.export OMP_NUM_THREADS=1

6.6. Example parallel MPI job submission scripts 31


# Launch the parallel job# Using 144 MPI processes and 36 MPI processes per nodempiexec_mpt -n 144 -ppn 36 ./my_mpi_executable.x arg1 arg2 > my_stdout.txt 2> my_→˓stderr.txt

This will run your executable “my_mpi_executable.x” in parallel on 144 MPI processes using 2 nodes (36 cores pernode, i.e. not using hyper-threading). PBS will allocate 4 nodes to your job and mpirun_mpt will place 36 MPIprocesses on each node (one per physical core).

See above for a more detailed discussion of the different PBS options

6.6.2 Example: SGI MPT job submission script for MPI+OpenMP (mixed mode) par-allel job

Mixed mode codes that use both MPI (or another distributed memory parallel model) and OpenMP should take care toensure that the shared memory portion of the process/thread placement does not span more than one node. This meansthat the number of shared memory threads should be a factor of 18.

In the example below, we are using 4 nodes for 6 hours. There are 4 MPI processes in total and 18 OpenMP threadsper MPI process. Note the use of the omplace command to specify the number of threads.

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N Example_MixedMode_Job# Select 4 full nodes#PBS -l select=4:ncpus=72# Parallel jobs should always specify exclusive node access#PBS -l place=excl#PBS -l walltime=6:0:0



# Load any required modulesmodule load mptmodule load intel-compilers-17

# Set the number of threads to 18# There are 18 OpenMP threads per MPI processexport OMP_NUM_THREADS=18

# Launch the parallel job# Using 8 MPI processes# 2 MPI processes per node# 18 OpenMP threads per MPI processmpiexec_mpt -n 8 -ppn 2 omplace -nt 18 ./my_mixed_executable.x arg1 arg2 > my_stdout.→˓txt 2> my_stderr.txt



6.6.3 Example: job submission script for parallel non-MPI based jobs

If you want to run on multiple nodes, where each node is running a self-contained job, not using MPI (e.g.) forprocessing data or a parameter sweep, you can use the SGI MPT mpiexec_mpt launcher to control job placement.

In the example script below, work.bash is a bash script which runs a threaded executable with a command-lineinput and perf.bash is a bash script which copies data from the CPU performance counters to an output file. Asboth handle the threading themselves, it is sufficient to allocate 1 MPI rank. Using the ampersand & allows both toexecute simultaneously. Both work.bash and perf.bash run on 4 nodes.

#!/bin/bash --login# PBS job options (name, compute nodes, job time)#PBS -N Example_MixedMode_Job# Select 4 full nodes#PBS -l select=4:ncpus=72# Parallel jobs should always specify exclusive node access#PBS -l place=excl#PBS -l walltime=6:0:0



# Load any required modulesmodule load mpt

# Set this variable to inform mpiexec_mpt these are not MPI jobsexport MPI_SHEPHERD=true

# Execute work and perf scripts on nodes simultaneously.mpiexec_mpt -n 4 -ppn 1 work.bash &mpiexec_mpt -n 4 -ppn 1 perf.bash &wait

Note: the wait command is required to stop the PBS job finishing before the scripts finish. If you find odd behaviour,especially with respect to the values of bash variables, double check you have set MPI_SHEPHERD=true

6.7 Serial Jobs

Serial jobs are setup in a similar way to parallel jobs on Cirrus. The only changes are:

1. You should request a single core with select=1:ncpus=1

2. You will not need to use a parallel job launcher to run your executable

3. You will generally not specify exclusive node access

A simple serial script to compress a file would be:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N Example_Serial_Job#PBS -l select=1:ncpus=1#PBS -l walltime=0:20:0

6.7. Serial Jobs 33




# Load any required modulesmodule load intel-compilers-16

# Set the number of threads to 1 to ensure serialexport OMP_NUM_THREADS=1

# Run the serial executablegzip my_big_file.dat

6.8 Job arrays

The PBSPro job scheduling system offers the job array concept, for running collections of almost-identical jobs, forexample running the same program several times with different arguments or input data.

Each job in a job array is called a subjob. The subjobs of a job array can be submitted and queried as a unit, making iteasier and cleaner to handle the full set, compared to individual jobs.

All subjobs in a job array are started by running the same job script. The job script also contains information on thenumber of jobs to be started, and PBSPro provides a subjob index which can be passed to the individual subjobs orused to select the input data per subjob.

6.8.1 Job script for a job array

As an example, to start 56 subjobs, with the subjob index as the only argument, and 4 hours maximum runtime persubjob, save the following content into the file job_script.pbs:

#!/bin/bash --login#PBS -l select=1:ncpus=1#PBS -l walltime=04:00:00#PBS -J 1-56#PBS -q workq#PBS -V

cd ${PBS_O_WORKDIR}

/path/to/exe $PBS_ARRAY_INDEX

Another example of a job script for submitting a job array is given here.

6.8.2 Starting a job array

When starting a job array, most options can be included in the job file, but the project code for the resource billing hasto be specified on the command line:

qsub -A [project code] job_script.pbs


../software-packages/flacs.html#submitting-many-flacs-jobs-as-a-job-array


6.8.3 Querying a job array

In the normal PBSPro job status, a job array will be shown as a single line:

> qstatJob id Name User Time Use S Queue---------------- -------------- ------ -------- - -----112452[].indy2-lo dispsim user1 0 B workq

To monitor the subjobs of the job 112452, use

> qstat -t 1235[]Job id Name User Time Use S Queue---------------- ---------------- ---------------- -------- - -----112452[].indy2-lo dispsim user1 0 B flacs112452[1].indy2-l dispsim user1 02:45:37 R flacs112452[2].indy2-l dispsim user1 02:45:56 R flacs112452[3].indy2-l dispsim user1 02:45:33 R flacs112452[4].indy2-l dispsim user1 02:45:45 R flacs112452[5].indy2-l dispsim user1 02:45:26 R flacs...

6.9 Interactive Jobs

When you are developing or debugging code you often want to run many short jobs with a small amount of editingthe code between runs. This can be achieved by using the login nodes to run MPI but you may want to test on thecompute nodes (e.g. you may want to test running on multiple nodes across the high performance interconnect). Oneof the best ways to achieve this on Cirrus is to use interactive jobs.

An interactive job allows you to issue mpirun_mpt commands directly from the command line without using a jobsubmission script, and to see the output from your program directly in the terminal.

To submit a request for an interactive job reserving 8 nodes (288 physical cores) for 1 hour you would issue thefollowing qsub command from the command line:

qsub -IVl select=8:ncpus=72,walltime=1:0:0,place=excl -A [project code]

When you submit this job your terminal will display something like:

qsub: waiting for job 19366.indy2-login0 to start

It may take some time for your interactive job to start. Once it runs you will enter a standard interactive terminalsession. Whilst the interactive session lasts you will be able to run parallel jobs on the compute nodes by issuing thempirun_mpt command directly at your command prompt (remember you will need to load the mpt module and anycompiler modules before running) using the same syntax as you would inside a job script. The maximum number ofcores you can use is limited by the value of select you specify when you submit a request for the interactive job.

If you know you will be doing a lot of intensive debugging you may find it useful to request an interactive sessionlasting the expected length of your working session, say a full day.

Your session will end when you hit the requested walltime. If you wish to finish before this you should use the exitcommand.

6.9. Interactive Jobs 35


6.10 Reservations

Resource reservations are available on Cirrus. These allow users to reserve a number of nodes for a specified lengthof time starting at a particular time on the system.

Examples of the reasons for using reservations could be:

• An exceptional job requires longer than 96 hours runtime.

• You require a job/jobs to run at a particular time e.g. for a demonstration or course.

Warning: For multi-node jobs we strongly recommend requesting a reservation two nodes larger than the sizeyou want to stop the reservation failing if a node crashes. This is particularly important if the reservation involveslong jobs or those of a time critical nature.

Note: Reservations will be charged at 1.5 times the usual rate and you will be charged the full rate for the entirereservation whether or not you use the resources reserved for the full time. In addition, you will not be refunded theresources if you fail to use them due to a job crash unless this crash is due to a system failure. To allow people to createmulti-node reservations, we will charge at number of nodes - 2 for resevations (with a minimum of 2 nodes charged).

6.10.1 Requesting reservations

You request a reservation on Cirrus using PBS from the command line. Before requesting the reservation, you willneed the following information:

• The start time for the resevation

• The duration of the reservation

• The number of cores (or nodes for multi-node, node-exclusive jobs)

• The project ID you wish to charge the reservation to

You use the pbs_rsub command to create a reservation. This command has a similar syntax to the qsub commandfor requesting resources but takes the additional parameters -R (to specify the reservaiton start time); -D (to specifythe reservation duration); and -G (to specify the project ID to charge the reservation to). For example, to create areservation for 3 hours at 10:30 (UK time) on Saturday 26 August 2017 for 4 full nodes (144 physical cores, 288hyperthreads) and charge to project “t01” you would use the command:

[auser@cirrus-login0 ~]$ pbs_rsub -R 1708261030 -D 3:0:0 -l select=6:ncpus=72,→˓place=excl -G +t01R122604.indy2-login0 UNCONFIRMED

The command will return a reservation ID (R122604 in the example above) and note that it is currentlyUNCONFIRMED. PBSPro will change the status to CONFIRMED once it has checked that it is possible to schedulethe reservation. Note that we requested 6 nodes rather than the required 4 to reduce the risk of hardware failureaffecting the reservation.

Note: Only the user that requested this reservation will be able to submit jobs to it. To create a reservation that isavailable to all users in a particular project, see the instructions below.

There are many other options to the pbs_rsub command. Please check the man page for a full description.



6.10.2 Checking the status of your reservation

You can cheack the status of your reservation request with the pbs_rstat command:

[auser@cirrus-login0 ~]$ pbs_rstatResv ID Queue User State Start / Duration / End---------------------------------------------------------------------R122604.in R122605 auser@ CO Sat 10:30 / 10800 / Sat 13:30

and, as you can see, the status of the requested reservation is now CO (CONFIRMED).

6.10.3 Submitting jobs to a reservation

You submit jobs to reservations in the same way as you do for all other jobs using the qsub command. The onlyadditional information required is to specify the reservation ID to the -q option. For example, to submit to thereservation created above you would use:

qsub -q R122604 ...usual qsub options/job script name...

Note: You can submit jobs to the reservation ahead of the start time and the job will start as soon as the reservationbegins.

6.10.4 Reservations for all project users

By default, a reservation will only be available to the user who requested it. If you wish to create a reservation that isusable by all members of your project you need to modify the user permissions using the -U option.

For example, to create a reservation for 192 hours, starting at 16:15 (UK time) on Monday 18 September 2017 for 64nodes accessible by all users in the t01 project you would use:

[auser@cirrus-login0 ~]$ pbs_rsub -R 1709181615 -D 192:0:0 -l select=66:ncpus=72,→˓place=excl -G +t01 -U +R122605.indy2-login0 UNCONFIRMED

Here, the -G +t01 option charges the reservation to the t01 project and restricts access to users in the t01 project;the -U + option allows all users (in the t01 project) access to the reservation. Note that, as above, we created thereservation with 66 nodes instead of the required 64 to reduce the risk of hardware failures affecting the reservation.

Note: You can restrict access to specific users within a project, see the pbs_rsub man page for more information onhow to do this.

6.10.5 Deleting a reservation

Use the pbs_rdel command to delete a reservation:

[auser@cirrus-login0 ~]$ pbs_rdel R122605

6.10. Reservations 37

CHAPTER 7

Singularity Containers

This page was originally based on the documentation at the University of Sheffield HPC service:.

Designed around the notion of mobility of compute and reproducible science, Singularity enables users to have fullcontrol of their operating system environment. This means that a non-privileged user can “swap out” the Linuxoperating system and environment on the host for a Linux OS and environment that they control. So if the host systemis running CentOS Linux but your application runs in Ubuntu Linux with a particular software stack; you can createan Ubuntu image, install your software into that image, copy the image to another host (e.g. Cirrus), and run yourapplication on that host in it’s native Ubuntu environment.

Singularity also allows you to leverage the resources of whatever host you are on. This includes high-speed intercon-nects (i.e. Infinband on Cirrus), file systems (i.e. /lustre on Cirrus) and potentially other resources (e.g. the licensedIntel compilers on Cirrus).

Note: Singularity only supports Linux containers. You cannot create images that use Windows or macOS (this is arestriction of the containerisation model rather than Singularity).

7.1 Useful Links

• Singularity website

• Singularity user documentation

7.2 About Singularity Containers (Images)

Similar to Docker, a Singularity container (or, more commonly, image) is a self-contained software stack. As Sin-gularity does not require a root-level daemon to run its images (as is required by Docker) it is suitable for use on amulti-user HPC system such as Cirrus. Within the container/image, you have exactly the same permissions as you doin a standard login session on the system.

In practice, this means that an image created on your local machine with all your research software installed for localdevelopment will also run on Cirrus.

39

http://docs.hpc.shef.ac.uk/en/latest/sharc/software/apps/singularity.html

http://singularity.lbl.gov/

http://singularity.lbl.gov/user-guide


Pre-built images (such as those on DockerHub or SingularityHub) can simply be downloaded and used on Cirrus (oranywhere else Singularity is installed); see Using Singularity Images on Cirrus).

Creating and modifying images requires root permission and so must be done on a system where you have such access(in practice, this is usually within a virtual machine on your laptop/workstation); see Creating Your Own SingularityImages.

7.3 Using Singularity Images on Cirrus

Singularity images can be used on Cirrus in a number of ways, including:

• Interactively on the login nodes

• Interactively on compute nodes

• As serial processes within a non-interactive batch script

• As parallel processes within a non-interactive batch script (not yet documented)

We provide information on each of these scenarios (apart from the parallel use where we are still preparing the docu-mentation) below. First, we describe briefly how to get exisitng images onto Cirrus so you can use them.

7.3.1 Getting existing images onto Cirrus

Singularity images are simply files so, if you already have an image file, you can use scp to copy the file to Cirrus asyou would with any other file.

If you wish to get a file from one of the container image repositories then Singularity allows you to do this from Cirrusitself.

This functionality requires tools that are not part of the standard OS on Cirrus so we have provided a Singularityimage that allows you to build images from remote repositories (i.e. you use a Singularity image to build Singularityimages!).

For example, to retrieve an image from DockerHub on Cirrus we fist need to enter an interactive session in the imagewe provide for building Singularity images:

[user@cirrus-login0 ~]$ module load singularity[user@cirrus-login0 ~]$ singularity exec $CIRRUS_SIMG/cirrus-sbuild.simg /bin/bash --→˓loginSingularity>

This invokes a login bash shell within the $CIRRUS_SIMG/cirrus-sbuild.simg image as indicated by ourprompt change. (We need a login shell to allow module commands to work within the image.)

Now we are in the image we can load the singularity module (to get access to the Singularity commands) and pull animage from DockerHub:

Singularity> module load singularitySingularity> singularity build lolcow.simg docker://godlovedc/lolcowDocker image path: index.docker.io/godlovedc/lolcow:latestCache folder set to /lustre/home/t01/user/.singularity/dockerImporting: base Singularity environmentImporting: /lustre/home/t01/user/.singularity/docker/→˓sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118.tar.gzImporting: /lustre/home/t01/user/.singularity/docker/→˓sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a.tar.gzImporting: /lustre/home/t01/user/.singularity/docker/→˓sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2.tar.gz

40 Chapter 7. Singularity Containers

http://hub.docker.com

https://singularity-hub.org/


Importing: /lustre/home/t01/user/.singularity/docker/→˓sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e.tar.gzImporting: /lustre/home/t01/user/.singularity/docker/→˓sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9.tar.gzImporting: /lustre/home/t01/user/.singularity/docker/→˓sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945.tar.gzImporting: /lustre/home/t01/user/.singularity/metadata/→˓sha256:ab22e7ef68858b31e1716fa2eb0d3edec81ae69c6b235508d116a09fc7908cff.tar.gzWARNING: Building container as an unprivileged user. If you run this container as rootWARNING: it may be missing some functionality.Building Singularity image...Singularity container built: lolcow.simgCleaning up...

The first argument to singularity build (lolcow.simg) specifies a path and name for your container. The secondargument (docker://godlovedc/lolcow) gives the DockerHub URI from which to download the image.

Now we can exit the image and run our new image we have just built on the Cirrus login node:

[user@cirrus-login0 ~]$ singularity run lolcow.simg

This image contains a runscript that tells Singularity what to do if we run the image. We demonstrate different waysto use images below.

Similar syntax can be used for Singularity Hub. For more information see the Singularity documentation:

• Build a Container

7.3.2 Interactive use on the login nodes

Once you have an image file, using it on the login nodes in an interactive way is extremely simple: you use thesingularity shell command. Using the image we built in the example above:

[user@cirrus-login0 ~]$ module load singularity[user@cirrus-login0 ~]$ singularity shell lolcow.simgSingularity: Invoking an interactive shell within container...

Singularity lolcow.simg:~>

Within a Singularity image your home directory will be available. The directory with centrally-installed software (/lustre/sw) is also available in images by default. Note that the module command will not work in images unlessyou have installed he required software and configured the environment correctly; we describe how to do this below.

Once you have finished using your image, you return to the Cirrus login node command line with the exit command:

Singularity lolcow.simg:~> exitexit[user@cirrus-login0 ~]$

7.3.3 Interactive use on the compute nodes

The process for using an image interactively on the compute nodes is very similar to that for using them on the loginnodes. The only difference is that you have to submit an interactive serial job to get interactive access to the computenode first.

For example, to reserve a full node for you to work on interactively you would use:

7.3. Using Singularity Images on Cirrus 41

http://singularity.lbl.gov/docs-build-container


[user@cirrus-login0 ~]$ qsub -IVl select=1:ncpus=72,walltime=0:20:0,place=excl -A t01qsub: waiting for job 234192.indy2-login0 to start

...wait until job starts...

qsub: job 234192.indy2-login0 ready

[user@r1i2n13 ~]$

Note the prompt has changed to show you are on a compute node. Now you can use the image in the same way as onthe login node

[user@r1i2n13 ~]$ module load singularity[user@r1i2n13 ~]$ singularity shell lolcow.simgSingularity: Invoking an interactive shell within container...

Singularity lolcow.simg:~> exitexit[user@r1i2n13 ~]$ exit[user@cirrus-login0 ~]$

Note we used exit to leave the interactive image shell and then exit again to leave the interactive job on thecompute node.

7.3.4 Serial processes within a non-interactive batch script

You can also use Singularity images within a non-interactive batch script as you would any other command. If yourimage contains a runscript then you can use singularity run to execute the runscript in the job. You can alsouse singularity exec to execute arbitrary commands (or scripts) within the image.

An exmaple job submission script to run a serial job that executes the runscript within the lolcow.simg we builtabove on Cirrus would be:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N simg_test#PBS -l select=1:ncpus=1#PBS -l walltime=0:20:0



# Load any required modulesmodule load singularity

# Run the serial executablesingularity run $HOME/lolcow.simg

You submit this in the usual way and the output would be in the STDOUT/STDERR files in the usual way.



7.4 Creating Your Own Singularity Images

As we saw above, you can create Singularity images by importing from DockerHub or Singularity Hub on Cirrusitself. If you wish to create your own custom image then you must install Singularity on a system where you have root(or administrator) privileges - often your own laptop or workstation.

We provide links below to instructions on how to install Singularity locally and then cover what options you needto include in a Singularity recipe file to create images that can run on Cirrus and access the software developmentmodules. (This can be useful if you want to create a custom environment but still want to compile and link againstlibraries that you only have access to on Cirrus such as the Intel compilers, HPE MPI libraries, etc.)

7.4.1 Installing Singularity on Your Local Machine

You will need Singularity installed on your machine in order to locally run, create and modify images. How you installSingularity on your laptop/workstation depends on the operating system you are using.

If yout are using Windows or macOS, the simplest solution is to use Vagrant to give you an easy to use virtualenvironment with Linux and Singularity installed. The Singularity website has instructions on how to use this methodto install Singularity:

• Installing Singularity on macOS with Vagrant

• Installing Singularity on Windows with Vagrant

If you are using Linux then you can usually install Singularity directly, see:

• Installing Singularity on Linux

7.4.2 Singularity Recipes to Access modules on Cirrus

You may want your custom image to be able to access the modules environment on Cirrus so you can make use ofcustom software that you cannot access elsewhere. We demonstrate how to do this for a CentOS 7 image but the stepsare easily translated for other flavours of Linux.

For the Cirrus modules to be available in your Singularity container you need to ensure that theenvironment-modules package is installed in your image.

In addition, when you use the container you must invoke access as a login shell to have access to the module commands.

Here is an example recipe file to build a CentOS 7 image with access to TCL modules alread installed on Cirrus:

BootStrap: dockerFrom: centos:centos7

%postyum update -yyum install environment-modules -y

If we save this recipe to a file called cirrus-mods.def then we can use the following command to build this image(remember this command must be run on a system where you have root access, not Cirrus):

me@my-system:~> sudo singularity build cirrus-mods.simg cirrus-mods.def

The resulting image file (cirrus-mods.simg) can then be compied to Cirrus using scp.

When you use the image interactively on Cirrus you must start with a login shell, i.e.:

7.4. Creating Your Own Singularity Images 43

http://www.vagrantup.com

http://singularity.lbl.gov/install-mac

http://singularity.lbl.gov/install-windows

http://singularity.lbl.gov/install-linux


[user@cirrus-login0 ~]$ module load singularity[user@cirrus-login0 ~]$ singularity exec cirrus-mods.simg /bin/bash --loginSingularity> module avail intel-compilers

------------------------- /lustre/sw/modulefiles ---------------------intel-compilers-16/16.0.2.181intel-compilers-16/16.0.3.210(default)intel-compilers-17/17.0.2.174(default)


CHAPTER 8

Using Python

Python on Cirrus is provided by the Anaconda distribution. Both Python 2 and Python 3 versions of the distributionsare supported.

The central installation provides many of the most common packges used for scientific computation and data analysis.

If the packages you require are not included in the central Anaconda Python distribution, then the simplest way tomake these available is often to install your own version of Miniconda and add the packages you need. We provideinstructions on how to do this below. An alternative way to provide your own packages (and to make them availablemore generally to other people in your project and beyond) would be to use a Singularity container, see the SingularityContainers chapter of this User Guide for more information on this topic.

8.1 Accessing central Anaconda Python

Users have the standard system Python available by default. To setup your environment to use the Anaconda distribu-tions you should use:

module load anaconda/python2

for Python 2, or:

module load anaconda/python3

for Python 3.

You can verify the current version of Python with:

[user@cirrus-login0 ~]$ module load anaconda/python2[user@cirrus-login0 ~]$ python --versionPython 2.7.12 :: Anaconda 4.2.0 (64-bit)

Full details on the Anaconda distributions can be found on the Continuum website at:

• http://docs.continuum.io/anaconda/index.html

45

https://www.continuum.io/

https://conda.io/miniconda.html

http://docs.continuum.io/anaconda/index.html


8.1.1 Packages included in Anaconda distributions

You can list the packages currently available in the distribution you have loaded with the command conda list:

[user@cirrus-login0 ~]$ module load anaconda[user@cirrus-login0 ~]$ conda list# packages in environment at /lustre/sw/anaconda/anaconda2:#_license 1.1 py27_1alabaster 0.7.10 py27_0anaconda custom py27_0anaconda-client 1.6.3 py27_0anaconda-navigator 1.2.3 py27_0anaconda-project 0.6.0 py27_0argcomplete 1.0.0 py27_1asn1crypto 0.22.0 py27_0astroid 1.4.9 py27_0...

8.1.2 Adding packages to the Anaconda distribution

Adding packages to the central Anaconda distribution cannot be done by users. If you wish to have additional pack-ages, we recommend installing your own local version of Miniconda and adding the packages you need. This approachis described in Custom Environment below.

8.2 Custom Environments

To setup a custom Python environment including packages that are not in the central installation, the simplest approachis the install Miniconda locally in your own directories.

8.2.1 Installing Miniconda

First, you should download Miniconda. You can use wget to do this, for example:

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

You can find links to the various miniconda versions on the Miniconda website:

• https://conda.io/miniconda.html

For Cirrus, you should use the Linux 64-bit (bash installer).

Once you have downloaded the installer, you can run it. For example:

[user@cirrus-login0 ~]$ bash Miniconda3-latest-Linux-x86_64.sh

Welcome to Miniconda3 4.3.31

In order to continue the installation process, please review the licenseagreement.Please, press ENTER to continue>>>====================================Miniconda End User License Agreement

46 Chapter 8. Using Python

https://conda.io/miniconda.html


====================================

---snip---

Do you accept the license terms? [yes|no][no] >>> yes

Miniconda3 will now be installed into this location:/lustre/home/t01/user/miniconda3

- Press ENTER to confirm the location- Press CTRL-C to abort the installation- Or specify a different location below

[/lustre/home/t01/user/miniconda3] >>>PREFIX=/lustre/home/t01/user/miniconda3installing: python-3.6.3-h6c0c0dc_5 ...installing: ca-certificates-2017.08.26-h1d4fec5_0 ...installing: conda-env-2.6.0-h36134e3_1 ...installing: libgcc-ng-7.2.0-h7cc24e2_2 ...installing: libstdcxx-ng-7.2.0-h7a57d05_2 ...installing: libffi-3.2.1-hd88cf55_4 ...installing: ncurses-6.0-h9df7e31_2 ...installing: openssl-1.0.2n-hb7f436b_0 ...installing: tk-8.6.7-hc745277_3 ...installing: xz-5.2.3-h55aa19d_2 ...installing: yaml-0.1.7-had09818_2 ...installing: zlib-1.2.11-ha838bed_2 ...installing: libedit-3.1-heed3624_0 ...installing: readline-7.0-ha6073c6_4 ...installing: sqlite-3.20.1-hb898158_2 ...installing: asn1crypto-0.23.0-py36h4639342_0 ...installing: certifi-2017.11.5-py36hf29ccca_0 ...installing: chardet-3.0.4-py36h0f667ec_1 ...installing: idna-2.6-py36h82fb2a8_1 ...installing: pycosat-0.6.3-py36h0a5515d_0 ...installing: pycparser-2.18-py36hf9f622e_1 ...installing: pysocks-1.6.7-py36hd97a5b1_1 ...installing: ruamel_yaml-0.11.14-py36ha2fb22d_2 ...installing: six-1.11.0-py36h372c433_1 ...installing: cffi-1.11.2-py36h2825082_0 ...installing: setuptools-36.5.0-py36he42e2e1_0 ...installing: cryptography-2.1.4-py36hd09be54_0 ...installing: wheel-0.30.0-py36hfd4bba0_1 ...installing: pip-9.0.1-py36h6c6f9ce_4 ...installing: pyopenssl-17.5.0-py36h20ba746_0 ...installing: urllib3-1.22-py36hbe7ace6_0 ...installing: requests-2.18.4-py36he2e5f8d_1 ...installing: conda-4.3.31-py36_0 ...installation finished.WARNING:

You currently have a PYTHONPATH environment variable set. This may causeunexpected behavior when running the Python interpreter in Miniconda3.For best results, please verify that your PYTHONPATH only points todirectories of packages that are compatible with the Python interpreterin Miniconda3: /lustre/home/t01/user/miniconda3

Do you wish the installer to prepend the Miniconda3 install location

8.2. Custom Environments 47


to PATH in your /lustre/home/t01/user/.bashrc ? [yes|no][no] >>>

You may wish to edit your .bashrc to prepend the Miniconda3 install location to PATH:

export PATH=/lustre/home/t01/user/miniconda3/bin:$PATH

Thank you for installing Miniconda3!

Miniconda is now installed in your local directories but we still need to setup a way to access it correctly. There are anumber of ways to do this.

• If you are always going to be using this Python environment on ARCHER and do not wish to use any otherPython environment, you can follow the advice of the Miniconda installer and add a line to your .bashrc file

• You can export PATH every time you wish to use you local install by using the bash command exportPATH=/lustre/home/t01/user/miniconda3/bin:$PATH (using the correct PATH as specified bythe installer). This will become tedious if you use the environment often!

• You can create an alias in your .bashrc file to set the path. For example, adding the line aliascondasetup="export PATH=/lustre/home/t01/user/miniconda3/bin:$PATH" would al-low you to use the command condasetup to initialise the Miniconda environment.

• You could also create a modulefile to provide a way to initialise the environment using module load ...as we do for our Anaconda environments. Please contact the helpdesk if you want help to do this.

8.2.2 Installing packages into Miniconda

Once you have installed Miniconda and setup your environment to access it, you can then add whatever packages youwish to the installation using the conda install ... command. For example:

[user@cirrus-login0 ~]$ conda install numpyFetching package metadata ...............Solving package specifications: .

Package plan for installation in environment /lustre/home/t01/user/miniconda3:

The following NEW packages will be INSTALLED:

blas: 1.1-openblas conda-forgelibgfortran: 3.0.0-1numpy: 1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]openblas: 0.2.20-7 conda-forge

The following packages will be UPDATED:

conda: 4.3.31-py36_0 --> 4.3.33-py36_0 conda-→˓forge

The following packages will be SUPERSEDED by a higher-priority channel:

conda-env: 2.6.0-h36134e3_1 --> 2.6.0-0 conda-→˓forge

Proceed ([y]/n)? y

conda-env-2.6. 100% |#################################################################→˓#######| Time: 0:00:00 33.71 kB/s



libgfortran-3. 100% |#################################################################→˓#######| Time: 0:00:00 7.85 MB/sopenblas-0.2.2 100% |#################################################################→˓#######| Time: 0:00:03 4.84 MB/sblas-1.1-openb 100% |#################################################################→˓#######| Time: 0:00:00 1.33 MB/snumpy-1.14.0-p 100% |#################################################################→˓#######| Time: 0:00:01 5.00 MB/sconda-4.3.33-p 100% |#################################################################→˓#######| Time: 0:00:00 5.71 MB/sHere we see the numpy module has been installed in the local environment:

[user@cirrus-login0 ~]$ conda list# packages in environment at /lustre/home/t01/user/miniconda3:#asn1crypto 0.23.0 py36h4639342_0blas 1.1 openblas conda-forgeca-certificates 2017.08.26 h1d4fec5_0certifi 2017.11.5 py36hf29ccca_0cffi 1.11.2 py36h2825082_0chardet 3.0.4 py36h0f667ec_1conda 4.3.33 py36_0 conda-forgeconda-env 2.6.0 0 conda-forgecryptography 2.1.4 py36hd09be54_0idna 2.6 py36h82fb2a8_1libedit 3.1 heed3624_0libffi 3.2.1 hd88cf55_4libgcc-ng 7.2.0 h7cc24e2_2libgfortran 3.0.0 1libstdcxx-ng 7.2.0 h7a57d05_2ncurses 6.0 h9df7e31_2numpy 1.14.0 py36_blas_openblas_200 [blas_openblas]→˓conda-forgeopenblas 0.2.20 7 conda-forgeopenssl 1.0.2n hb7f436b_0pip 9.0.1 py36h6c6f9ce_4pycosat 0.6.3 py36h0a5515d_0pycparser 2.18 py36hf9f622e_1pyopenssl 17.5.0 py36h20ba746_0pysocks 1.6.7 py36hd97a5b1_1python 3.6.3 h6c0c0dc_5readline 7.0 ha6073c6_4requests 2.18.4 py36he2e5f8d_1ruamel_yaml 0.11.14 py36ha2fb22d_2setuptools 36.5.0 py36he42e2e1_0six 1.11.0 py36h372c433_1sqlite 3.20.1 hb898158_2tk 8.6.7 hc745277_3urllib3 1.22 py36hbe7ace6_0wheel 0.30.0 py36hfd4bba0_1xz 5.2.3 h55aa19d_2yaml 0.1.7 had09818_2zlib 1.2.11 ha838bed_2

Please note, for some package installations it may also be necessary to specify a channel such as conda-forge. Forexample, the following command installs the pygobject module.

8.2. Custom Environments 49


[user@cirrus-login0 ~]$ conda install -c conda-forge pygobject


CHAPTER 9

References and further reading

9.1 Online Documentation and Resources

• GNU compiler online documentation: http://gcc.gnu.org/onlinedocs/

• MPI Home pages: http://www-unix.mcs.anl.gov/mpi/

• Free MPI implementation useful for testing: http://www.open-mpi.org/software/ompi/v1.2/

• Various HPC Workshops by NCCS: http://www.nccs.gov/user-support/training-education/workshop-archives/

• An HPC tutorial: http://www.llnl.gov/computing/hpc/training/

• An MPI tutorial: http://www-unix.mcs.anl.gov/mpi/tutorial/gropp/talk.html

• HPC tutorials by NCSA http://www.citutor.org/login.html

9.2 MPI programming

• MPI: The Complete Reference. Snir, Otto, Huss-Lederman, Walker and Dongarra. MIT Press. ISBN 0 26269184 1

• MPI: The Complete Reference, volume 2. Gropp et al. MIT Press. ISBN 0262571234

• Using MPI. Gropp, Lusk, Skjellum. MIT Press. ISBN 0 262 57104 8

9.3 OpenMP programming

• Parallel Programming in OpenMP. Chandra, Kohr, Menon, Dagum, Maydan, McDonald. Morgan Kaufmann.ISBN: 1558606718

51

http://gcc.gnu.org/onlinedocs/

http://www-unix.mcs.anl.gov/mpi/

http://www.open-mpi.org/software/ompi/v1.2/

http://www.nccs.gov/user-support/training-education/workshop-archives/

http://www.llnl.gov/computing/hpc/training/

http://www-unix.mcs.anl.gov/mpi/tutorial/gropp/talk.html

http://www.citutor.org/login.html


9.4 Parallel programming

• Practical Parallel Programming. Gregory V. Wilson. MIT Press. ISBN 0 262 23186 7

• Designing and Building Parallel Programs. Ian Foster. Addison-Wesley. ISBN 0 201 57594 9 http://www.mcs.anl.gov/dbpp/

• Parallel Computing Works! Roy D. Williams, Paul C. Messina (Editor), Geoffrey Fox (Editor), Mark FoxMorgan Kaufmann Publishers; ISBN: 1558602534

• Parallel programming with MPI. Peter S. Pancheco. The complete set of C and Fortran example programs forthis book are available at: http://www.cs.usfca.edu/mpi

9.5 Programming languages

• Fortran90/95 Explained. Metcalf and Reid. Oxford Science Publications. ISBN 0 19 851888 9

• Fortran 90 Programming. Ellis, Philips, Lahey. Addison-Wesley. ISBN 0-201-54446-6

• Programmers Guide to Fortran90. Brainerd, Goldberg, Adams. Unicomp. ISBN 0-07-000248-7

• The High Performance Fortran Handbook. Koelbel, Loveman, Schreiber, Steele, Zosel. ISBN 0-262-11185-3 /0-262-61094-9

• Parallel Programming using C++. G.V.Wilson and P Lu. MIT Press. ISBN 0 262 73118 5

9.6 Programming skills

• Debugging and Performance Tuning for Parallel Computing Systems, Simmons et al.

• Foundations of Parallel Programming, A Machine-independent Approach, Lewis.

52 Chapter 9. References and further reading

http://www.mcs.anl.gov/dbpp/

http://www.mcs.anl.gov/dbpp/

http://www.cs.usfca.edu/mpi

CHAPTER 10

Altair Hyperworks

Hyperworks includes best-in-class modeling, linear and nonlinear analyses, structural and system-level optimization,fluid and multi-body dynamics simulation, electromagnetic compatibility (EMC), multiphysics analysis, model-baseddevelopment, and data management solutions.

10.1 Useful Links

• Hyperworks 14 User Guide

10.2 Using Hyperworks on Cirrus

Hyperworks is licenced software so you require access to a Hyperworks licence to access the software. For queries onaccess to Hyperworks on Cirrus and to enable your access please contact the Cirrus helpdesk.

The standard mode of using Hyperworks on Cirrus is to use the installation of the Desktop application on your localworkstation or laptop to set up your model/simulation. Once this has been doen you would transsfer the required filesover to Cirrus using SSH and then launch the appropriate Solver program (OptiStruct, RADIOSS, MotionSolve).

Once the Solver has finished you can transfer the output back to your local system for visualisation and analysis in theHyperworks Desktop.

10.3 Running serial Hyperworks jobs

Each of the Hyperworks Solvers can be run in serial on Cirrus in a similar way. You should construct a batch submis-sion script with the command to launch your chosen Solver with the correct command line options.

For example, here is a job script to run a serial RADIOSS job on Cirrus:

53

http://www.altairhyperworks.com/

http://www.altairhyperworks.com/hwhelp/Altair/hw14.0/help/altair_help/altair_help.htm?welcome_page.htm


#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N HW_RADIOSS_test#PBS -l select=1:ncpus=1#PBS -l walltime=0:20:0



# Load Hyperworks modulemodule load altair-hwsolvers/14.0.210

# Run the RADIOSS Solver in serialradioss box.fem

10.4 Running parallel Hyperworks jobs

Only the OptiStruct Solver currently supports parallel execution. OptiStruct supports a number of parallel executionmodes of which two can be used on Cirrus:

• Shared memory (SMP) mode uses multiple cores within a single node

• Distributed memory (SPMD) mode uses multiple cores across multiple nodes via the MPI library

10.4.1 OptiStruct SMP

• OptiStruct SMP documentation

You can use up to 36 physical cores (or 72 virtual cores using HyperThreading) for OptiStruct SMP mode as these arethe maximum numbers available on each Cirrus compute node.

You use the -nt option to OptiStruct to specify the number of cores to use.

For example, to run an 18-core OptiStruct SMP calculation you could use the following job script:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N HW_OptiStruct_SMP

# Use 18 cores for this calculation#PBS -l select=1:ncpus=18#PBS -l walltime=0:20:0



# Load Hyperworks modulemodule load altair-hwsolvers/14.0.210

54 Chapter 10. Altair Hyperworks

http://www.altairhyperworks.com/hwhelp/Altair/hw14.0/help/hwsolvers/hwsolvers.htm?shared_memory_parallelization.htm


# Run the OptStruct SMP Solveroptistruct box.fem -nt 18

10.4.2 OptiStruct SPMD (MPI)

• OptiStruct SPMD documentation

There are four different parallelisation schemes for SPMD OptStruct that are selected by different flags:

• Load decomposition (master/slave): -mpimode flag

• Domain decompostion: -ddmmode flag

• Multi-model optimisation: -mmomode flag

• Failsafe topology optimisation: -fsomode flag

You should launch OptiStruct SPMD using the standard Intel MPI mpirun command.

Note: OptiStruct does not support the use of SGI MPT, you must use Intel MPI.

Example OptiStruct SPMD job submission script:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N HW_OptiStruct_SPMD

# Use 2 nodes for this calculation#PBS -l select=2:ncpus=72#PBS -l walltime=0:20:0



# Load Hyperworks module and Intel MPImodule load altair-hwsolvers/14.0.210module load intel-mpi-17

# Run the OptStruct SPMD Solver (domain decompostion mode)# Use 72 cores, 36 on each node (i.e. all physical cores)mpirun -n 72 -ppn 36 $ALTAIR_HOME/hwsolvers/optistruct/bin/linux64/optistruct_14.0.→˓211_linux64_impi box.fem -ddmmode

10.4. Running parallel Hyperworks jobs 55

http://www.altairhyperworks.com/hwhelp/Altair/hw14.0/help/hwsolvers/hwsolvers.htm?optistruct_spmd.htm


56 Chapter 10. Altair Hyperworks

CHAPTER 11

ANSYS Fluent

ANSYS Fluent is a computational fluid dynamics (CFD) tool. Fluent includes well-validated physical modellingcapabilities to deliver fast, accurate results across the widest range of CFD and multi-physics applications.

11.1 Useful Links

• ANSYS Fluent User Guides

11.2 Using ANSYS Fluent on Cirrus

ANSYS Fluent on Cirrus is only available to researchers who bring their own licence. Other users cannot accessthe version centrally-installed on Cirrus.

If you have any questions regarding ANSYS Fluent on Cirrus please contact the Cirrus Helpdesk.

11.3 Running parallel ANSYS Fluent jobs

The following batch file starts Fluent in a command line mode (no GUI) and starts the Fluent batch file “inputfile”.One parameter that requires particular attention is “-t504”. In this example 14 Cirrus nodes (14 * 72 = 1008 cores) areallocated; where half of the 1008 cores are physical and the other half are virtual. To run fluent optimally on Cirrus,only the physical cores should be employed. As such, fluent’s -t flag should reflect the number of physical cores: inthis example, “-t504” is employed.

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N ANSYS_test#PBS -l select=14:ncpus=72#PBS -l walltime=23:04:0

57

http://www.ansys.com/Products/Fluids/ANSYS-Fluent

http://www.ansys.com/Products/Fluids/ANSYS-Fluent

http://www.cirrus.ac.uk/support/


#PBS -l place=excl#PBS -k oe


module purgemodule load mpt perfboostmodule load ansys

export OMP_NUM_THREADS=1export SGI_MPI_HOME=$MPI_ROOTuniq $PBS_NODEFILE | cut -d . -f 1 > ~/fluent.launcher.host.txt

cd $PBS_O_WORKDIR./fluent 3ddp -g -i inputfile.fl \

-pinfiniband -alnamd64 -r17.2.0 -t504 -mpi=intel \-cnf=~/fluent.launcher.host.txt \-path/lustre/sw/ansys/v172/fluent/ -ssh >& outputfile.txt

Below is the Fluent “inputfile.fl” batch script. Anything that starts with a “;” is a comment. This script does thefollowing:

• Starts a transcript (i.e. Fluent output is redirected to a file [transcript_output_01.txt])

• Reads a case file [a case file in Fluent is a model]

• Reads a data file [a data file in Fluent is the current state of a simulation (i.e. after X iterations)]

• Prints latency and bandwidth statistics

• Prints and resets timers

• Run 50 iterations of the simulation

• Prints and resets timers

• Save the data file (so that you can continue the simulation)

• Stops the transcript

• Exits Fluent

11.4 Actual Fluent script (“inputfile.fl”):

Replace [Your Path To ] before running

; Start transcript/file/start-transcript [Your Path To ]/transcript_output_01.txt; Read case filerc [Your Path To ]/200M-CFD-Benchmark.cas; Read data file/file/read-data [Your Path To ]/200M-CFD-Benchmark-500.dat; Print statistics/parallel/bandwidth/parallel/latency/parallel/timer/usage/parallel/timer/reset; Calculate 50 iterations

58 Chapter 11. ANSYS Fluent


it 50; Print statistics/parallel/timer/usage/parallel/timer/reset; Write data filewd [Your Path To ]/200M-CFD-Benchmark-500-new.dat; Stop transcript/file/stop-transcript; Exit Fluentexityes

11.4. Actual Fluent script (“inputfile.fl”): 59


60 Chapter 11. ANSYS Fluent

CHAPTER 12

CASTEP

CASTEP is a leading code for calculating the properties of materials from first principles. Using density functionaltheory, it can simulate a wide range of properties of materials proprieties including energetics, structure at the atomiclevel, vibrational properties, electronic response properties etc. In particular it has a wide range of spectroscopicfeatures that link directly to experiment, such as infra-red and Raman spectroscopies, NMR, and core level spectra.

12.1 Useful Links

• CASTEP User Guides

• CASTEP Tutorials

• CASTEP Licensing

12.2 Using CASTEP on Cirrus

CASTEP is only available to users who have a valid CASTEP licence.

If you have a CASTEP licence and wish to have access to CASTEP on Cirrus please contact the Cirrus Helpdesk.

12.3 Running parallel CASTEP jobs

CASTEP can exploit multiple nodes on Cirrus and will generally be run in exclusive mode over more than one node.

For example, the following script will run a CASTEP job using 4 nodes (144 cores).

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N CASTEP_test#PBS -l select=4:ncpus=72

61

http://www.castep.org

http://www.castep.org/CASTEP/Documentation

http://www.castep.org/CASTEP/OnlineTutorials

http://www.castep.org/CASTEP/GettingCASTEP



#PBS -l place=excl#PBS -l walltime=0:20:0



# Load CASTEP and MPI modulesmodule load castepmodule load mpt

# Run using input in test_calc.in# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by defaultmpiexec_mpt -n 144 -ppn 36 castep.mpi test_calc

62 Chapter 12. CASTEP

CHAPTER 13

CP2K

CP2K is a quantum chemistry and solid state physics software package that can perform atomistic simulations of solidstate, liquid, molecular, periodic, material, crystal, and biological systems. CP2K provides a general framework fordifferent modelling methods such as DFT using the mixed Gaussian and plane waves approaches GPW and GAPW.Supported theory levels include DFTB, LDA, GGA, MP2, RPA, semi-empirical methods (AM1, PM3, PM6, RM1,MNDO, . . . ), and classical force fields (AMBER, CHARMM, . . . ). CP2K can do simulations of molecular dynamics,metadynamics, Monte Carlo, Ehrenfest dynamics, vibrational analysis, core level spectroscopy, energy minimisation,and transition state optimisation using NEB or dimer method.

13.1 Useful Links

• CP2K Reference Manual

• CP2K HOWTOs

• CP2K FAQs

13.2 Using CP2K on Cirrus

CP2K is available through the cp2k-mpt module. MPI only cp2k.popt and MPI/OpenMP Hybrid cp2k.psmpbinaries are available.

IMPORTANT: Running cp2k in hybrid OpenMP/MPI mode requires some non-standard steps. Please see the Run-ning CP2K in OpenMP/MPI Hybrid Mode section below for further details.

13.3 Running parallel CP2K jobs - MPI Only

To run CP2K using MPI only, load the cp2k-mpt module and use the cp2k.popt executable.

For example, the following script will run a CP2K job using 4 nodes (144 cores):

63

https://www.cp2k.org/

https://manual.cp2k.org/#gsc.tab=0

https://www.cp2k.org/howto

https://www.cp2k.org/faq


#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N CP2K_test#PBS -l select=4:ncpus=72#PBS -l place=excl#PBS -l walltime=0:20:0



# Load CASTEP and MPI modulesmodule load cp2k-mptmodule load mptmodule load intel-cmkl-16

#Ensure that no libraries are inadvertently using threadingexport OMP_NUM_THREADS=1

# Run using input in test.inp# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by defaultmpiexec_mpt -n 144 -ppn 36 cp2k.popt -i test.inp

13.4 Running Parallel CP2K Jobs - MPI/OpenMP Hybrid Mode

To run CP2K using MPI and OpenMP, load the cp2k-mpt module and use the cp2k.psmp executable.

Due to a thread placement bug in SGI MPT’s omplace, tool for GCC-compiled software, launching the executablemust be achieved in a different way to other hybrid OpenMP/MPI codes on Cirrus.

You must first run the placement tool (included in the module) to produce a thread placement file, place.txt. Forexample, if you wish to use 6 threads per process, use:

export OMP_NUM_THREADS=6placement $OMP_NUM_THREADS

to produce the placement file. Then launch the executable using mpiexec_mpt and dplace (instead of omplace)as follows:

mpiexec_mpt -n 6 dplace -p place.txt cp2k.psmp ...

For example, the following script will run a CP2K job using 4 nodes, with 6 OpenMP threads per MPI process:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N CP2K_test#PBS -l select=4:ncpus=72#PBS -l place=excl#PBS -l walltime=0:20:0

# Replace [budget code] below with your project code (e.g. t01)

64 Chapter 13. CP2K


#PBS -A [budget code]


# Load CASTEP and MPI modulesmodule load cp2k-mptmodule load mptmodule load intel-cmkl-16

export OMP_NUM_THREADS=6placement $OMP_NUM_THREADS

# Run using input in test.inp# Notes:# - '-ppn 6' is required to use six processes per node# - we use 'dplace' with the placement file 'place.txt' to specify# thread bindingmpiexec_mpt -n 24 -ppn 6 dplace -p place.txt cp2k.psmp -i test.inp

13.4. Running Parallel CP2K Jobs - MPI/OpenMP Hybrid Mode 65


66 Chapter 13. CP2K

CHAPTER 14

FLACS

FLACS from Gexcon is the industry standard for CFD explosion modelling and one of the best validated tools formodeling flammable and toxic releases in a technical safety context.

The Cirrus cluster is ideally suited to run multiple FLACS simulations simultaneously, via its batch system PBS.Short lasting simulations (of typically up to a few hours computing time each) can be processed efficiently and youcould get a few hundred done in a day or two. In contrast, the Cirrus cluster is not particularly suited for runningsingle big FLACS simulations with many threads: each node on Cirrus has 2x4 memory channels, and for memory-bound applications like FLACS multi-threaded execution will not scale linearly beyond eight cores. On most systems,FLACS will not scale well to more than four cores (cf. the FLACS User’s Manual), and therefore multi-core hardwareis normally best used by increasing the number of simulations running in parallel rather than by increasing the numberof cores per simulation.

Gexcon has two different service offerings on Cirrus: FLACS-Cloud and FLACS-HPC. From FLACS v10.7, FLACS-Cloud is the preferable way to exploit the HPC cluster, directly from the FLACS graphical user interfaces. For userswho are familiar with accessing remote Linux HPC systems manually, FLACS-HPC may be an option. Both servicesare presented below.

14.1 FLACS-Cloud

FLACS-Cloud is a high performance computing service available right from the FLACS-Risk user interface, as wellas from the FLACS RunManager. It allows you to run FLACS simulations on the high performance cloud computinginfrastructure of Gexcon’s partner EPCC straight from the graphical user interfaces of FLACS – no need to manuallylog in, transfer data, or start jobs!

By using the FLACS-Cloud service, you can run a large number of simulations very quickly, without having to investinto in-house computing hardware. The FLACS-Cloud service scales to your your demand and facilitates runningprojects with rapid development cycles.

The workflow for using FLACS-Cloud is described in the FLACS User’s Manual and in the FLACS-Risk documen-tation; you can also find basic information in the knowledge base of the FLACS User Portal (accessible for FLACSlicense holders).

67

http://www.gexcon.com/index.php?/flacs-software/article/FLACS-Overview

http://www.gexcon.com

../user-guide/batch.html

https://gexcon.freshdesk.com/solution/categories/14000072843


14.2 FLACS-HPC

Compared to FLACS-Cloud, the FLACS-HPC service is built on more traditional ways of accessing and using a remoteLinux cluster. Therefore the user experience is more basic, and FLACS has to be run manually. For an experienceduser, however, this way of exploiting the HPC system can be at least as efficient as FLACS-Cloud.

Follow the steps below to use the FLACS-HPC facilities on Cirrus.

Note: The instructions below assume you have a valid account on Cirrus. To get an account please first get in touchwith FLACS support at [email protected] and then see the instructions in the Tier-2 SAFE Documentation.

Note: In the instructions below you should substitute “username” by your actual Cirrus username.

14.2.1 Log into Cirrus

Log into Cirrus following the instructions at Connecting to Cirrus.

14.2.2 Upload your data to Cirrus

Transfer your data to Cirrus by following the instructions at Data Transfer Guide.

For example, to copy the scenario definition files from the current directory to the folder project_folder in yourhome directory on Cirrus run the following command on your local machine:

rsync -avz c*.dat3 [email protected]:project_folder

Note that this will preserve soft links as such; the link targets are not copied if they are outside the current directory.

14.2.3 Submit a FLACS job to the queue

To run FLACS on Cirrus you must first change to the directory where your FLACS jobs are located, use the cd (changedirectory) command for Linux. For example

cd projects/sim

Load the flacs module to make the application available:

module load flacs

Submit your FLACS jobs using the qsub command. For example:

qsub -A xyz -l select=1:ncpus=1 -l walltime=6:00:00 -q flacs -- /lustre/sw/flacs/10.5.→˓1/FLACS_v10.5/bin/run_runflacs -dir projects/sim 010101

The -A xyz option is obligatory and states the account xyz that the CPU consumption will be billed to. You cancheck your account in SAFE.

The -l select=x:ncpus=y option specifies the resource allocation for the job you are starting. The parameterx is the number of nodes required and the parameter y is the number of cores required. For a serial FLACS job youwould use -l select=1:ncpus=1

The maximum length of time (i.e. walltime) you want the job to run is specified with the -lwalltime=[hh:mm:ss] option. After this time, your job will be stopped by the job scheduler. Setting a veryhigh walltime limit may lead to your job being given lower priority and thus wait longer in the queue. The defaultwalltime is 12 hours.

68 Chapter 14. FLACS




All Flacs jobs must be submitted to the flacs queue using the option -q flacs; the flacs queue ensures FLACSlicenses are provisioned correctly for the jobs.

After the -- which marks the beginning of the command to run, the Flacs executable is given with its absolute path.Having loaded the flacs module (see above) you can find the location by

which run_runflacs

The run_runflacs command in turn needs two arguments: first, after -dir, the directory where to run the joband create the output; if it is the current directory then you can pass -dir `pwd`. Second, the job number of theFLACS scenario.

14.2.4 Submit FLACS jobs from a script

In your script, change to the directory with the job files and load the flacs module as explained above.

When submitting several jobs it is advisable to add the -N name option to the qsub command, with the FLACS jobnumber being part of the first ten characters of the name. In this way you can easily identify the jobs in the queue (seebelow).

During testing it has been shown that job submission to the queue runs more smoothly when there is a short delay of5 seconds before subsequent qsub commands.

A script submitting the scenarios 000012, 000023 and 000117 to the queue could look like this:

module load flacs/10.5.1sleep 5; qsub -A xyz -l select=1:ncpus=1 -l walltime=24:00:00 -N f-000012 -q flacs -V→˓-- `which run_runflacs` -dir `pwd` 000012sleep 5; qsub -A xyz -l select=1:ncpus=1 -l walltime=24:00:00 -N f-000023 -q flacs -V→˓-- `which run_runflacs` -dir `pwd` 000023sleep 5; qsub -A xyz -l select=1:ncpus=1 -l walltime=24:00:00 -N f-000117 -q flacs -V→˓-- `which run_runflacs` -dir `pwd` 000117

This is also easy to formulate as a loop.

14.2.5 Monitor your jobs

You can monitor the progress of your jobs with the qstat command. This will list all jobs that are running or queuedon the system. To list only your jobs use:

qstat -u username

14.2.6 Submitting many FLACS jobs as a job array

Running many related scenarios with the Flacs simulator is ideally suited for using job arrays, i.e. running the simu-lations as part of a single job.

A job script for running a job array with 128 Flacs scenarios that are located in the current directory could look likethis:

#!/bin/bash --login#PBS -l select=1:ncpus=1#PBS -N disp2#PBS -J 1-128#PBS -j oe

14.2. FLACS-HPC 69

../user-guide/batch.html#job-arrays


#PBS -l walltime=48:00:00#PBS -q flacs#PBS -V

cd ${PBS_O_WORKDIR}

CS_FILES=(`ls -1 cs??????.dat3`)# NR_OF_JOBS=${#CS_FILES[@]}JOB_FIRST=1JOB_LAST=128for (( i=0; i<$(expr ${JOB_LAST} - ${JOB_FIRST}); i++ ));do

JOB_IDS[${i}]=${CS_FILES[$(expr $i + ${JOB_FIRST})]:2:6}done

module load flacsJOB_INDEX=$(( $PBS_ARRAY_INDEX - 1 ))

`which run_runflacs` ${JOB_IDS[${JOB_INDEX}]}

Due to the way the job scheduler interprets this script, the number of jobs has to be hard-coded in the first (non-bash)part of the job script and cannot be determined based on the number of scenarios in the current directory.

14.2.7 Transfer data from Cirrus to your local system

After your simulations are finished, transfer the data back from Cirrus following the instructions at Data TransferGuide.

For example, to copy the result files from the directory project_folder in your home directory on Cirrus to thefolder /tmp on your local machine use:

rsync -rvz --include='r[13t]*.*' --exclude='*' [email protected]:project_→˓folder/ /tmp

14.2.8 Billing for FLACS-HPC use on Cirrus

CPU time on Cirrus is measured in CPUh for each job run on a compute node, based on the number of physicalcores employed. Only jobs submitted to compute nodes via qsub are charged. Any processing on a login nodeis not charged. However, using login nodes for computations other than simple pre- or post- processing is stronglydiscouraged.

Gexcon normally bills monthly for the use of FLACS-Cloud and FLACS-HPC, based on the Cirrus CPU usage logging.

14.3 Getting help

Get in touch with FLACS Support by email to [email protected] if you encounter any problems. For issues relatedto Cirrus rather than FLACS contact the Cirrus helpdesk.

70 Chapter 14. FLACS



CHAPTER 15

Gaussian

Gaussian is a general-purpose computational chemistry package.

15.1 Useful Links

• Gaussian User Guides

15.2 Using Gaussian on Cirrus

Gaussian on Cirrus is only available to University of Edinburgh researchers through the University’s site li-cence. Users from other institutions cannot access the version centrally-installed on Cirrus.

If you wish to have access to Gaussian on Cirrus please contact the Cirrus Helpdesk.

Gaussian cannot run across multiple nodes. This means that the maximum number of cores you can use for Gaussianjobs is 36 (the number of cores on a compute node). In reality, even large Gaussian jobs will not be able to makeeffective use of more than 8 cores. You should explore the scaling and performance of your calculations on the systembefore running production jobs.

15.3 Scratch Directories

Before using Gaussian for the first time, you should create a directory in your home directories to hold temporary filesused by Gaussian, e.g.

mkdir ~/g09tmp

71

http://www.gaussian.com/

http://gaussian.com/techsupport/



15.4 Running serial Gaussian jobs

In many cases you will use Gaussian in serial mode. The following example script will run a serial Gaussian job onCirrus (before using, ensure you have created a Gaussian scratch directory as outlined above).

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N G09_test#PBS -l select=1:ncpus=1#PBS -l walltime=0:20:0



# Load Gaussian modulemodule load gaussian

# Setup the Gaussian environmentsource $g09root/g09/bsd/g09.profile

# Location of the scratch directoryexport GAUSS_SCRDIR=$HOME/g09tmp

# Run using input in "test0027.com"g09 test0027

15.5 Running parallel Gaussian jobs

Gaussian on Cirrus can use shared memory parallelism through OpenMP by setting the OMP_NUM_THREADS envi-ronment variable. The number of cores requested in the job should also be modified to match.

For example, the following script will run a Gaussian job using 4 cores.

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N G09_test#PBS -l select=1:ncpus=4#PBS -l walltime=0:20:0



# Load Gaussian modulemodule load gaussian

# Setup the Gaussian environmentsource $g09root/g09/bsd/g09.profile

72 Chapter 15. Gaussian


# Location of the scratch directoryexport GAUSS_SCRDIR=$HOME/g09tmp

# Run using input in "test0027.com"export OMP_NUM_THREADS=4g09 test0027

15.5. Running parallel Gaussian jobs 73


74 Chapter 15. Gaussian

CHAPTER 16

GROMACS

GROMACS GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equationsof motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules likeproteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremelyfast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it forresearch on non-biological systems, e.g. polymers.

16.1 Useful Links

• GROMACS User Guides

• GROMACS Tutorials

16.2 Using GROMACS on Cirrus

GROMACS is Open Source software and is freely available to all Cirrus users. Four versions are available:

• Serial/shared memory, single precision: gmx

• Serial/shared memory, double precision: gmx_d

• Parallel MPI/OpenMP, single precision: gmx_mpi

• Parallel MPI/OpenMP, double precision: gmx_mpi_d

16.3 Running parallel GROMACS jobs: pure MPI

GROMACS can exploit multiple nodes on Cirrus and will generally be run in exclusive mode over more than onenode.

For example, the following script will run a GROMACS MD job using 4 nodes (144 cores) with pure MPI.

75

http://www.gromacs.org/

http://manual.gromacs.org/documentation/

http://www.gromacs.org/Documentation/Tutorials


#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N mdrun_test#PBS -l select=4:ncpus=72# Make sure you are not sharing nodes with other users#PBS -l place=excl#PBS -l walltime=0:20:0



# Load GROMACS and MPI modulesmodule load gromacsmodule load mpt

# Run using input in test_calc.tpr# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by defaultmpiexec_mpt -n 144 -ppn 36 gmx_mpi mdrun -s test_calc.tpr

16.4 Running parallel GROMACS jobs: hybrid MPI/OpenMP

The following script will run a GROMACS MD job using 4 nodes (144 cores) with 6 MPI processes per node (24 MPIprocesses in total) and 6 OpenMP threads per MPI process.

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N mdrun_test#PBS -l select=4:ncpus=72# Make sure you are not sharing nodes with other users#PBS -l place=excl#PBS -l walltime=0:20:0



# Load GROMACS and MPI modulesmodule load gromacsmodule load mpt

# Run using input in test_calc.tprexport OMP_NUM_THREADS=6mpiexec_mpt -n 24 -ppn 6 omplace -nt 6 gmx_mpi mdrun -s test_calc.tpr

76 Chapter 16. GROMACS

CHAPTER 17

HELYX®

HELYX is a comprehensive, general-purpose computational fluid dynamics (CFD) software package for engineeringanalysis and design optimisation developed by ENGYS. The package features an advanced open-source CFD sim-ulation engine and a client-server GUI to provide a flexible and cost-effective HPC solver platform for enterpriseapplications.

17.1 Useful Links

• Information about HELYX

• Information about ENGYS

17.2 Using HELYX on Cirrus

HELYX is only available on Cirrus to authorised users with a valid license to use the software. For any queriesregarding HELYX on Cirrus, please contact ENGYS or the Cirrus Helpdesk.

HELYX applications can be run on Cirrus in two ways:

• Manually from the command line, using a SSH terminal to access the cluster’s master node.

• Interactively from within the GUI, using the dedicated client-server node to connect remotely to the cluster.

A complete user’s guide to access HELYX on demand via Cirrus is provided by ENGYS as part of the service offering.

17.3 Running Parallel HELYX Jobs

The standard execution of HELYX applications on Cirrus is handled through the command line using a submissionscript to control PBSPro. A basic submission script for running multiple HELYX applications in parallel using theSGI-MPT (Message Passing Toolkit) module is included below. In this example the applications helyxHexMesh andcaseSetup are run sequentially in 144 cores.

77

https://engys.com/products/helyx

https://engys.com/about-us

https://engys.com/contact-us



#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N HELYX_MPI_Job#PBS -l select=4:ncpus=72#PBS -l place=excl#PBS -l walltime=00:20:00



# Load any required modulesmodule load mpt/2.14module load gcc/6.2.0

# Load HELYX-Core environmentexport FOAM_INST_DIR=/lustre/home/y07/helyx/v3.0.2/CORE. /lustre/home/y07/helyx/v3.0.2/CORE/HELYXcore-3.0.2/etc/bashrc

# Set the number of threads to 1export OMP_NUM_THREADS=1

# Launch HELYX applications in parallelexport myoptions="-parallel"jobs="helyxHexMesh caseSetup"

for job in `echo $jobs`do

case "$job" in

* ) options="$myoptions" ;;esac

mpiexec_mpt -n 144 -ppn 36 $job $myoptions 2>&1 | tee log/$job.$PBS_JOBID.out

done

Alternatively, the user can execute most HELYX applications on Cirrus interactively via the GUI by following thesesimple steps:

1. Launch HELYX GUI in the local Windows or Linux machine.

2. Create a client-server connection to Cirrus using the dedicated node provided for this service. Enter the userlogin details and the total number of processors to be employed in the cluster for parallel execution.

3. Use the GUI in the local machine to access the remote file system in Cirrus to load a geometry, create a compu-tational grid, set up a simulation, solve the flow, and post-process the results using the HPC resources availablein the cluster. The scheduling associated with every HELYX job is handled automatically by the client-server.

4. Visualise the remote data from the local machine, perform changes in the model, and complete as many flowsimulations in Cirrus as required. Disconnect the client-server at any point during execution, leave the solverrunning in the cluster, and resume the connection to Cirrus from another client machine to reload an existingcase when needed.

78 Chapter 17. HELYX®

CHAPTER 18

LAMMPS

LAMMPS, is a classical molecular dynamics code, and an acronym for Large-scale Atomic/Molecular MassivelyParallel Simulator. LAMMPS has potentials for solid-state materials (metals, semiconductors) and soft matter(biomolecules, polymers) and coarse-grained or mesoscopic systems. It can be used to model atoms or, more generi-cally, as a parallel particle simulator at the atomic, meso, or continuum scale.

18.1 Useful Links

• LAMMPS User Guide

• LAMMPS Tutorials

18.2 Using LAMMPS on Cirrus

LAMMPS is freely available to all Cirrus users.

18.3 Running parallel LAMMPS jobs

LAMMPS can exploit multiple nodes on Cirrus and will generally be run in exclusive mode over more than one node.

For example, the following script will run a LAMMPS MD job using 4 nodes (144 cores) with pure MPI.

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N lammps_test#PBS -l select=4:ncpus=72# Make sure you are not sharing nodes with other users#PBS -l place=excl#PBS -l walltime=0:20:0

79

http://lammps.sandia.gov/

http://www.ks.uiuc.edu/Research/namd/2.12/ug/

http://www.ks.uiuc.edu/Training/Tutorials/index-all.html#namd




# Load LAMMPS modulemodule load lammps

# Run using input in in.test# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by defaultmpiexec_mpt -n 144 -ppn 36 lmp_mpi < in.test

18.4 Compiling LAMMPS on Cirrus

Compile instructions for LAMMPS on Cirrus can be found on GitHub:

• Cirrus LAMMPS compile instructions

80 Chapter 18. LAMMPS

https://github.com/EPCCed/cirrus-packages/tree/master/LAMMPS

CHAPTER 19

MATLAB

MATLAB combines a desktop environment tuned for iterative analysis and design processes with a programminglanguage that expresses matrix and array mathematics directly.

19.1 Useful Links

• MATLAB Documentation

19.2 Using MATLAB on Cirrus

MATLAB R2016b is available on Cirrus.

This installation of MATLAB on Cirrus is covered by an Academic License - for use in teaching, academic research,and meeting course requirements at degree granting institutions only. Not for government, commercial, or otherorganizational use.

If your use of MATLAB is not covered by this license then please do not use this installation. Please contact theCirrus Helpdesk to arrange use of your own MATLAB license on Cirrus.

This is MATLAB Version 9.1.0.441655 (R2016b) and provides the following toolboxes

MATLAB Version 9.1Simulink Version 8.8Bioinformatics Toolbox Version 4.7Communications System Toolbox Version 6.3Computer Vision System Toolbox Version 7.2Control System Toolbox Version 10.1Curve Fitting Toolbox Version 3.5.4DSP System Toolbox Version 9.3Database Toolbox Version 7.0Datafeed Toolbox Version 5.4Econometrics Toolbox Version 3.5

81

https://uk.mathworks.com

https://uk.mathworks.com/help/index.html



Financial Instruments Toolbox Version 2.4Financial Toolbox Version 5.8Fixed-Point Designer Version 5.3Fuzzy Logic Toolbox Version 2.2.24Global Optimization Toolbox Version 3.4.1HDL Coder Version 3.9HDL Verifier Version 5.1Image Acquisition Toolbox Version 5.1Image Processing Toolbox Version 9.5Instrument Control Toolbox Version 3.10LTE System Toolbox Version 2.3MATLAB Coder Version 3.2MATLAB Compiler Version 6.3MATLAB Compiler SDK Version 6.3MATLAB Report Generator Version 5.1Mapping Toolbox Version 4.4Neural Network Toolbox Version 9.1Optimization Toolbox Version 7.5Parallel Computing Toolbox Version 6.9Partial Differential Equation Toolbox Version 2.3Phased Array System Toolbox Version 3.3RF Toolbox Version 3.1Robust Control Toolbox Version 6.2Signal Processing Toolbox Version 7.3SimBiology Version 5.5SimEvents Version 5.1Simscape Version 4.1Simscape Multibody Version 4.9Simscape Power Systems Version 6.6Simulink 3D Animation Version 7.6Simulink Coder Version 8.11Simulink Control Design Version 4.4Simulink Design Optimization Version 3.1Stateflow Version 8.8Statistics and Machine Learning Toolbox Version 11.0Symbolic Math Toolbox Version 7.1System Identification Toolbox Version 9.5Wavelet Toolbox Version 4.17

19.3 Running parallel MATLAB jobs

MATLAB is intended to be used on the compute nodes within PBS job scripts. Use on the login nodes should berestricted to setting preferences, accessing help, etc. It is recommended that MATLAB is used without a GUI on thecompute nodes, as the interactive response is slow.

The license for this installation of MATLAB provides only 32 workers via MATLAB Distributed Computing Server(MDCS) but provides 36 workers via the local cluster profile (there are 36 cores on a Cirrus compute node), so we donot recommend the use of MDCS for parallel computations.

MATLAB will normally use up to the total number of cores on a node for multi-threaded operations (e.g., matrixinversions) and for parallel computations. It also makes no restriction on its memory use. These features are incom-patible with the shared use of nodes on Cirrus. A wrapper script is provided to limit the number of cores and amountof memory used, in proportion to the number of CPUs selected in the PBS job script. Please use this wrapper insteadof using MATLAB directly. Select an even number of CPUs (ncpus) in the PBS select statement, since MATLAB usesthe cores without Hyper-Threading (number of cores = ncpus / 2).

82 Chapter 19. MATLAB


An example job script would be

#PBS -N Example_MATLAB_Job#PBS -l select=1:ncpus=12#PBS -l walltime=00:20:00



module load matlab

matlab_wrapper -nodisplay < /lustre/sw/cse-matlab/examples/testp.m > testp.log

This would run the testp.m script, without a display, and exit when testp.m has finished. 12 CPUs are selected, whichcorrespond to 6 cores, and the following limits would be set

ncores = 6memory = 42GB

Maximum number of computational threads (maxNumCompThreads) = 6Preferred number of workers in a parallel pool (PreferredNumWorkers) = 6Number of workers to start on your local machine (NumWorkers) = 6Number of computational threads to use on each worker (NumThreads) = 1

Note that these settings persist between MATLAB sessions but will be updated correctly if you use the wrapper eachtime.

NumWorkers and NumThreads can be changed (using parcluster and saveProfile) but NumWorkers * NumThreadsshould be less than the number of cores (ncores above).

If you specify exclusive node access, then all the cores and memory will be available. On the login nodes, a singlecore is used and memory is not limited.

19.3. Running parallel MATLAB jobs 83


84 Chapter 19. MATLAB

CHAPTER 20

NAMD

NAMD, recipient of a 2002 Gordon Bell Award and a 2012 Sidney Fernbach Award, is a parallel molecular dynamicscode designed for high-performance simulation of large biomolecular systems. Based on Charm++ parallel objects,NAMD scales to hundreds of cores for typical simulations and beyond 500,000 cores for the largest simulations.NAMD uses the popular molecular graphics program VMD for simulation setup and trajectory analysis, but is alsofile-compatible with AMBER, CHARMM, and X-PLOR.

20.1 Useful Links

• NAMD User Guide

• NAMD Tutorials

20.2 Using NAMD on Cirrus

NAMD is freely available to all Cirrus users.

20.3 Running parallel NAMD jobs

NAMD can exploit multiple nodes on Cirrus and will generally be run in exclusive mode over more than one node.

For example, the following script will run a NAMD MD job using 4 nodes (144 cores) with pure MPI.

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N namd_test#PBS -l select=4:ncpus=72# Make sure you are not sharing nodes with other users#PBS -l place=excl

85

http://www.ks.uiuc.edu/Research/namd/

http://www.ks.uiuc.edu/Research/namd/2.12/ug/

http://www.ks.uiuc.edu/Training/Tutorials/index-all.html#namd


#PBS -l walltime=0:20:0



# Load NAMD modulemodule load namd

# Run using input in input.namd# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by default# Note: NAMD uses Intel MPI so mpirun should be used instead of# mpiexec_mpt (which is SGI MPI)mpirun -n 144 -ppn 36 namd2 input.namd

86 Chapter 20. NAMD

CHAPTER 21

OpenFOAM

OpenFOAM is an open-source toolbox for computational fluid dynamics. OpenFOAM consists of generic tools tosimulate complex physics for a variety of fields of interest, from fluid flows involving chemical reactions, turbulenceand heat transfer, to solid dynamics, electromagnetism and the pricing of financial options.

The core technology of OpenFOAM is a flexible set of modules written in C++. These are used to build solvers andutilities to perform pre- and post-processing tasks ranging from simple data manipulation to visualisation and meshprocessing.

21.1 Available Versions

You can query the versions of OpenFOAM available on Cirrus from the command line with module availopenfoam.

We currently have OpenFOAM v1706 available on Cirrus.

21.2 Useful Links

• OpenFOAM Documentation

21.3 Using OpenFOAM on Cirrus

To use OpenFOAM on Cirrus you should first load the OpenFOAM module:

module add openfoam

After that you need to source the etc/bashrc file provided by OpenFOAM:

source $OPENFOAM_CURPATH/etc/bashrc

87

https://www.openfoam.com/documentation/


You should then be able to use OpenFOAM. The above commands will also need to be added to any job/batch sub-mission scripts you want to run OpenFOAM from.

88 Chapter 21. OpenFOAM

CHAPTER 22

Quantum Espresso (QE)

Quantum Espresso is an integrated suite of Open-Source computer codes for electronic-structure calculations andmaterials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials.

22.1 Useful Links

• QE User Guides

• QE Tutorials

22.2 Using QE on Cirrus

QE is Open Source software and is freely available to all Cirrus users.

22.3 Running parallel QE jobs

QE can exploit multiple nodes on Cirrus and will generally be run in exclusive mode over more than one node.

For example, the following script will run a QE pw.x job using 4 nodes (144 cores).

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N pw_test#PBS -l select=4:ncpus=72# Make sure you are not sharing nodes with other users#PBS -l place=excl#PBS -l walltime=0:20:0

# Replace [budget code] below with your project code (e.g. t01)

89

http://www.quantum-espresso.org/

http://www.quantum-espresso.org/users-manual/

http://www.quantum-espresso.org/tutorials/


#PBS -A [budget code]


# Load QE and MPI modulesmodule load qemodule load mpt

# Run using input in test_calc.in# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by defaultmpiexec_mpt -n 144 -ppn 36 pw.x test_calc.in

90 Chapter 22. Quantum Espresso (QE)

CHAPTER 23

STAR-CCM+

STAR-CCM+ is a computational fluid dynamics (CFD) code and beyond. It provides a broad range of validatedmodels to simulate disciplines and physics including CFD, computational solid mechanics (CSM), electromagnetics,heat transfer, multiphase flow, particle dynamics, reacting flow, electrochemistry, aero-acoustics and rheology; thesimulation of rigid and flexible body motions with techniques including mesh morphing, overset mesh and six degreesof freedom (6DOF) motion; and the ability to combine and account for the interaction between the various physicsand motion models in a single simulation to cover your specific application.

23.1 Useful Links

• Information about STAR-CCM+ by Siemens

23.2 Licensing

All users must provide their own license for STAR-CCM+. This licence can be provided as:

1. FLEXlm licence key to be installed on Cirrus

2. IP address and port of publicly accesible remote licence (your STAR-CCM+ license server must use the same portsas our License Server Gateway: this is achieved by simply setting an environment variable) 3. Power on Demand(PoD) (nothing needs to be provided to Cirrus in this case)

For options 1 and 2, you should contact the Cirrus Helpdesk with the relevant details.

23.3 Using STAR-CCM+ on Cirrus: Interactive remote GUI Mode

A fast and responsive way of running with a GUI is to install STAR-CCM+ on your local Windows(7 or 10) or Linuxworkstation. You can then start your local STAR-CCM+ and connect to Cirrus in order to submit new jobs or querythe status of running jobs. When you install your local version, de-activate the FLEXIm installation. FLEXIm is notrequired, as you will either be using the FLEXIm server on Cirrus or the Power on Demand (PoD) licence option.

91

https://mdx.plm.automation.siemens.com/star-ccm-plus



You also need to setup passwordless SSH connections to Cirrus.

23.3.1 Jobs using FLEXlm licence server on Cirrus

Before you can use the FLEXlm server on Cirrus, you must provide us with your licence key to install on Cirrus (seeabove).

You can then start the STAR-CCM+ server on the compute nodes. The following script starts the server:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N STAR-CCM_test#PBS -l select=1008#PBS -l walltime=02:00:00#PBS -l place=excl#PBS -k oe


# Change to the directory yhat the job was submitted fromcd $PBS_O_WORKDIR# Load any required modulesmodule load mptmodule load starccm+

export SGI_MPI_HOME=$MPI_ROOT

uniq $PBS_NODEFILE | cut -d . -f 1 > ~/starccm.launcher.host.txtstarccm+ -server -machinefile ~/starccm.launcher.host.txt -np 504 -rsh ssh -port 42333

The port number “42333” should be free. If it is not free STAR-CCM+ will return with an error. You must then try touse another random port in the 42XXX range. You can then use the ‘qstat’ command to find out the first node of yourapplication.

23.3.2 Jobs using remote licence server

The documentation for this option is currently under construction.

23.3.3 Jobs using Power on Demand (PoD) licences

You can then start the STAR-CCM+ server on the compute nodes. The following script starts the server:

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N STAR-CCM_test#PBS -l select=1008#PBS -l walltime=02:00:00#PBS -l place=excl#PBS -k oe


92 Chapter 23. STAR-CCM+


# Change to the directory yhat the job was submitted fromcd $PBS_O_WORKDIR# Load any required modulesmodule load mptmodule load starccm+

export SGI_MPI_HOME=$MPI_ROOTexport PATH=$STARCCM_EXE:$PATHexport [email protected] [email protected]

uniq $PBS_NODEFILE | cut -d . -f 1 > ~/starccm.launcher.host.txtstarccm+ -power -podkey <PODkey> -licpath [email protected] -server -machinefile ~/→˓starccm.launcher.host.txt -np 504 -rsh ssh -port 42333

You should replace “<PODkey>” with your PoD licence key.

23.3.4 Local Star-CCM+ client configuration

Start your local STAR-CCM+ application and connect to your server. Click on the File -> “Connect to Server. . . ”option and use the following settings:

• Host: name of first Cirrus compute node (use ‘qtsat’, e.g. r1i0n32)

• Port: the number that you specified in the submission script

Select the “Connect through SSH tunnel” option and use:

• SSH Tunnel Host: cirrus-login0.epcc.ed.ac.uk

• SSH Tunnel Host Username: use your Cirrus username

• SSH Options: -agent

Your local STAR-CCM+ client should now be connected to the remote server. You should be able to run a newsimulation or interact with an existing one.

23.3. Using STAR-CCM+ on Cirrus: Interactive remote GUI Mode 93


94 Chapter 23. STAR-CCM+

CHAPTER 24

VASP

The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g.electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

VASP computes an approximate solution to the many-body Schrödinger equation, either within density functional the-ory (DFT), solving the Kohn-Sham equations, or within the Hartree-Fock (HF) approximation, solving the Roothaanequations. Hybrid functionals that mix the Hartree-Fock approach with density functional theory are implemented aswell. Furthermore, Green’s functions methods (GW quasiparticles, and ACFDT-RPA) and many-body perturbationtheory (2nd-order Møller-Plesset) are available in VASP.

In VASP, central quantities, like the one-electron orbitals, the electronic charge density, and the local potential are ex-pressed in plane wave basis sets. The interactions between the electrons and ions are described using norm-conservingor ultrasoft pseudopotentials, or the projector-augmented-wave method.

To determine the electronic groundstate, VASP makes use of efficient iterative matrix diagonalisation techniques, likethe residual minimisation method with direct inversion of the iterative subspace (RMM-DIIS) or blocked Davidsonalgorithms. These are coupled to highly efficient Broyden and Pulay density mixing schemes to speed up the self-consistency cycle.

24.1 Useful Links

• VASP Manual

• VASP Licensing

24.2 Using VASP on Cirrus

VASP is only available to users who have a valid VASP licence.

If you have a VASP licence and wish to have access to VASP on Cirrus please contact the Cirrus Helpdesk.

95

http://www.vasp.at

http://cms.mpi.univie.ac.at/vasp/vasp/vasp.html

http://www.vasp.at/index.php/faqs/71-how-can-i-purchase-a-vasp-license



24.3 Running parallel VASP jobs

VASP can exploit multiple nodes on Cirrus and will generally be run in exclusive mode over more than one node.

To access VASP you should load the vasp module in your job submission scripts:

module add vasp

Once loaded, the executables are called:

• vasp_std - Multiple k-point version

• vasp_gam - GAMMA-point only version

• vasp_ncl - Non-collinear version

All 5.4.* executables include the additional MD algorithms accessed via the MDALGO keyword.

You can access the LDA and PBE pseudopotentials for VASP on Cirrus at:

/lustre/home/y07/vasp5/5.4.4-intel17-mpt214/pot

The following script will run a VASP job using 4 nodes (144 cores).

#!/bin/bash --login

# PBS job options (name, compute nodes, job time)#PBS -N VASP_test#PBS -l select=4:ncpus=72#PBS -l place=excl#PBS -l walltime=0:20:0



# Load VASP modulemodule load vasp

# Run standard VASP executable# Note: '-ppn 36' is required to use all physical cores across# nodes as hyperthreading is enabled by defaultmpiexec_mpt -n 144 -ppn 36 vasp_std

96 Chapter 24. VASP

CHAPTER 25

Intel MKL: BLAS, LAPACK, ScaLAPACK

The Intel MKL libraries contain a variety of optimised numerical libraries including BLAS, LAPACK, and ScaLA-PACK.

25.1 Intel Compilers

25.1.1 BLAS and LAPACK

To use MKL libraries with the Intel compilers you first need to load the Intel compiler module and the Intel toolsmodule:

module load intel-compilers-17module load intel-tools-17

To include MKL you specify the -mkl option on your compile and link lines. For example, to compile a single sourcefile, Fortran program with MKL you could use:

ifort -c -mkl -o lapack_prb.o lapack_prb.f90ifort -mkl -o lapack_prb.x lapack_prb.o

The -mkl flag without any options builds against the threaded version of MKL. If you wish to build against the serialversion of MKL, you would use -mkl=sequential.

25.1.2 ScaLAPACK

The distributed memory linear algebra routines in ScaLAPACK require MPI in addition to the compilers and MKLlibraries. On Cirrus, this is usually provided by SGI MPT.

module load intel-compilers-17module load intel-tools-17module load mpt

97


Once you have the modules loaded you need to use the SGI versions of BLACS at link time to include ScaLAPACK.Remember to use the MPI versions of the compilers:

mpif90 -c -o linsolve.o linsolve.f90mpif90 -o linsolve.x linsolve.o -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_→˓intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_sgimpt_lp64 -lpthread -lm -ldl

25.2 GNU Compiler

25.2.1 BLAS and LAPACK

To use MKL libraries with the GNU compiler you first need to load the GNU compiler module and Intel tools module:

module load gccmodule load intel-tools-16

To include MKL you need to explicitly link against the MKL libraries. For example, to compile a single source file,Fortran program with MKL you could use:

gfortran -c -o lapack_prb.o lapack_prb.f90gfortran -o lapack_prb.x lapack_prb.o -L$MKLROOT/lib/intel64 -lmkl_gf_lp64 -lmkl_core→˓-lmkl_sequential

This will build against the serial version of MKL, to build against the threaded version use:

gfortran -c -o lapack_prb.o lapack_prb.f90gfortran -fopenmp -o lapack_prb.x lapack_prb.o -L$MKLROOT/lib/intel64 -lmkl_gf_lp64 -→˓lmkl_core -lmkl_gnu_thread

25.2.2 ScaLAPACK

The distributed memory linear algebra routines in ScaLAPACK require MPI in addition to the compilers and MKLlibraries. On Cirrus, this is usually provided by SGI MPT.

module load gccmodule load intel-tools-16module load mpt

Once you have the modules loaded you need to link against two additional libraries to include ScaLAPACK. Rememberto use the MPI versions of the compilers:

mpif90 -f90=gfortran -c -o linsolve.o linsolve.f90mpif90 -f90=gfortran -o linsolve.x linsolve.o -L${MKLROOT}/lib/intel64 -lmkl_→˓scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_sgimpt_lp64→˓-lpthread -lm -ldl

25.2.3 ILP vs LP libraries

If you look in the $MKLROOT/lib/intel64 directory then you will see ILP and LP libraries, in the above we werelinking against the LP libraries and you can choose either. ILP use a 64 bit integer type, whereas LP use a 32 bitinteger type. For very large arrays then ILP is the best choice (as it can index far more data), but there are somelimitations. For more information see the Intel documentation here.

98 Chapter 25. Intel MKL: BLAS, LAPACK, ScaLAPACK

https://software.intel.com/en-us/node/528682

CHAPTER 26

Debugging using Allinea DDT

Allinea’s Forge tool suite is installed on Cirrus and DDT, which is a debugging tool for scalar, multi-threaded and large-scale parallel applications, is available for use in debugging your codes. To compile your code for debugging you willusually want to specify the -O0 option to turn off all code optimisation (as this can produce a mismatch between sourcecode line numbers and debugging information) and -g to include debugging information in the compiled executable.To use this package you will need to load the Allinea Forge module and execute forge:

module load allineaforge

26.1 Debugging runs on the login nodes

You can execute and debug your MPI code on the login node which is useful for immediate development work withshort, simple runs to avoid having to wait in the queue. Firstly ensure you have loaded the mpt library, then start forgeand click Run. Fill in the nescesary details of your code under the application pane, then check the MPI tick box,specify the number of MPI processes you wish to run and ensure implementation is set to SGI MPT (2.10+). If this isnot set correctly then you can update the configuration via clicking the Change button and selecting this option on theMPI/UPC Implementation field of the system pane. When you are happy with this hit Run to start.

26.2 Debugging runs on the compute nodes

This involves DDT submitting your job to the queue, this then running and as soon as the compute nodes start executingyou will drop into the debug session and be able to interact with your code. Similarly to running on the login node,fill in details of your application and ensure that MPI is ticked. But now change the implementation from SGI MPT(2.10+) to SGI MPT (2.10+, batch) as we are running via the batch system. Then check the Submit to Queue tickbox and click the Configure button. In the settings window that pops up you can specify the submission template,one has been prepared one for Cirrus at /lustre/sw/allinea/forge-7.0.0/templates/cirrus.qtfwhich we suggest you use - although you are very free to chose another one and/or specialise this as you require.Back on the run page, click the Parameters button and fill in the maximum wallclock time, the budget to charge to andthe total number of virtual cores required which determine the number of nodes and are provided as an argument to

99


the -l select= PBS option. Back on the run dialog ensure look at the MPI pane, ensure the number of processes andprocesses per node settings are correct and then hit Submit.

26.3 Memory debugging with DDT

If you are dynamically linking your code and debugging it on the login node then this is fine (just ensure that thePreload the memory debugging library option is ticked in the Details pane.) If you are dynamically linking but intend-ing to debug running on the compute nodes, or statically linking then you need to include the compile option -Wl,--allow-multiple-definition and explicitly link your executable with Allinea’s memory debugging library.The exactly library to link against depends on your code; -ldmalloc (for no threading with C), -ldmallocth(for threading with C), -ldmallocxx (for no threading with C++) or -ldmallocthcxx (for threading with C++).The library locations are all set up when the allinea module is loaded so these libraries should be found without furtherarguments.

26.4 Getting further help on DDT

• DDT website

• DDT support page

• DDT user guide

100 Chapter 26. Debugging using Allinea DDT

http://www.allinea.com/products/ddt/

https://www.allinea.com/get-support

https://www.allinea.com/user-guide/forge/userguide.html

CHAPTER 27

Transferring data from INDY to Cirrus

This document outlines methods and tips for transferring data from INDY to Cirrus for users who are migrating fromthe old INDY-linux system to the new Cirrus system.

Note: This guide is not intended for users of the INDY-windows service. Different arrangements will be made forINDY Windows HPC users.

If you have any questions about transferring data from INDY-linux to Cirrus then please contact the helpdesk

27.1 Introduction

As INDY-linux and Cirrus are both colocated at EPCC’s ACF data centre the most efficient way to transfer databetween them is directly using SSH. This will use the high-bandwidth data links between the two systems at the ACF.

A two stange transfer process via your home site will generally see worse performance than direct transfer. However,this approach may be useful if you want to take a local, backup copy of your data during the move.

Note: Unlike INDY-linux, the Cirrus file systems are currently not backed-up in any way so we strongly recommendthat you take a copy of any critical data to a local resource before the end of the INDY-linux service.

27.2 Minimum Requirements

In order to transfer data from INDY-linux to Cirrus using a direct copy over SSH you will need the following:

• Access to your account on INDY-linux

• An account on Cirrus with enough disk quota to hold the data you are transferring

• A SSH client application that you can use to log into Cirrus or INDY-linux

101



27.3 Data Transfer via SSH

The easiest way of transferring data to Cirrus from INDY-linux is to use one of the standard programs based on theSSH protocol such as scp, sftp or rsync. These all use the same underlying mechanism (ssh) as you normallyuse to log-in to Cirrus/INDY-linux. So, once the the command has been executed via the command line, you will beprompted for your password for the specified account on the remote machine.

To avoid having to type in your password multiple times you can set up a ssh-key as documented in the User Guide atConnecting to Cirrus

27.3.1 SSH Transfer Performance Considerations

The ssh command encrypts all traffic it sends. This means that file-transfer using ssh consumes a relatively largeamount of cpu time at both ends of the transfer. The login nodes for Cirrus and INDY-linux have fairly fast processorsthat can sustain about 100 MB/s transfer. The encryption algorithm used is negotiated between the ssh-client and thessh-server. There are command line flags that allow you to specify a preference for which encryption algorithm shouldbe used. You may be able to improve transfer speeds by reqeusting a different algorithm than the default. The arcfouralgorithm is usually quite fast if both hosts support it.

A single ssh based transfer will usually not be able to saturate the available network bandwidth or the available diskbandwidth so you may see an overall improvement by running several data transfer operations in parallel. To reducemetadata interactions it is a good idea to overlap transfers of files from different directories.

In addition, you should consider the following when transferring data:

• Only transfer those files that are required. Consider which data you really need to keep.

• Combine lots of small files into a single tar file, to reduce the overheads associated in initiating many separatedata transfers (over SSH each file counts as an individual transfer).

• Compress data before sending it, e.g. using gzip.

27.3.2 scp command

The scp command creates a copy of a file, or if given the -r flag a directory, on a remote machine.

For example, to transfer files from INDY-linux to Cirrus (assuming you are logged into INDY-linux):

scp [options] source [email protected]:[destination]

(Remember to replace user with your Cirrus username in the example above.)

In the above example, the [destination] is optional, as when left out scp will simply copy the source into theuser’s home directory. Also the source should be the absolute path of the file/directory being copied or the commandshould be executed in the directory containing the source file/directory.

If you want to request a different encryption algorithm add the -c [algorithm-name] flag to the scp options.For example, to use the (usually faster) arcfour encryption algorithm you would use:

scp [options] -c arcfour source [email protected]:[destination]


102 Chapter 27. Transferring data from INDY to Cirrus


27.3.3 rsync command

The rsync command can also transfer data between hosts using a ssh connection. It creates a copy of a file or, ifgiven the -r flag, a directory at the given destination, similar to scp above.

Given the -a option rsync can also make exact copies (including permissions), this is referred to as mirroring. In thiscase the rsync command is executed with ssh to create the copy on a remote machine.

To transfer files to Cirrus from INDY-linux using rsync (assuming you are logged into INDY-linx) the commandshould have the form:

rsync [options] -e ssh source [email protected]:[destination]


In the above example, the [destination] is optional, as when left out rsync will simply copy the source into theusers home directory. Also the source should be the absolute path of the file/directory being copied or the commandshould be executed in the directory containing the source file/directory.

Additional flags can be specified for the underlying ssh command by using a quoted string as the argument of the -eflag. e.g.

rsync [options] -e "ssh -c arcfour" source [email protected]:[destination]


27.3. Data Transfer via SSH 103

Cirrus Documentation - Read the Docs · PDF file17.2 Using LAMMPS on Cirrus ... A typical...

Documents

Transcript of Cirrus Documentation - Read the Docs · PDF file17.2 Using LAMMPS on Cirrus ... A typical...