CamGrid

Mark Calleja

Cambridge eScience Centre

What is it?• A number of like minded groups and departments

(10), each running their own Condor pool(s), which federate their resources (12).

• Coordinated by the Cambridge eScience Centre (CeSC), but no overall control.

• Been running now for ~2.5 years, ~70+ users.• Currently have ~950 processors/cores available.• “All” linux (various), mostly x86_64, running 24/7.• Mostly Dell PowerEdge 1950 (like HPCF), four

cores with 8GB.• Around 2M CPU hours to date.

Some details

• Pools run the latest stable version of Condor (currently 6.8.6).

• All machines get an (extra) IP address in a CUDN-only routeable range for Condor.

• Each pool sets its own policies, but these must be visible to other users of CamGrid.

• Currently we see vanilla, standard and parallel (MPI) universe jobs.

• Users get accounts on a machine in their local pool; jobs are then distributed around the grid by Condor using its flocking mechanism.

• MPI jobs on single SMP machines have proved very useful.

NTE of Ag3[Co(CN)6] with SMP/MPI sweep

Monitoring Tools

• A number of web based tools provided to monitor the state of the grid and of jobs.

• CamGrid is based on trust, so must make sure that machines are fairly configured.

• The university gave us £450k (~$950k) to buy new hardware; need to ensure that it’s online as promised.

CamGrid’s file viewer

• Standard universe uses RPCs to echo I/O operations back to submit host.

• What about other universes? How can I check the health of my long running simulation?

• We’ve provided our own facility, which involves an agent installed on each execute node and accessed via a web interface.

• Works with vanilla and parallel (MPI) jobs.

• Requires local sysadmins to install and run it.

CamGrid’s file viewer

Checkpointable vanilla universe

• Standard universe is fine, if you can link to Condor’s libraries (Pete Keller – “getting harder”).

• Investigating using BLCR (Berkeley Lab Checkpoint/Restart) kernel modules for linux.

• Uses kernel resources, and can thus restore resources that user-level libraries cannot.

• Supported by some flavours of MPI (late LAM, OpenMPI).

• The idea was to use Parrot’s user-space FS to wrap a vanilla job and save the job’s state on a chirp server.

• However, currently Parrot breaks some BLCR functionality.

What doesn’t work so well…

• Each pool is run by local sysadmin(s), but these are of variable quality/commitment.

• We’ve set up mailing lists for users and sysadmins: hardly ever used (don’t want to advertise ignorance?).

• Some pools have used SRIF hardware to redeploy machines committed earlier. Naughty…

• Don’t get me started on merger with UCS’s central resource (~400 nodes).

But generally we’re happy bunnies

• “CamGrid was an invaluable tool allowing us to reliably sample the large parameter space in a reasonable amount of time. A half-year's worth of CPU running was collected in a week."

-- Dr. Ben Allanach

• “CamGrid was essential in order for us to be able to run the different codes in real time.”

-- Prof. Fernando Quevedo

• “I needed to run simulations that took a couple of weeks each. Without access to the processors on CamGrid, it would have taken a couple of years to get enough results for a publication.“

-- Dr. Karen Lipkow

Current issues• Protecting resources on execute nodes; Condor

seems lax at this, e.g. memory, disk space. • Increasingly interested in VMs (i.e. Xen). Some

pools run it, but not concerted (effects on SMP MPI jobs?).

• Green issues: will we be forced to buy WoL cards in the near future?

• Altruistic computing: a recent wave of interest for BOINC/backfill jobs for medical, protein folding, etc., but who runs the jobs? Audit trail?

• How do we interact with outsiders? Ideally keep it to Condor (some Globus, toyed with VPNs). Most CamGrid stakeholders just dish out conventional, ssh-accessible accounts.

Finally…

• CamGrid:http://www.escience.cam.ac.uk/projects/camgrid/

• Contact:mc321@cam.ac.uk

Questions?