NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

1

Scaling Up User Codes on the SP

David Skinner, NERSC Division, Berkeley Lab

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

2

Motivation

• NERSC’s focus is on capability computation– Capability == jobs that use ¼ or more of the machines resources

• Scientists whose work involves large scale computation or HPC should keep ahead of workstation sized problems

• “Big Science” problems are more interesting!

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

3

Challenges

• CPU’s are outpacing memory bandwidth and switches, leaving FLOPs increasingly isolated.

• Vendors often have machines < ½ the size of NERSC machines: system software may be operating in uncharted regimes– MPI implementation– Filesystem metadata systems– Batch queue system

Users need information on how to mitigate the impact of these issues for large concurrency applications.

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

4

Seaborg.nersc.gov

MP_EUIDEVICE(switch fabric)

MPI Bandwidth(MB/sec)

MPI Latency(usec)

css0 500 / 350 8 / 16

css1

csss 500 / 350(single task)

8 / 16

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

5

Switch Adapater Performance

csss

css0

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

6

Switch considerations

• For data decomposed applications with some locality partition problem along SMP boundaries (minimize surface to volume ratio)

• Use MP_SHAREDMEMORY to minimize switch traffic

• csss is most often the best route to the switch

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

7

Synchronization

• On the SP each SMP image is scheduled independently and while use code is waiting, OS will schedule other tasks

• A fully synchronizing MPI call requires everyone’s attention

• By analogy, imagine trying to go to lunch with 1024 people

• Probability that everyone is ready at any given time scales poorly

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

8

Synchronization (continued)

• MPI_Alltoall and MPI_Allreduce can be particularly bad in the range of 512 tasks and above

• Use MPI_Broadcast if possible – Not fully synchronizing

• Remove un-needed MPI_Barrier calls

• Use Asynchronous I/O when possible

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

9

Load Balance

• If one task lags the others in time to complete synchronization suffers, e.g. a 3% slowdown in one task can mean a 50% slowdown for the code overall

• Seek out and eliminate sources of variation• Distribute problem uniformly among nodes/cpus

0 20 40 60 80 100

0

1

2

3 FLOPI/OSYNCFLOPI/OSYNC

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

10

Alternatives to MPI

• CHARM++ and NAMD – Spatially decomposed molecular dynamics with periodic load balancing, data decomposition is adaptive

• AMPI http://charm.cs.uiuc.edu/– An automatic approach to load balancing

• BlueGene L type machines with > 10K cpus will need re-examine these issues altogether

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

11

Improving MPI Scaling on Seaborg

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

12

The SP switch

• Use MP_SHAREDMEMORY=yes (default)

• Use MP_EUIDEVICE=csss for 32 bit applications (default)

• Run /usr/common/usg/bin/phost prior to your parallel program to map machine names to POE tasks– MPI and LAPI versions available– Hostslists are useful in general

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

13

64 bit MPI

• 32 bit MPI has inconvenient memory limits – 256MB per task default and 2GB maximum– 1.7GB can be used in practice, but depends on MPI usage – The scaling of this internal usage is complicated, but larger

concurrency jobs have more of their memory “stolen” by MPI’s internal buffers and pipes

• 64 bit MPI removes these barriers– But must run on css0 only, less switch bandwidth

• Seaborg has 16,32, and 64 GB per node available

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

14

64 bit MPI Howto

At compile time:

* module load mpi64 * compile with the "-q64" option using mpcc_r, mpxlf_r, or mpxlf90_r.

At run time:

* module load mpi64 * use "#@ network.MPI = css0,us,shared" in your job scripts. The multilink adapter "csss" is not currently supported. * run your POE code as you normally would

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

15

MP_LABELIO, phost

• Labeled I/O will let you know which task generated the message “segmentation fault” , gave wrong answer, etc.

export MP_LABELIO=yes

• Run /usr/common/usg/bin/phost prior to your parallel program to map machine names to POE tasks– MPI and LAPI versions available– Hostslists are useful in general

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

16

Core files

• Core dumps don’t scale (no parallel work)

• MP_COREDIR=/dev/null No corefile I/O• MP_COREFILE_FORMAT=light_core Less I/O• LL script to save just one full fledged core file, throw away

others … if MP_CHILD !=0 export MP_COREDIR=/dev/nullendif…

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

17

Debugging

• In general debugging 512 and above is error prone and cumbersome.

• Debug at a smaller scale when possible.

• Use shared memory device MPICH on a workstation with lots of memory to simulate 1024 cpus.

• For crashed jobs examine LL logs for memory usage history.

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

18

Parallel I/O

• Can be a significant source of variation in task completion prior to synchronization

• Limit the number of readers or writers when appropriate. Pay attention to file creation rates.

• Output reduced quantities when possible

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

19

OpenMP

• Using a mixed model, even when no underlying fine grained parallelism is present can take strain off of the MPI implementation,

e.g. on seaborg a 2048 way job can run with only 128 MPI tasks and 16 OpenMP threads

• Having hybrid code whose concurrencies can be tuned between MPI and OpenMP tasks has portability advantages

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

20

Summary

• Resources are present to face the challenges posed by scaling up MPI applications on seaborg.

• Scientists should expand their problem scopes to tackle increasingly challenging computational problems.

• NERSC consultants can provide help in achieving scaling goals.

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

21

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

22

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

23

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

24