Support for Computational Science and Engineering (CSE)

Experts in numerical algorithms and High Performance Computing services

Support for Computational Science & Engineering

SeIUCCR Summer School

STFC Daresbury

August 2013

Andrew Jones

Vice-President, HPC Expertise & Services

Numerical Algorithms Group (NAG)

2 HPC software engineering services | HPC consulting and training | numerical computing products and expertise

Summary

Computational Science & Engineering Support

A supercomputing roadmap

The UK National Supercomputing Service

Introduction to NAG


Delivering Trusted Numerical & HPC Expertise

Numerical Computing

• High quality maths libraries and compiler for wide range of platforms & languages

• Practical expertise spanning hardware (CPU/GPU/MIC/…), software (languages, etc.), algorithms, …

• Consulting & Numerical Solutions

HPC Consulting & Services

• HPC application development: performance, scalability, algorithms, …

• Training, advice and support for HPC users, developers and procurement

• Technology evaluation & strategy

• International provider of HPC expertise to industry, academia & government


Comprehensive CSE Support Service

including Training

CSE Support, HPC Training, …

CSE Support, HPC Training, …

Commercial, e.g., Oil & Gas, …

Procurements: UK, Australia,

South Africa, …

HPC R&D: MIC, GPU, scalability, tools,

algorithms, …


THE NATIONAL SUPERCOMPUTING SERVICE


97 98 99 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 …

ARCHER

A Research Councils UK High End Computing Service


2007-2014, £100M+

UK Government (BIS) acting through

EPSRC

on behalf of

Research Councils UK and the academic

community

Procurement & Project Management

Cray Supercomputers

NAG Computational Science & Engineering Support and Training

EPCC Service coordination, datacentre & system management


Phase3: Cray XE6

• 827 TF peak performance

• 90,000 cores (AMD Interlagos)

• 32 cores & 32GB per node

• 1 PB disk, 8PB Tertiary Storage


HECToR Technology Roadmap

XT4: 60TF

XT4 208TF

XT6->XE6 (Gemini)

XE6 827TF

2007

Cap

abili

ty

2008 2009 2010 2011 2012 2013

XT6/XE6 374TF

11320 cores 22656 cores 44544 cores

2.8GHz Santa Rosa 2 cores & 6GB per node

2.3GHz Barcelona 4 cores & 8 GB per node

2.1GHz Magny-Cours 24 cores & 32GB per node

89856 cores

2.3GHz Interlagos 32 cores & 32GB per node

XT4 113TF


A SUPERCOMPUTING ROADMAP


Peak DP GF Clock GHz DP FLOPS parallel Total parallel

Opteron (Abu Dhabi) 160 2.5 8fpu x 8ops 64

Xeon (Sandy Bridge EP) 166 2.6 8fpu x 8ops 64

Xeon Phi (Knights Corner) 1,074 1.1 61c x 16ops 976

Tesla GPU (Kepler K20x) 1,312 0.7 14sm x 64c x 2ops 1,792

FirePro GPU (S10k Tahiti) 1,478 0.8 2gpu x 28cu x 4v x 8ops? 1,792


2.3GHz processor 8 FPU per processor 8 FLOPs/clock (vector) per FPU

Theoretical Peak Performance (GFLOPS)

Against Maximum

multi-threaded and vectorized 147.2 100.00%

2.3GHz processor 8 FPU per processor 8 FLOPs/clock (vector) per FPU


Against Maximum


multi-threaded & not vectorized 18.4 12.5%

serial and vectorized 18.4 12.5%

2.6 GHz processor 8 cores (FPU) per processor 8 DP FLOPs/cycle per FPU


Against Maximum


multi-threaded & not vectorized 20.8 12.5%

serial and vectorized 20.8 12.5%

serial and not vectorized 2.6 1.6%

>98% of the possible FLOPS performance comes from parallel processing techniques


Packaged Solutions and/or Expertise Use parallelized ISV or lib Parallelize your own software

Scalable software methods

Parallel processing hardware/techniques

Need for more performance


Enhanced modeling capability (application software)

Performance Scalability Algorithms Functionality

Better science & engineering capability

Better models of real world

New regions of parameter space

Multi-scale & multi-physics

Understanding of simulation validity


HECTOR CSE SUPPORT


CSE Support Service ~20 persons/year

Short support projects

porting tuning testing

…

Training courses

Fortran, MPI, Software

Engineering, CUDA, …

Focused software development projects (“dCSE”)

6-48 months

performance scalability algorithms

…

embedded -> sustainability

& dissemination

25

NAG HECToR CSE Team

Core Team

dCSE1

dCSE2

dCSE3

dCSE4 dCSE5

dCSE …

dCSEn

Core Team (~8)

Support calls

Short-term CSE support

Training

Distributed Teams (~12)

Dedicated science-led CSE projects

Embedded staff

Build the skill base

Knowledge transfer key


Result: Originally limited to solving 100k grid points in days, the code can now solve 100k grid points in a few minutes, or can now model >50M grid points using hundreds of multicore nodes on supercomputers.

Solution: NAG dCSE project to implement a distributed parallel version of the application using MPI and OpenMP.

Problem: Studying aircraft noise modelling using high fidelity LES methods. Essential to capture wing-flap, free jet and wing-jet-flap interactions. This requires HPC – but current application is desktop based.


RESULT: Current model sizes (0.5B points) run up to 6x faster by using 8000 instead of 512 cores, and model sizes beyond 60B points now possible.

SOLUTION: NAG dCSE project to introduce 2D decomposition to improve scalability, along with other performance improvements.

PROBLEM: Need to reduce runtime for current sized models and to enable simulation of much larger models.


Result: Net performance improvements of a factor of 20 for key scientific study (egg shell formation), plus multiple order-of-magnitude performance improvements in data file I/O aspects of the code.

Solution: NAG dCSE project to improve distributed data management and implement efficient parallel I/O.

Problem: Studying proteins for egg shell formation using a proven HPC molecular dynamics code on supercomputers. Scalability and performance limited by I/O bottlenecks.


RESULT: The code can now scale to over 50x as many cores, enabling much bigger models to be run with more realistic chemistry, and more efficient monitoring of the simulation. Now possible to study more types of fuels.

SOLUTION: NAG dCSE project to improve the scalability of the code by adding 2D decomposition; to optimize the I/O for monitoring the physical quantities; and refactor the legacy F77 code for better sustainability.

PROBLEM: Need to solver larger and more complex models, with more realistic chemistry, while monitoring the evolution of key physical quantities


Materials Science 4x scale

Oceanography faster

Quantum Monte-Carlo

4x performance

Materials Science improved speed and scalability

Atmospheric Chemistry

faster performance

Geodynamic Thermal

Convection enhanced

Fluid Turbulence 40x scalability

Catalytic Chemistry 8x faster

Ocean Modelling dramatically

improved scalability

Molecular Dynamics 20x faster

Heart Modeling 20x

performance

Turbulent Flow 6x performance

Materials Science enhanced

capabilities

Aircraft Noise >100x

models/speed

Combustion 50x model size

DNS 100x model size


Materials Science running at

one quarter of its potential scale

Oceanography limited speed

and I/O performance

Quantum Monte-Carlo

limited to 25% of performance

Materials Science limited speed

and scalability

Atmospheric Chemistry restricted

performance

Geodynamic Thermal

Convection restricted

Fluid Turbulence 40x less than

possible scalability

Catalytic Chemistry 8x slower

than possible

Ocean Modelling dramatically

limited scalability

Molecular Dynamics

20x slower than possible

Heart Modelling 20x less than

possible performance

Turbulent Fluid Flow

restricted by factor of 6x

Materials Science limited

performance and capabilities

Aircraft Noise slower and

limited to smaller models

Combustion limited to 1/50

model size

DNS limited to 1/100

model size


CFD, Materials Science,

Physics, … expertise from

the users or customers

Expertise in HPC software engineering,

performance & algorithms


Ease of use? Efficiency of use?

47

Training Courses

HECToR Specific

Introduction to HECToR

Transitioning to the XE6

Programming

Fortran 95

MPI, OpenMP

CAF, UPC

CUDA, OpenCL

Parallel I/O

Other

Debugging, Profiling and Optimisation

Multicore

Core Algorithms for High Performance Scientific Computing

Best Practice in HPC Software Development

Application Specific

R, DL_POLY, CASTEP


The single most important part of e-infrastructure

People!


SUMMARY


Best science output from e-infrastructure (esp. HPC) critically dependent on software innovation (CSE)

£80m supercomputer + £20m CSE investment delivers much more science than £100m supercomputer alone

NAG now also delivering this HPC expertise to industry and other HPC centres internationally


EXPERTS IN HPC & NUMERICAL COMPUTING

Results Matter. Trust NAG.

www.nag.com | blog.nag.com | @NAGtalk

@hpcnotes (Andrew Jones)

Support for Computational Science and Engineering (CSE)

Documents

Transcript of Support for Computational Science and Engineering (CSE)