BoM Modelling and Computing Update

1

BoM Modelling and Computing Update

Michael NaughtonBMRC

Acknowlegements: Bill Bourke & BMRC colleaguesPhil Tannenbaum & HPCCC colleaguesNEC/A applications support staff

• BoM/CSIRO HPCCC Supercomputer Upgrade• New BoM Head Office• MPI versions of BoM major applications codes• BoM Modelling

2

BoM High Performance Computing from 1982 to 1996

• FACOM • CSIRO CYBER205• ETA-10• CRAY X-MP, Y-MP

BoM/CSIRO COMPUTERS PEAK PERFORMANCE

0.001

0.01

0.1

1

10

100

1000

100001982 1987 1992 1997 2002 2007 2012

Years

Peak

Gflo

ps

3

BoM/CSIRO HPCCC

• High Performance Computing and Communications Centre– established 1997 as joint high performance computing

facility– 50:50 BoM-CSIRO partnership 1997-2003

• NEC SX-4/16 1997• NEC SX-4/32 1997-2000• NEC SX-4/32 & SX-5/16 2000-2001• 2 X NEC SX-5/16 2001-2003

– Continued BoM-CSIRO partnership 2003-2007

4




0.001

0.01

0.1

1

10

100

1000

100001982 1987 1992 1997 2002 2007 2012

Years

Peak

Gflo

ps

5



• NEC SX-4, SX-5


0.001

0.01

0.1

1

10

100

1000

100001982 1987 1992 1997 2002 2007 2012

Years

Peak

Gflo

ps

6

1. BoM Supercomputer Upgrade Timelines

• MPI development versions of BoM major applications: 2000-2002• Pre-tender benchmarks release: August 2002• RFT: October 2002• Tender deadline: February 2003

7

BoM Upgrade Phases

• 4+2+2 Phases

• Initial 4 year contract: 4 years from acceptance

• First optional contract extension: 2008-2009

• Second optional contract extension: 2010-2011

RFT Sustained Gflops Targetsfor BoM Operational Applications

10

100

1000

10000

Jan-02 Jan-04 Jan-06 Jan-08 Jan-10 Jan-12

Time

Gflo

ps

BoM SX-5Initial 4-year periodExtension period

• RFT Performance Target: 150 / 300 / 750 / 2000 Gflops RFT Performance Target: 150 / 300 / 750 / 2000 Gflops sustained performance on BoM operational applications (cf. 35 sustained performance on BoM operational applications (cf. 35 Gflops on BoM SX-5)Gflops on BoM SX-5)

8

BoM Supercomputer Upgrade Timelines

• MPI development versions of BoM major applications: 2000-2002• Pre-tender benchmarks release: August 2002• RFT: October 2002• Tender deadline: February 2003

• NEC selected as preferred vendor: April 2003• Contract signed with NEC: June 2003 • Interim 2-node SX-6 machine for porting: September 2003• Delivery: December 2003• Installation and acceptance testing: Dec-Feb• Operational: March 2004

9



• NEC SX-4, SX-5


0.001

0.01

0.1

1

10

100

1000

100001982 1987 1992 1997 2002 2007 2012

Years

Peak

Gflo

ps

10



• NEC SX-4, SX-5

• NEC SX-6


0.001

0.01

0.1

1

10

100

1000

100001982 1987 1992 1997 2002 2007 2012

Years

Peak

Gflo

ps

11



• NEC SX-4, SX-5

• NEC SX-6

Slope: approx. factor 2.5 every 2 years


0.001

0.01

0.1

1

10

100

1000

100001982 1987 1992 1997 2002 2007 2012

Years

Peak

Gflo

ps

12

New HPCCC Facility Summary

• Initial installation: 4Q 2003 – on availability of new CCF– 18 node NEC SX-6 (SX-6/144M18)

• multi-node vector processor based architecture• 8 CPUs x 8 Gfl peak performance per node• 1152 Gfl peak capacity

– 2 x 12 CPU TX-7 Linux front-end• 1.3 GHz 64-bit Itanium processor• GFS server with failover• ~ 14 TB GFS disk storage• cross-compilation environment

13

New HPCCC Facility Summary

• Upgrade: 4Q 2004– 28 node NEC SX-6

(SX-6/224M28)– 1792 Gfl peak capacity– 2 x 16 CPU TX-7 front-end– ~ 22 TB GFS disk storage

• Optional Extension 1: Jan 2008• Optional Extension 2: Jan 2010

• Initial: Dec 2003– 18 node NEC SX-6

(SX-6/224M18)– 1152 Gfl peak capacity– 2 x 12 CPU TX-7 front-end– ~ 14 TB GFS disk storage

14

SX-6 8 Node Chassis

15

Comparisons Between SX-5 and SX-6

• SX-5 Processors– 16 CPU per Cabinet– 4, 8, 10 GFLOPS Flavours– Long Vector Tuned

• 200+• SX-5 Memory

– SDRAM– 256 GB per Cabinet (128)

• IOP– 32 bit PCI

• SUPER-UX OS

• SX-6 Processors– 8 CPU per Cabinet– 8 GFLOPS Flavour– Short Vector Tuned

• 32+• SX-6 Memory

– DDR-SDRAM– 64 GB per Cabinet

• IOP– 64 bit PCI

• SUPER-UX OS

16

SX-6 SX-6 ProcessorProcessor Architecture Architecture

XMU

SHA

RED

MA

IN M

EMO

RY

Scalar Registers

Logically 8 Wide Vector

Unit

Input/Output Processor (IOP)

Scalar Unit

Scalar Execution

Vector RegistersandVector Data Registers

CacheMemory

Load／ Store

Mask Reg. Mask

Logical

Multiply

Add/Shift

Divide

Central Processing Unit

17

HPCCCHPCCC Configuration Configuration

.......

8 CPU

....CPU CPU CPU

MM

....CPU CPU CPU

MM

....CPU CPU CPU

IOP IOP IOP IOP IOP IOP IOP IOP IOP IOP IOP IOP

Node #1 Node #2 Node #18

MM

8 GB/s Bidirectional

Internode Crossbar (IXS)

Initial 18 nodes CPU : 144 CPUs (1152 GFLOPS) MMU : 1152 GB (64 GB/node) Upgrade 28 Nodes CPU : 224 CPUs (1792 GFLOPS) MMU : 1792 GB (64 GB/node)

18

Data-Centric BoM CCF Forward Plan (2004-05)Data-Centric BoM CCF Forward Plan (2004-05)

HP-UXHP-UX

AIXAIX

SolarisSolaris

SUPER-UXSUPER-UX

LinuxLinux

StorageStorage

GFS / SANGFS / SAN

http://commerce.www.ibm.com/content/home/shop_ShopIBM/en_US/eServer/pSeries/high_end/690.html

https://www.hpc.comp.nec.co.jp/product/sx-6/

http://www.sun.com/

19

Scheduling

• Some SMP Nodes– Traditional Scheduling Strategy

• Some MPI Clusters– Huge Resources– Non Swapping Status– Similar Treatment to MPP

20

2. New BoM Head Office

Background behind move• 29 years at “Celsius House”, 150 Lonsdale St, Melbourne• Inadequacy of building for current and future CCF needs

– Floor loading problems for computers– Space limitation, esp. for dual operation during upgrades– Supercomputer CCF area: ~100 m2 to 1000 m2 – 25% reduction in individuals’ office space

• e.g. mine: 16 m2 to 12 m2

• Decision to remain in Melbourne • Decision to continue to have co-located CCF and HO

21

New BoM Head Office• Henry Hunt Building

– named after Australia’s first Commonwealth Meteorologist, 1907-1931

CCF

22

New BoM Head Office

23

New BoM Head Office

• 700 Collins Street, Docklands Precinct, Melbourne CBD

24

New BoM Head Office

25

New BoM Head Office

26

3. BoM Tender Benchmarks

• Target Performance– 150 Gfl on BoM Operational Applications

• Applications– GASP :: BoM Global Spectral NWP Model

» T479L50

– LAPS :: BoM Regional NWP Model» 8km, 800x600, 29 lvls

– ASSIM :: BoM MVSI Data Analysis» T479L50, 1780 analysis subvolumes

– CLIMAT :: BoM Global Spectral Climate Model» T95L50

– OCEAN :: BoM version of MOM-2 Ocean Model– clog :: I/O tester and I/O interference program

27

BoM Tender Benchmarks

• Best Effort calculations for each application, both – single run and – ensembles of several runs (to take account of possible

scaling limitations in applications)• Workload Test

– simulation of real-time combination of research jobs and pre-emptive, high priority operational suites.

• Data and timing started and finished on a front end server to reflect that network speed matters for delivery of applications results.

• MPI Developments for BoM Applications

28

MPI Version of Global Spectral Model (GASP)

• Initial development done during research visit by Dr Atsushi Kubota, Hiroshima City University

• Transpose method – grid point arrays distributed over latitude– spectral arrays distributed over zonal (Fourier) wavenumber– transpose Fourier coefficients from spectral to grid MPI

distributions and v.v. during each spectral transform – hybrid approach: combination of MPI and multitasking

parallelism

29

MPI Version of Global Spectral Model (GASP)

• Other relevant computational features– multi-pass strategy used to allow memory saving in storage

of grid point arrays; allowed T479L50 benchmark calculations to be developed and run on SX-5 during benchmark preparations

– "slab physics" drivers combine multiple latitudes together to achieve longer vectors (especially beneficial on SX-5 for climate applications, i.e. at lower resolutions than NWP)

• Eulerian dynamics MPI version completed, used in benchmarks• Semi-Lagrangian dynamics MPI version under development• 2-D decompositions possible within code architecture;

considering 2-D version for future scalability extension

30

MPI Version of Australian Region Model (LAPS)

• MPI distribution over latitudes• Semi-Lagrangian dynamics, semi-implicit timestepping MPI

version completed, used in tender benchmarks• Eulerian dynamics, explicit timestepping MPI version now

developed (corresponds to current operational LAPS)• Hybrid M/T and MPI

31

MPI Version of GenSI Data Analysis (ASSIM)

• MPI distribution over analysis subvolumes – approximately 1800 subvolumes over the globe

• Hybrid M/T and MPI

• Observation space assimilation scheme• Uses iterative method to calculate localised inverse

– use all obs that influence subvolume– update analysis for obs only inside subvolume– designed to support global and regional analysis systems

32

BoM Applications Performance on NEC SX-6

Sustained performance in optimum conditions• ~ 30% Peak on O(100) processors• 98% -- 99.7% scalable

• Expect ~ 80% of this performance for operational models on busy system

33


Single processor % of peak

• measure of vectorisation efficiency of BoM applications

SINGLE CPU % OF PEAK

0

10

20

30

40

50

60

70

GASP LAPS ASSIM CLIMAT OCEAN

34


Scalability of MPI versions

• 99.7% parallel on up to O(100) processors => potentially scalable to

O(300) processors.

• MPI and multitasked scaling results very similar.

AMDAHL SCALING

97

97.5

98

98.5

99

99.5

100

GASP LAPS ASSIM CLIMAT

35

Percentage of peak performance comparison between SX-4 & SX-5 & SX-6

• GASP global NWP model results• SX-4 and SX-6 deliver better % peak than SX-5

36

Performance scaling to 16 SX-6 nodes

TOTAL GFLOPS

0

50

100

150

200

250

300

350

400

0 16 32 48 64 80 96 112 128

Number of CPUs

GASP LAPSASSIM

PERCENTAGE OF PEAK

0

10

20

30

40

50

60

70

0 16 32 48 64 80 96 112 128

Number of CPUs

GASP LAPSASSIM

37

Capability vs Capacity

• Attempt to balance needs for high peak capability for operational systems together with good throughput capacity for research and trials.

• Workload and Ensemble Best Effort Benchmarks both addressed this requirement.

• Peak capability to meet deadlines for operational systems remains the most significant determinant.

38

4. BoM Modelling -- Current Systems

Current operational or quasi-operational systems

41

4. BoM Modelling -- Current Systems

Current operational or quasi-operational systems

• Short Range– LAPS 0.375 deg Australian Region data assimilation and 3

day prediction system– LAPS 0.125 deg Australian Region 2 day prediction system– MESOLAPS 0.05 deg mesoscale 2 day prediction system– TCLAPS 0.15 deg 2 day Tropical Cyclone prediction system– AAQFS: Australian Air Quality Forecast System

42

BoM Modelling -- Current Systems

• Medium Range– GASP T239L29 global data assimilation and 10 day

prediction system– GASP 33 member T119L19 Ensemble Prediction System

• Seasonal– POAMA coupled atmosphere-ocean 9 mths forecast system– T47L17 BMRC atmospheric model, ACOM-2 ocean model

• Decadal and Climate Change– AMIP, CMIP, C20C, Climate Feedback research

43

BoM Modelling -- Future Plans

• Short Range– non-hydrostatic mesoscale and regional model– LAPS Ensemble Prediction System– warm running assimilation cycle– demonstrate impact of increased vertical resolution – demonstrate impact of increased horizontal resolution

44


• Medium Range– T359L50 likely resolution for 2004-2005– T479L50 likely resolution for 2006-2007– Generalised Statistical Interpolation analysis scheme– Ensemble Kalman Filter assimilation scheme research– demonstrate impact of greater use of available observational data – demonstrate impact of increased vertical resolution – demonstrate impact of increased horizontal resolution– GASP EPS: experiment with increased number of members and

resolution

45


• Seasonal– POAMA2:

• increase ensemble size for hindcast phase– POAMA3:

• update ocean model to MOM-4• increase ocean & atmosphere resolution

• Decadal, Climate Change– further development on coupled model for longer time scales

applications, esp. to reduce coupled model biases

46

Summary

• BoM is in process of upgrading its supercomputer facility for the next 4-8 year period.

• NEC has been successful vendor in 2003 BoM tender.• BoM is moving to new Head Office in Melbourne Docklands with

enhanced CCF facilities.• New MPI versions of BoM major applications codes have

proved to be fast and scalable to O(100+) of processors.• BoM is continuing to expand modelling capability and

applications in line with scientific advances in NWP, seasonal and climate modelling fields.

47

Usual Suspects -- where are they now?

• Bill Bourke– BMRC modelling & HPC

expert; invited last time, invited this time, apologies both times

• Geoff Love– last time’s BoM presenter– newly appointed Director,

BoM, 1 month ago• Phil Tannenbaum

– ex HPC industry– BoM-CSIRO HPCCC Manager

since mid-2002

49

WGNE Table of Operational NWP Centres (November 2002; ref. Kamal Puri)

BoM Modelling and Computing Update

Documents

Transcript of BoM Modelling and Computing Update