Computing at NCMRWF for Short to Seasonal Range Forecast Guidance A K Bohra NCMRWF, New Delhi

CAS2K3,ANNECY,SEPT 10,2003

Computing at NCMRWF for Short to Seasonal Range Forecast

Guidance

A K BohraNCMRWF, New Delhi

CAS2K3,ANNECY, France: September 10, 2003


OVERVIEW

IntroductionObjectives ActivitiesComputing ResourcesFuture PlansConclusions


ORGANISATIONAL SET UP OF NCMRWF

DEPARTMENT OF SCIENCE & TECHNOLOGY

N C M R W F

PROJECT IMPLEMENTATIONCOMMITTEE (PIC)

SCIENTIFIC ADVISORYCOMMITTEE (SAC)

IMD CELL ICAR CELL

RESEARCH DIVISION

APPLICATIONDIVISION

ADMINISTRATIONDIVISION

COMPUTER &NETWORKING

DIVISION

OBJECTIVES

Development/Implementation of Global and Regional scale NWP models Set-up a state-of-the-art supercomputing infrastructure Set-up agro-meteorological advisory service (AAS) units, in each of the 127 agroclimatic zones of the country and provide weather based guidance Set-up a stable/fast dedicated communication network with AAS units


NETWORK OF AGROMET ADVISORY NETWORK OF AGROMET ADVISORY SERVICE (AAS) UNITS OF NCMRWFSERVICE (AAS) UNITS OF NCMRWF


NWP Models at NCMRWF• Global Models

T80 L18(150 x 150) km resolution [operational]

T170 L28 resolution [semi-operational]

• Regional Spectral Model 50 x 50 km resolution

• Mesoscale Models MM5 (Nested 90, 30, 10 km)

[operational] ETA (48 x 48) km [operational]


SPECIAL/ CUSTOMIZED FORECASTS/ SPECIAL/ CUSTOMIZED FORECASTS/ PRODUCTSPRODUCTS

Forecasts for Defence Operations and Exercises.Forecasts for Defence Operations and Exercises.

• Launch of Space Vehicles Launch of Space Vehicles • Weekly weather forecasts Weekly weather forecasts • Forecasts of Onset/Advancement/Withdrawal of Forecasts of Onset/Advancement/Withdrawal of Monsoon to IMDMonsoon to IMD• Forecasts on special events Forecasts on special events • Forecasts for Avalanche predictionForecasts for Avalanche prediction• Forecasts for tourism and adventure Forecasts for tourism and adventure • Organization of field experiments Organization of field experiments • Wind fields for Ocean State Forecast Wind fields for Ocean State Forecast • Agrinet Agrinet • Antarctica Expedition Antarctica Expedition • Govt. of Andhra Pradesh Disaster Mitigation Govt. of Andhra Pradesh Disaster Mitigation

ProjectProject• Initial conditions for Climate Experiments, Initial conditions for Climate Experiments,

Extended Range forecasting and Meso-scale Extended Range forecasting and Meso-scale modeling.modeling.

• Merged Gauge and Satellite data sets.Merged Gauge and Satellite data sets.


8

THE NCMRWF WEATHER FORECASTING SYSTEM


CURRENT OBSERVATIONAL PROGRAM

Indian Climate Research Program (ICRP) with sub components like

I The Bay of Bengal Monsoon Experiment (BOBMEX)1. BOBMEX- Pilot 1-3 November 19982. BOBMEX- Field Phase July-August 1999

Understand the processes governing the variability of organized convection over the Bay of Bengal.

II The Arabian Sea Monsoon Experiment (ARMEX)

1. ARMEX I (May- August 2002) Off-shore trough Experiment (OST) Aimed at probing the mesoscale convective episodes (high intensity

rainfall events ) occurring along west coast of India

2. ARMEX II (April- June 2003)Warm Pool Experiment(WP)

Aimed at studying evolution, maintenance and collapse of Arabian Sea warm pool observed in summer months over SE Arabian Sea


Computing Resources

•Dec-Alpha Cluster (Anupam Alpha)•Origin Cluster•Cray SV1 •Param


ANUPAM-ALPHA System

Indigenously developed Parallel Processing

Computer.

2 Nos. DEC-ALPHA Servers AS4100 @600MHz each with Memory of 1GB each 8MB Cache.

9 Nos. DEC-ALPHA Work Stations @600MHz each

with Memory of 512MB each and 4MB Cache

Inter-Node communication: Gigabit Ethernet Smart Switch Router.


Anupam-Alpha system


Cray SV1

24-Processors 1.2 GFlops per processor 8 GB Main Memory 800GB Disk.


Softwares available on Cray

•CF90 Programming

Environment for unlimited users.

•C++

•DMF

•IMSL LIBRARY(fortran 77/90 and C)


Origin Cluster ConfigurationOrigin Cluster Configuration

2 Servers(4-cpu each) Origin200@255MHz , Memory 1 GB each3 Servers (1cpu) Origin200@270MHz, Memory 512 MB each1 Server (1cpu) Origin200@180MHz, Memory 512 MB each4 Workstations O2@200MHz, Memory 512 MB each


LINK TO LAN

craysgi1_fddi1 CPU ORIGIN 200

CRAYSGI\21 CPU ORIGIN 200

CRAYSGI131 CPU ORIGIN 200

AUTOLOADER DLT

RAID BOX

AUTOLOADER DLT

AUTOLOADER DLT

IBM 3490

ADMINISTRATION WORK STATION

CRAY SV1

CRAYSGI1 1 CPU ORIGIN 200

craysgi2_fddi1 CPU ORIGIN 200 craysgi3_fddi

1 CPU ORIGIN 200

O200-34 CPU ORIGIN 200

O200-11 CPU ORIGIN 200

O200-2 4 CPU ORIGIN 200

ORIGIN VAULT

9TRACK SPOOLER TAPE DRIVE


Model is PARAM 10000 2 Sun Ultra Sparc-II Servers (4-CPU each) @300 MHz Memory 1 GB each MYRINET switch

PARAM SYSTEMPARAM SYSTEM PARAM SYSTEMPARAM SYSTEM


Parallel Processing at NCMRWF

Latitude Parallel Strategy followed

Performances were compared objectively

Actual Timings noted are no longer relevant

now.

The Exercise showed that indigenously developed clusters could be used for NWP


T126 on SV18 CPU12 CPU16 CPU24 CPU

T80 onXMPOriginAlphaPARAM


LOAD IMBALANCE IN WEATHER MODELS

… Land-Sea: In a given Latitude Circle, Number

of Land or Sea Points are not the same (LSP)

Same is true for Mountains also !!!… Convection: Convection conditions are satisfied

only in Tropics, Near the Pole, there

is No Convection !!!… Clouds: Cloud Formation, Type, etc. can be

different in different latitude circles

It is very difficult to achieve Load Balance in Weather Models


Optimized Convection (one call to convection scheme)

NCMRWF T80 Model

tropicsmid-lat

polar

Mixed latitudes


Firewall

Proxy

V.35

Network Servers

Servers - Dell PowerEdge 1300

Modem

64 kbps Leased Line

WWW

PDNSFTP

Mail

SDNS


Archival Details/DAY• T80• T170• MM5• ETA• MISC

• 643.6 MB• 936.8 MB• 233 MB• 70.1 MB• 500MB


MEDIA DETAILSMEDIA DETAILS

DAT,DLT,CDROM,EXABYTE drives are available in the center. Data is archived in various Media. For operational purpose usually DLT media is used. When a specific data set is prepared for an outside agency, data is sent in DAT / DLT/ CDROM. Data is also sent using the FTP public outgoing area, were it can be accessed by the outside agency.


Types of data Types of data distributeddistributed

Types of data Types of data distributeddistributed

•Post processed model output

•Monsoon Data

•Observation ,Analysis and Forecast


Data Visualisation Tools

• MAGICS

• GrADS

• GMT

• GIS

• NCAR Graphics

• Other Public Domain Software


ARMEX ATMOSPHERIC DATA CENTRE NCMRWF - IITD - IMD

Objectives of Data Centre

• Processing, quality control and archiving conventional and non-conventional data of atmospheric and oceanographic observations collected during ARMEX period

• Storing gridded analysis prepared by Operational Analysis Forecasting system of NCMRWF, gridded satellite data (viz INSAT, METEOSAT, SSMI, TRMM) during ARMEX period

• Assimilation of ARMEX observations using NCMRWF Global T170 Model to generate reanalysis at 70 X 70 Km resolution


Future Road Map of Modeling Activity at NCMRWF

Model ResolutionT340L60 Ensemble Runs

Meso-Scale10km over the entire

Indian Region1 km models for

Clouds

Use of more Satellite Data

Direct Radiance Assimilation

Real-time Seasonal Prediction

Climate Variability and Change

AMIP-type Runs


Infrastructure upgradation Plans

(a)Procurement of Param Padma(b)Upgradation of Cray SV1(c) Upgradation of Anupam(d)Increase Storage to 20TB(e)Create a SAN solution(f) Upgradation ILL capacity


PARAM Padma• Member of the Top500 HPC list• Supports applications based on professional Unix

and Linux• At present it is certified for AIX and Linux• By supporting above Operating Systems, PARAM

technology becomes more or less Vendor independent

• Supports Parallel Development tools and Parallel File System (PFS)

• Proven Platform running applications ranging from Scientific to Databases to Financial Modeling


Gigabit Ethernet

FC-AL

5 TB 10 TB

PARAM Padma HPC Cluster based on 4 way Power 4 + Systems

Compute Nodes connected through PARAM Padma System Area Network and Gigabit Ethernet

PARAM Padma System Area Network using PARAMNet II 16 port Switches

+ Gigabit Network

Storage Array

BackupLibrary

SAN Switch

SAN Switch

LinuxConsol

e

1 8

9 16

1 4

Graphics Workstations

PARAM Padma Compute Cluster Layout

http://www.osp.ru/lan/2001/01/107.jpg

http://www.osp.ru/lan/2001/01/107.jpg


CRAY SV1 ex

A 500 MHz clockLow-latency high bandwidth cacheA 32 Gbyte internal SSDSDRAM twice the bandwidth of SV1

DRAMVector processing rate of 2 Gflops per

CPUScalar processing rate of 500MIPS


ANUPAM-Xeon/128

• Processor :64 Dual Xeon Servers• Speed :2.4 GHz• Peak Performance :202 Gflops• Memory/processor :2GB• Interconnect Tech. : Scalable Coherent

Interface(SCI)• OS : LINUX


Anupam-Xeon/128


We began with …

1988

1992

CRAY XMP/14

CRAY XMP/216

VAX 8810

VAX8250VAX 4000/300

MFLOPs

234

MFLOPs

468

7.1 MIPS

1.2 MIPS

8.5 MIPS

Main Memory

4mw

Main Memory

128mw

48 MB

32 MB

32 MB

Storage 9.6GB

Storage

48GB


We have…..1999 Dec ALPHA

Cluster

Main Memory

1 GB

Storage

108GB

Origin200 Main Memory

1GB

Storage

108GB

Param 10000 Main Memory

1GB

Storage

180GB

2001 CRAY SV1 Main Memory

8GB

Storage

360+800GB


Near Future Plans…Param Padma 16 Nos of 4 way

64 p630 processor128 GB Aggregate memory 1.1 TB Internal Storage5 TB Storage Array10 TB Tape Library

Cray SV1 ex 24 processors with 500 MHz clock64 GB Memory40 GB/s Memory Bandwidth

ILL Increase the Leased Line capacity to 32 MB


Conclusion

• Computing Resources at NCMRWF are moderate

• Plans to enhance resources are underway


Thank You


Proposed PARAM Padma Configuration to NCMRWF

• Proposed 16 Node, 4 CPU / Node based PARAM Padma running on AIX Operating System

• Based on the high performance Power 4 plus Processor technology of IBM

• Dual PARAMNet interconnectivity for Fail over• Parallel software development environment• Fiber channel based network centric architecture

with 5 TB Storage with Parallel File System(PFS) support and 10 TB High Speed Tape Backup

• Provides balanced performance and I/O for high data intensive applications.

• Applications like T80, T170, T126 and MM5 Weather codes are running successfully


ANUPAM-Xeon/128

• Processor :64 Dual Xeon Servers

• Speed :2.4 GHz

• Peak Performance :202 Gflops

• Memory/processor :2GB

• Interconnect Tech. : Scalable Coherent Interface(SCI)

• OS : LINUX

Why more computing power?

• Computing Requirement increases as more data are ingested in data assimilation system

• Assimilation of Direct Radiance data from Satellites

• 4-D Variational Technique needs most compute intensive running of Adjoint Models (as at ECMWF and Meteo-France)


Parallel Processing Software Indigenously developed

ANULIB.

In ANULIB Message Passing Communication Calls aredivided into 4 categories a. Initialisation calls b. Termination calls c. Communication calls d. Miscellaneous calls

ANUPAM-ALPHA System runs at NCMRWF as Master-Slave Parallel System

with maximum up to 1+8 nodes.


Local Area Network Details

• All LAN servers are DELL PowerEdge 1300 Machines.

• Red Hat Linux 8.0• Pheonix Firewall• Cabletron Smart switch 2200

Computing at NCMRWF for Short to Seasonal Range Forecast Guidance A K Bohra NCMRWF, New Delhi

Documents

Transcript of Computing at NCMRWF for Short to Seasonal Range Forecast Guidance A K Bohra NCMRWF, New Delhi