Changes at ECMWF
description
Transcript of Changes at ECMWF
ECMWFCAS 2001, Annecy Slide 1
Changes at ECMWF
Walter Zwieflhofer
Head of Computer Division
ECMWFCAS 2001, Annecy Slide 2
Operational system (October 2001)Data assimilation
4D-Var with a 12-hour data window Horizontal resolution: 120 km for inner and 40 km for outer loops
Global atmospheric forecasts Forecasts to 10 days from 12 UTC at 40 km resolution Ensemble forecasts to 10 days with 51 members at 80 km resolution Four forecasts per day to 3 days to provide boundary conditions for
some of the ECMWF Member States (40 km, 3D-Var)
Ocean wave forecasts Global forecasts to 10 days at 40 km resolution European waters forecasts to 5 days at 15 km resolution
Seasonal forecasts Global forecasts to 6 months (ensemble with 30 members)
ECMWFCAS 2001, Annecy Slide 3
Forecasting-system changes since 1999
N Hem Mean reduction in day-one error measured in pre-operational trials S Hem
19r2 Increased resolution in stratosphere21r1 Assimilation of raw TOVS/ATOVS radiances21r4 New error statistics for 4D-Var Jb, SSMI wind data, corrected use of humidity observations, increased PBL resolution, improved cloud, convection and orography.22r1 Improved SSM/I QC and second SSM/I satellite
22r3 New error variances (for Jb and Jo) additional ATOVS data, improved land-surface, sea-ice and LW radiation23r1 12-hourly 4D-Var23r3 T511/159 resolution, second ATOVS satellite, increased angular resolution in wave model
ECMWFCAS 2001, Annecy Slide 4
Evolution of forecast skill for northern and southern hemispheres
ECMWFCAS 2001, Annecy Slide 5
Error distribution for Day 5 (Europe)
ECMWFCAS 2001, Annecy Slide 6
Annual-mean RMS errors against analyses from WMO scores
ECMWFCAS 2001, Annecy Slide 7
Planned improvements2002 Increase of the inner loop resolution of
4D-Var from 120 km to 80 km2003-2004 Assimilation of cloud and rain in 4D-Var
Higher vertical resolution in free troposphere and lower stratosphere
Assimilation of AIRS data
Operational running of a Severe Weather forecasting system
2005-2006 Assimilation of IASI data
ECMWFCAS 2001, Annecy Slide 8
Computer configuration – October 2001
ECMWFCAS 2001, Annecy Slide 9
PrepIFSECMWF tool to prepare forecast and data assimilation
experimentsUses the operational configuration of the forecasting
system and allows researcher to introduce modifications to:
input data (including the selection of observations) the model code the resolution to be used and many other parameters
Written Java, with a client part that can be installed on remote machines
Allows registered external users to configure and submit experiments via the Internet
ECMWFCAS 2001, Annecy Slide 10
Major changes to thecomputing environmentDesktop systems (procurement completed in 2000)
Decision was taken to replace all proprietary UNIX workstations with PCs running LINUX over a period of three years
Data Handling System – DHS (procurement in progress) A scalable system that will be able to support the expected
growth over 6+ years is required Invitation to Tender for a new system was issued earlier this year
Supercomputer – HPC (procurement in progress) Fujitsu systems are rented and the contract ends on 31 Dec 2002 Invitation to Tender for a replacement to cover the period
2003-2006 was issued in March this year
ECMWFCAS 2001, Annecy Slide 11
DHS procurementArchive holds currently some 400 TB of data (primary data)
and has been growing by some 60% per annumTwo services need to be supported at ECMWF:
ECFS, a large-scale UNIX-like filesystem for general use MARS, a customised meteorological archival and retrieval system
Vendors were asked to offer a scalable storage system and were given the option to offer a replacement for the ECFS and/or MARS software layer
DHS and HPC procurements were separated because: Lifetime of the data handling systems tends to be considerably
longer than that of the HPC systems Allows a simpler HPC procurement not affected by the complexities
associated with the replacement of the archive system
ECMWFCAS 2001, Annecy Slide 12
HPC procurementECMWF asked for a phased introduction of resources covering
the period 2003-2006, with a parallel run during the second half of 2002
The specifications allowed for some flexibility, enabling vendors to adapt their response to their product development plans. This included the option to propose a 3-month shift of the service period
The funds available were publishedVendors were required to commit to performance increases,
expressed as factors, over the performance of the Fujitsu systems installed at ECMWF
Requirements for disk space, I/O and network bandwidth were expressed as function of these factors
ECMWFCAS 2001, Annecy Slide 13
The Benchmark SuiteIFS tests (for the performance commitments)
4D-VAR (T511/255 L60; 2, 4 or 6 copies depending on PHASE) DETERMINISTIC FORECAST (T799 L90; 2 or 4 copies) EPS (T399 L60; 50 or 100 copies)
Burton Smith taxonomy: Size * g >> 2 * O + L Suitable for both T-type and C-type systems
IFS model runs well on all architectures4D-Var is very sensitive to communication bandwidth and
has sections on which scalar machines do wellResolutions representative for the workload planned for
2003-2006 were used
ECMWFCAS 2001, Annecy Slide 14
Observations Vendors were allowed to optimize the codesHowever, changes that would affect the readability or
portability were not permittedSome vendors used a hybrid MPI / OpenMP approachSome vendors found that such a hybrid approach did not
improve the results obtained by using MPI only Percentages of peak:
4D-Var scaled not as well the deterministic forecast and the EPS Scalar machines: 3% to 12%, depending on test/product Vector machines: 14% to 38%, depending on test/product
Rule of thumb for ECMWF’s codes was confirmed: ~10% for scalar architectures and ~30% for vector architectures
ECMWFCAS 2001, Annecy Slide 15
Status of both procurementsThe Tender Evaluation Boards have completed their work and
have made recommendations to the ECMWF DirectorThe Director has awarded both contracts, subject to approval
by ECMWF CouncilDocuments detailing the respective procurement process and
the proposed solutions have been sent to the technical and financial committees of the ECMWF Council
These committees will meet in November to consider ECMWF’s proposals and to make recommendations to Council
Council will meet in December and (hopefully) authorize the Director to sign the appropriate contracts
ECMWFCAS 2001, Annecy Slide 16
Changes over the next 18 months
New DHS
NewServers
Linux PCs
New HPC systemNew
Network