12/23/2011

E.Mirvis : DTC ported NEMS source, libs, data and env: Dec 201104/19/2023 1

BUILDING FEE/FSE ENVIRONMENT @ JNT/DTC/DET

NEMS COMPONENTS PORTING AND AVAILABILITYon “JET” and “ Bluefire”

Eugene Mirvis, RAL/NCAR@EMC NOAA

mailto:RAL/NCAR@EMC

E.Mirvis : DTC ported NEMS source, libs, data and env: Dec 2011

DTC/DET Environment Componentson “JET” & “Bluefire”


Compilers

Porting :: FEE :: AvailabilitySufficient Computing and Storage Resources

Data Availability

3rd party libraries availability

Ported Apps Source Code & update capabilities

Building system scripting

Flexible Modular Environment

Version control repository, sources.

Regression testing Data and jobs

Re- scripting

GSI data, sufficient networking

Gridded NAM, GFS, snow, Ice, SST

SufficientTMP and HIST

space

WORKIG ENVIRONMENT ON JET ::.chsrc

#/bin/tcshmodule switch icc/11.1.072 icc/11.1.073module switch intel/11.1.072 intel/11.1.073module switch mvapich2/1.4.1-intel-11.1 mvapich2/1.6-intel-11.1setenv SVN_EDITOR nanoalias wdir "cd /lfs0/projects/dtc/NEMS"alias ldata "cd /mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data"setenv PATH ${PATH}:/opt/mvapich2/1.4.1-intel-11.1/includesetenv ESMF_DIR /mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data/NEMSlibs/esmf/esmf_3_1_0rp5m2gsetenv DIR_ESMF /mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data/NEMSlibs/esmf/esmf_3_1_0rp5m2gsetenv ESMF_COMM mpich2setenv ESMF_OPT Osetenv DIR_NETCDF "$NETCDF/lib/"##setenv ESMF_OPT gsetenv BASEDIR /mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data/NEMSlibssetenv NEMSdata /mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data###setenv ESMF_MPIRUN "mpirun_rsh -np 2 h1 h1 "

04/19/2023 4

04/19/2023 5

WORKIG ENVIRONMENT ON :: BLUEFIRE#/bin/tcshsource /contrib/Modules/3.2.6/init/cshmodule load makedepf90module load gradsmodule load svn-1.5module load netcdf/3.6.2_deprecatedmodule switch xlf/12.01.0000.0005.091127 xlf/12.01.0000.0003alias wdir "cd /glade/proj2/ral/RJNTB/NEMS/"#module switch icc/11.1.072 icc/11.1.073#module switch intel/11.1.072 intel/11.1.073#module switch mvapich2/1.4.1-intel-11.1 mvapich2/1.6-intel-11.1setenv SVN_EDITOR nanosetenv NEMS "/glade/proj2/ral/RJNTB/NEMS"alias wdir "cd ${NEMS}"alias ldata "cd ${NEMS}/NEMS_libs_data"setenv NEMSdata "${NEMS}/NEMS_libs_data"setenv NEMSlibs "${NEMSdata}/NEMSlibs"#setenv PATH ${PATH}:/opt/mvapich2/1.4.1-intel-11.1/includesetenv ESMF_DIR ${NEMSlibs}/esmf/esmf_3_1_0rp5m3setenv DIR_ESMF ${NEMSlibs}/esmf/esmf_3_1_0rp5m3#setenv ESMF_COMM mpich2setenv INC_ESMF $(DIR_ESMF)/mod/modO/AIX.default.64.mpi.defaultsetenv ESMF_OPT Osetenv DIR_NETCDF "$NETCDF/lib/"##setenv ESMF_OPT gsetenv BASEDIR $DIR_NETCDF###setenv ESMF_MPIRUN "mpirun_rsh -np 2 h1 h1 "setenv LOGIN emirvissetenv PATH ${PATH}:/usr/local/bin/setenv NWPROD $NEMSdata/nwprod


NEMS Source Code on “JET” & “Bluefire”

04/19/2023 7

Where is everything on JET? [emirvis@fe2 NEMS]$ pwd /mnt/lfs0/projects/dtc/NEMS[emirvis@fe2 NEMS_libs_data]$ pwd/mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data

Where all data and libraries on JET?[emirvis@fe2 NEMS_libs_data]$ ls -trlatotal 44drwxr-sr-x 3 emirvis dtc 4096 Sep 18 16:23 stmpdrwxr-sr-x 5 emirvis dtc 4096 Sep 29 21:04 nwproddrwxr-sr-x 3 emirvis dtc 4096 Oct 11 21:08 RTdatadrwxr-sr-x 22 emirvis dtc 4096 Nov 1 18:47 REGRESSION_TESTdrwxr-sr-x 27 emirvis dtc 4096 Nov 17 02:21 ..drwxr-sr-x 35 emirvis dtc 4096 Dec 6 00:11 ptmpdrwxr-sr-x 9 emirvis dtc 4096 Dec 14 23:39 NEMSlibsdrwxr-sr-x 11 emirvis dtc 4096 Dec 14 23:39 .drwxr-sr-x 3 emirvis dtc 4096 Dec 14 23:40 LIBdrwxr-sr-x 7 emirvis dtc 4096 Dec 14 23:40 INCMOD

Bluefire/glade/proj2/ral/RJNTB/NEMS:

The one which was running:/glade/proj2/ral/RJNTB/NEMS/14601/

/glade/proj2/ral/RJNTB/NEMS/14601/job/regression_tests

The one from NCO (I couldn’t run yet):/glade/proj2/ral/RJNTB/NEMS/nam_nems_nmmb_fcst.fd_em

LIBS+DATA:

/glade/proj2/ral/RJNTB/NEMS/NEMS_libs_data

My Bluefire RUN Results:

/glade/proj2/ral/RJNTB/NEMS/NEMS_libs_data/ptmp/RT_111/NMM_CNTRL

04/19/2023 8

JET: CODE/SCRIPTS:: /mnt/lfs0/projects/dtc/NEMS/

drwx--S--- 7 emirvis dtc 4096 Feb 11 2011 12204drwxr-sr-x 5 emirvis dtc 4096 Mar 31 2011 libdrwx--S--- 7 emirvis dtc 4096 Apr 28 2011 13612drwx--S--- 7 emirvis dtc 4096 Jun 1 2011 13537drwxr-sr-x 7 emirvis dtc 4096 Jun 16 22:46 13537wrkdrwxr-sr-x 7 emirvis dtc 4096 Jul 13 19:16 10566_emdrwx--S--- 7 emirvis dtc 4096 Aug 18 23:51 13537wrk2drwxr-sr-x 7 emirvis dtc 4096 Sep 8 12:46 15367drwxr-sr-x 7 emirvis dtc 4096 Sep 8 12:49 13537wrk_clndrwx--S--- 7 emirvis dtc 4096 Oct 6 15:52 14601ncodrwxrwsr-x 5 ligia dtc 4096 Oct 20 16:57 ..drwxr-xr-x 7 emirvis dtc 4096 Oct 21 20:35 14601nco2drwxr-sr-x 7 emirvis dtc 4096 Nov 4 19:37 15555drwxr-sr-x 7 emirvis dtc 4096 Nov 7 23:23 16000drwxr-sr-x 7 emirvis dtc 4096 Nov 8 20:16 16148drwxr-sr-x 7 emirvis dtc 4096 Nov 9 04:34 16149drwxr-sr-x 6 emirvis dtc 4096 Nov 9 05:01 good1drwxr-sr-x 7 emirvis dtc 4096 Nov 9 20:41 14601nco2.11.3drwxr-sr-x 7 emirvis dtc 4096 Nov 10 12:29 NEMS-15555drwxr-sr-x 7 emirvis dtc 4096 Nov 10 14:38 NEMS-NCO_ops-2011drwxr-sr-x 7 emirvis dtc 4096 Nov 17 02:22 dtrunk_jetdrwxr-sr-x 11 emirvis dtc 4096 Dec 14 23:39 NEMS_libs_datadrwxr-sr-x 2 emirvis dtc 4096 Dec 15 09:11 TARSdrwxr-sr-x 7 emirvis dtc 4096 Dec 15 09:29 dEM

ALL you want to use sould be available now at:

/mnt/lfs0/projects/dtc/NEMS/TARS (see next)04/19/2023 9

Where is a tarfile to get: /mnt/lfs0/projects/dtc/NEMS/TARS

04/19/2023 [email protected] 10

To compile:cd $NEMS_DIR/srcgmake cleangmake nmmOther options: gmake nmm_gfs --- for both NMM and GFS core - without the GOCART orgmake gen --- for GEN core only orgmake nmm_gfs_gen_post GOCART_MODE=full --- for GEN, NMM and GFS cores - with the GOCART and post orgmake nmm_gfs GOCART_MODE=full --- for both NMM and GFS core - with the GOCART orgmake nmm --- for NMM core only orgmake nmm_post --- for NMM core only - with post orgmake gfs --- for GFS core only - without the GOCART orgmake gfs_post --- for GFS core only - with post without the GOCART orgmake gfs GOCART_MODE=full --- for GFS core only - with the GOCART orgmake fim --- for FIM core only orgmake fim_gfs --- for both FIM and GFS core only - without the GOCART orgmake fim_gfs GOCART_MODE=full --- for both FIM and GFS core only - with the GOCART

04/19/2023 Eugene Mirvis 11

[emirvis@fe3 TARS]$ ls -trlatotal 3491916drwxr-sr-x 25 emirvis dtc 4096 Dec 15 19:39 ..-rw-r--r-- 1 emirvis dtc 172963840 Dec 15 20:30 16149.tar-rw-r--r-- 1 emirvis dtc 112517120 Dec 15 20:33 NEMS-15555.tar-rw-r--r-- 1 emirvis dtc 287057920 Dec 15 20:53 NEMS-NCO_ops-2011.tardrwxr-sr-x 5 emirvis dtc 4096 Dec 22 15:55 nwproddrwxr-sr-x 3 emirvis dtc 4096 Dec 22 15:56 RTdata-rw-r--r-- 1 emirvis dtc 2585804800 Dec 22 19:59 RTdata.tar-rw-r--r-- 1 emirvis dtc 229847040 Dec 22 20:00 nwprod.tar-rw-r--r-- 1 emirvis dtc 187473920 Dec 22 20:03 NEMSlibs-dtrb.tardrwxr-sr-x 7 emirvis dtc 4096 Dec 22 20:45 .drwxr-sr-x 10 emirvis dtc 4096 Dec 22 20:48 NEMSlibs-dtrbdrwxr-sr-x 2 emirvis dtc 4096 Dec 22 20:49 LIBdrwxr-sr-x 7 emirvis dtc 4096 Dec 22 20:49 INCMOD

You will need:1. One of the source revision tarball: NEMS-NCO_ops-2011.tar2. Regression Test data : RTdata.tar3. $NWPROD min tarball :: nwprod.tar4. NEMSlibs distr. Tarball :: NEMSlibs-dtrb.tar

5. Untar it in separate directories and goto: “NEMSlibs-dtrb”6. And run script: “only_3rd_libs_BUILT_LIBS”7. If you don’t have esmf or makedepf90 as modules – built it first usingScript GLOBAL_BUILT_LIBS-1

3rd-parties LIBS (all)

04/19/2023 12

emirvis@fe1 NEMSlibs]$ ls -trlatotal 1300drwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:33 baciodrwxr-sr-x 7 emirvis dtc 4096 Dec 14 18:37 esmfdrwxr-sr-x 4 emirvis dtc 4096 Dec 14 18:39 Linuxdrwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:39 spdrwxr-sr-x 2 emirvis dtc 12288 Dec 14 18:39 w3_gdrwxr-sr-x 4 emirvis dtc 4096 Dec 14 18:39 nemsiodrwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:39 libdrwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:39 LIBdrwxr-sr-x 7 emirvis dtc 4096 Dec 14 18:39 G_libsdrwxr-sr-x 6 emirvis dtc 4096 Dec 14 18:39 makedepf90-2.8.8drwxr-sr-x 6 emirvis dtc 4096 Dec 14 18:39 new_nemsiodrwxr-sr-x 6 emirvis dtc 4096 Dec 14 18:40 srcdrwxr-sr-x 3 emirvis dtc 4096 Dec 14 18:40 sp_g-rw-r--r-- 1 emirvis dtc 1200696 Dec 14 18:40 libw3.adrwxr-sr-x 3 emirvis dtc 4096 Dec 14 18:40 june_gfsiodrwxr-sr-x 6 emirvis dtc 4096 Dec 14 18:40 incmoddrwxr-sr-x 8 emirvis dtc 4096 Dec 14 18:40 INCMODdrwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:40 lib_gooddrwxr-sr-x 6 emirvis dtc 4096 Dec 14 18:40 incmod_gooddrwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:40 LIB_gooddrwxr-sr-x 8 emirvis dtc 4096 Dec 14 18:40 INCMOD_gooddrwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:40 ___w3drwxr-sr-x 3 emirvis dtc 4096 Dec 14 18:40 ___w3_opn_gbleventdrwxr-sr-x 3 emirvis dtc 4096 Dec 14 18:40 jun_new_nemsio_nemsio+drwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:40 jun_new_nemsio_w3+drwxr-sr-x 29 emirvis dtc 4096 Dec 14 18:40 .drwxr-sr-x 2 emirvis dtc 4096 Dec 14 18:40 jun_new_nemsio_bacio+drwxr-sr-x 3 emirvis dtc 4096 Dec 14 20:23 gfsiodrwxr-sr-x 2 emirvis dtc 4096 Dec 14 20:24 sigiodrwxr-sr-x 11 emirvis dtc 4096 Dec 15 18:58 ..[emirvis@fe1 NEMSlibs]$ pwd/mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data/NEMSlibs


REGRESSION TESTS @EMC (as designed)

Original NEMS Regression test data

/meso/noscrub

REGRESSION_TEST

REGRESSION_TEST_baselines

/stmp

/ptmp

Place to compare :: $NEMS_RT_ptmp

Originating | prev. results:: $NEMS_RT_stmp


Running RT job scripts

RT.sh

rt_gfs.sh

rt_nmm.sh

rt_fim.sh…

rt_*.sh

Set ting Computational Env Vars

nmm_ll.IN

nmm_conf.IN

nmm_ll.sh

gfs_ll.IN

gfs_conf.IN

*_ll.IN

*_conf.INModel switch

gfs_ll.sh

*_ll.sh

POE / PBS Submit


Submitting a single RT 1. Submit RT- Job script

2. Wait for the start

3. Follow your RUN…

4. Wait for RUN is finished

5. Check results or COPY the results


FEE Criterions DRAFT 2011-09-16 (EM : based on Mark Iredell’s thoughts) Procedure for defining functional equivalence on different environments Select key fields from the import state Fi

may include a selection of initial cases Run control run C of application on control environment Perturb each Fi in the last significant digit

nominally after the 6th decimal digit for 32-bit reals for each value Fi(x), construct a distribution

Gaussian distribution with width being | F i(x)/1.e6| randomly select perturbation from that distribution

Run perturbed run P from perturbed import state on control environment. Select key fields from the export states Cj (control run) and Pj (perturbed run).

may include a selection of output times Determine area weight w(x) Compute the butterfly vector Butterfly vector will have units of original export fields Now run control run but on different environment D Compute difference vector like butterfly vector but using D instead of P. Compute dimensionless difference butterfly number vector For a bit-identical requirement, must be zero For a functionally equivalent requirement, must be largely less than 10


Components of the TESTBED FSE/FEE

1. Architecture:: Hardware, Registries, Cache, Processors, Memory management , OS, Data storage and management, support

2. Resources:: PEs, memory, cycles, queues etc.

3. Environment:: OS version, SYS Libraries, Compilers/Versions, Flexible module environment, 3rd party libraries, flex Env variables mechanism

4. Built system:: scripting, workflows, launchers, parallel submission

5. Data :: Availability, formats, compatibility, precision,

6. Code :: Documentation, Self-documenting , auto configuration, testing/evaluation and benchmark system


EMCDev Trunk

EMCVer Y.mbranch

DTC Ver X.nbranch

Stablebranch

Stablebranch

DTCDev Trunk

EMC

DTC - Rep. Locations

Solid - Full ctrl

Ver Y.m

Ver .m+1

Ver X.n

Ver X.n+1

Dashed: Read or Link

E.Mirvis : Proposed NCAR-EMC cross- repository version control (DRAFT)

NEMS regression testsBased on Standard NEMS test development

E. M.: includes some materials from Ratko Vasic Apr 2011 presentation for Cirrus


Inspiration:

Several model cores + several people = accident waiting to happen

Complicated structure, more components – more room for errors

Solution:

• Series of small tests checking as many as possible combinations (regression tests)


What do we test?

Bit-identical result reproducibilityDifferent domain decompositionThreadingDifferent physics optionsTiming (only NMMB, should do it for GFS too)Different compilationsAnd many more…


Recent changes in Regression test:

Separation of the tests to “BIG” and “SMALL” (names are not determined yet, we may call them “long”, “quick”, “fast”…)

GFS submit scriptsNew compilation options


“BIG” and “SMALL” tests:“SMALL” test should be performed before any

repository commitIt consist of 8 NMMB runs, 7 GFS runs and 1 GENTakes about 1.5 hour to finish

“BIG” run is ran from cron once a week, every SaturdayIt has 46 tests and 4 different compilations (ESMF3,

ESMF5, NMM only, NMM with TRAPS ON, GFS only should be added)

Took 7 hours to finish


Regression test options:run RT.sh <arg1> <arg2>EM: I wouldn’t suggest you to run RT.sh … yet beyond the first test (baseline) on Jet

before you will be able to replicate my results by running for instance on Jet:

> cd /mnt/lfs0/projects/dtc/NEMS/NEMS-15555/job/regression_tests[emirvis@fe3 regression_tests]$ qsub nmm_ll_em15555Your job 5282788 ("NEMS_RT15555") has been submitted

Regular (SMALL) run No arguments

Create baseline(s) One argument: nmm, gfs, gen or all

Run “BIG” test Two arguments: “big test”

Other usage:

• Edit RT.sh and run either model/option/test• EM: again, while I’m working to extend tests please edit your copies of nmm_ll files

•

26

“If tests” for each run:

Smalltest

Createbaseline If tests

Yes Yes if [ ${CB_arg} != gfs -a ${CB_arg} != gen ]; then

Yes No if [ ${CREATE_BASELINE} = false ]; then

No Yes if [ ${CB_arg} != gfs -a ${CB_arg} != gen -a ${RT_BIG} = true ]; then

No No if [ ${CREATE_BASELINE} = false -a ${RT_BIG} = true ]; then

27

NMMB tests:

1 - Baseline global to compare with previous trunk version: free fcst, pure binary input. 2 - Baseline global with NEMSIO input: free fcst, NEMSIO input. 3 - Global restart run: restart, pure binary input. 4 - Global restart run with NEMSIO input: restart, NEMSIO input. 5 - Global with different domain decomposition: 3x5 compute tasks with single thread, opnl physics, free fcst, pure binary input. 6 - Global with multiple threading: 6x5 compute tasks with 2 threads, opnl physics, free fcst, pure binary input. 7 - Global with GFS physics: GFS physics, free fcst, pure binary input. 8 - Baseline regional to compare with previous trunk version: free fcst, pure binary input. 9 - Baseline regional with NEMSIO input: free fcst, NEMSIO input.10 - Regional restart run: restart, pure binary input.11 - Regional restart run with NEMSIO input: restart, NEMSIO input.12 - Regional with different domain decomposition: 3x5 compute tasks with single thread, opnl physics, free fcst, pure binary input.13 - Regional with multiple threading: 6x5 compute tasks with 2 threads, opnl physics, free fcst, pure binary input.14 - Regional with GFS physics: GFS physics, free fcst, pure binary input.15 - Regional with nesting: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, pure binary input16 - Regional restart with nesting: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, pure binary input17 - Regional with precipitation adjustment on: free fcst, pure binary input.18 - Regional writing time series: free fcst, pure binary input.19 - Regional with nesting: test digital filter

04/19/2023 [email protected] 28

The way you can run NMMB @ DTC for now:What do you need to RUN in your job/regression_test dir:1. ts_locations.nml2. phy_state.txt3. dyn_state.txt4. nmm_ll* (script)5. ../exe/NEMS.x

RESULTS::My 48 hrs run results on JET

[emirvis@fe1 ptmp]$ ls48_hst_comp RT_111-0 RT_15555 RT_203dtest_du1RT_05 RT_111-1 RT_15555+ RT_444RT_06 RT_111-2 RT_15555_o RT_555RT_07 RT_111-3 RT_201 RT_777RT_08 RT_111-4 RT_202 RT_NEMS-NCO_ops-2011RT_09 RT_112 RT_203-1 testRT_11 RT_12 RT_203-1.1-good1RT_111 RT_14 RT_203-16149-fullRT_111+1 RT_14601nco2.11.3 RT_203dtestLocated at:[emirvis@fe1 ptmp]$ pwd/mnt/lfs0/projects/dtc/NEMS/NEMS_libs_data/ptmp

My 48 hrs run results on “Bluefire”/glade/proj2/ral/RJNTB/NEMS/NEMS_libs_data/ptmp/RT_111/NMM_CNTRL

12/23/2011

Documents

Transcript of 12/23/2011