Skip navigation
NASA Logo, National Aeronautics and Space Administration
Modeling Guru Banner

Blog Posts

Blog Posts

Items per page
1 2 Previous Next
0

NU-WRF migration to GitHub in NU-WRF

Posted by Carlos Cruz May 28, 2019

NU-WRF has moved to NASA GitHub.

For NASA personnel NU-WRF can be accessed from the following URL:

https://developer.nasa.gov/NU-WRF/nuwrf

Note that users must be in a NASA network or using VPN if not on the NASA campus.

Furthermore, in order to collaborate users must have a NASA GitHub account (one can be requested via NAMS).

Non-NASA users

Note that the NASA GitHub is not visible outside the agency firewall. External users who have filled out a "Software Usage Agreement" should be able to access NU-WRF fron an external Git server.

Questions/comments about the NU-WRF project and/or collaborating via GitHub should be addressed to Carlos Cruz at carlos.a.cruz@nasa.gov


0

"Charney Patch 3" was released on 23 May 2019. Changes include:


  • Includes WRF-NoahMP.3.6 coupling, as well as some testcases.
  • Includes new physics parameterizations:
    • NTU (National Taiwan University) microphysics

        A multi-moment four-ice (pristine ice, aggregate, graupel, and hail)

        bulk microphysical scheme

    • WRF NSSL electrification scheme

        This scheme requires environment variable WRF_ELEC=1. This can be set

        in the build configuration files

  • Includes testcases for the above.
  • DOE CRM: utilizes WRF as a CRM/LES using doubly periodic lateral boundaries.
  • Upgraded Intel compiler version to 18.x, consistent with LISF.
  • Updated build system.
  • Updated NU-WRF User Guide.
  • Updated tutorial documentation and testcases.
  • NEVS: Updated documentation and organization for ease of use.

 

Tar files are available to NCCS and NAS s0942 group members.

On NCCS Discover:

/discover/nobackup/projects/nu-wrf/releases/stable/nu-wrf_v9p3-wrf391-lis72.tgz
/discover/nobackup/projects/nu-wrf/releases/stable/nu-wrf_v9p3-wrf391-lis72.tar. bz2

And on NAS Pleiades:

/nobackupp8/nuwrf/releases/stable/nu-wrf_v9p3-wrf391-lis72.tgz
/nobackupp8/nuwrf/releases/stable/nu-wrf_v9p3-wrf391-lis72.tar.bz2

0

1. Introduction

This document describes how to configure your environment and how to launch TotalView on discover.

 

Please see https://www.roguewave.com/products-services/totalview for documentation on using TotalView.

 

Note

The instructions that follow assume that you are using the bash shell on discover.  If you are using another POSIX-compatible shell (like zsh) the following shell-related statements/commands may be used as is.  If you are using a non-POSIX-compatible shell (like csh or tcsh), then you must modify the statements/commands accordingly.

2. Logging onto discover

Log onto discover with X11 forwarding enabled.  This allows the TotalView windows to pop up onto your local computer.

 

On your Linux or MacOS local computer, add this host block to your $HOME/.ssh/config file.

 

Host discover-sp3
   User your_username_here
   ProxyCommand ssh login.nccs.nasa.gov direct %h
   ForwardX11 yes
   ForwardX11Trusted yes
   Protocol 2

 

Then log onto discover via:

 

% ssh discover-sp3

 

The following sections assume that you are now on discover.


3. Authorized keys

To run multi-node jobs on the discover compute nodes, you must generate password-less SSH public/private keys.  If you do not have id_dsa, id_dsa.pub, and authorized_keys files in your in your $HOME/.ssh directory on discover, then run

 

% ssh-keygen -t dsa % cat id_dsa.pub >> authorized_keys

 

Important

When you run ssh-keygen, you will be prompted for a passphrase to protect the private key.  In general, this is good security practice; however, to run on multiple compute nodes, your private key must be unprotected.  So, when prompted for a passphrase, leave it blank; i.e., hit enter two times.

4. Compiling

Compile your program with '-g'.

 

For example, when configuring LIS, at this prompt:

 

Optimization level (-2=strict checks, -1=debug, 0,1,2,3, default=2):

 

enter either -1 or -2.

5. Interactive session

Small serial/non-MPI programs can be run in TotalView right on the discover login node; however, if your job is large or your program runs in parallel, then you must request an interactive session on discover.

 

% xsalloc --ntasks=<ntasks> --time=<walltime> --account=<account> --constraint=hasw

 

For example,

 

% xsalloc --ntasks=252 --time=6:00:00 --account=s0942 --constraint=hasw

 

For more information on these and other command line switches, see man salloc.

 

Note

When you are logged onto an interactive session on a compute node, your $HOME/.profile will not be sourced.  Depending on how you use your .profile and .bashrc scripts, you may need to explicitly source your .profile.

 

% source ~/.profile
Note

When you are logged onto an interactive session on a compute node, your environment may be empty.  You may need to reset your LIS environment variables and your required modules:

 

% module load lis_7_intel_14_0_3_174_sp3

 

Please see the blog post Creating a custom modulefile for more information.

Note

When you are logged onto an interactive session on a compute node, you will be placed in your $HOME directory.  You must change directory (cd) to your running directory.

 

For example:

 

% cd /discover/nobackup/jvgeiger/experiment_4

6. Loading TotalView

There are many TotalView modules available on discover.

 

I use either tool/tview-2017.1.21 or tool/tview-8.11.0.0.

 

NCCS-recommended version of TotalView
% module load tool/tview-2017.1.21

 

or

 

Old version of TotalView
% module load tool/tview-8.11.0.0

7. Running TotalView

7.1. Serial/non-MPI programs

For serial/non-MPI programs, for example ./LDT ldt.config, you run:

 

% totalview ./LDT

 

You will be presented with a 'Startup Parameters' dialog box:

startup.png

 

Select the 'Arguments' tab and fill in any command line arguments.

arguments.png

7.2. MPI programs

For MPI programs, for example mpirun -np 252 ./LIS  --file lis.config.exp4, you run:

 

% totalview ./LIS

 

Provide any command line arguments, in this example '--file lis.config.exp4' into the 'Arguments' tab (see above section).

 

Then you must specify how to run the program in parallel.  Select the 'Parallel' tab.

parallel.png

 

Click on the 'Parallel system' drop-down menu and select the appropriate parallel system.  For discover, this will be 'Intel MPI-Hydra'.

parallel_dropdown.png

 

Then fill in the number of nodes and tasks for the job.  In this example, I am running 252 processes, and each discover compute node has 28 cores, so I am using 9 nodes (252 / 28 = 9).

tasks.png

 

Note

Older versions of TotalView do not have 'Intel MPI-Hydra'.  For those versions of TotalView, select 'Intel MPI-MPD'.

 

If you are using an older version of TotalView, then you must run

 

% mpdboot -r sshmpi -n $SLURM_NNODES -f $PBS_NODEFILE

 

before you run TotalView.

8. Quick-list

8.1. Older TotalView

  • request interactive session

    xsalloc --ntasks=<ntasks> --time=<walltime> --account=<account> --constraint=hasw
  • source .profile

  • load modules

  • cd to running directory

  • mpdboot -r sshmpi -n $SLURM_NNODES -f $PBS_NODEFILE

  • totalview ./LIS

  • request interactive session

    xsalloc --ntasks=<ntasks> --time=<walltime> --account=<account> --constraint=hasw
  • source .profile

  • load modules

  • cd to running directory

  • totalview ./LIS

0

Below is a list of issues we encounter when switching from the Intel compilers to the GCC (gcc and gfortran) compilers.

Issue 1

Logical variables should be set using .false. or .true. values. Do not use 0 or 1.

 

Bad
  gCalcSOSmode = 0

Good
  gCalcSOSmode = .false.

Issue 2

Message

    ../core/LIS_histDataMod.F90:1523.33:


              n,2,ntiles,(/"kg/m2","mm"/),1,(/"-"/),1,1,1,&
                                   1

    Error: Different CHARACTER lengths (5/2) in array constructor at (1)

Correction

Strings within an array constructor must have the same length.  Use spaces to pad the shorter strings.

 

Bad
          call register_dataEntry(LIS_MOC_LSM_COUNT,LIS_MOC_SNOWICE,&
               LIS_histData(n)%head_lsm_list,&
               n,2,ntiles,(/"kg/m2","mm"/),1,(/"-"/),1,1,1,&
               model_patch=.true.)

Good
          call register_dataEntry(LIS_MOC_LSM_COUNT,LIS_MOC_SNOWICE,&
               LIS_histData(n)%head_lsm_list,&
               n,2,ntiles,(/"kg/m2","mm   "/),1,(/"-"/),1,1,1,&
               model_patch=.true.)

Issue 3

Message

    ../rtms/LIS-CMEM3/pe/CMEM3_set_pedecvars.F90:88.39:

      real          :: vdata(LIS_rc%npatch(n,LIS_rc%lsm_index))
                                           1

    Error: Symbol 'n' is used before it is typed at (1)

Correction

Variables must be declared before they are used.  Here n must be declared before vdata because vdata references n.

 

Bad
    implicit none
    type(ESMF_State)       :: DEC_State
    character*100          :: vname
    real          :: vdata(LIS_rc%npatch(n,LIS_rc%lsm_index))
    integer                :: n

Good
    implicit none
    integer                :: n
    type(ESMF_State)       :: DEC_State
    character*100          :: vname
    real          :: vdata(LIS_rc%npatch(n,LIS_rc%lsm_index))

Issue 4

Message

    ../surfacemodels/land/noahmp.3.6/NoahMP36_readrst.F90:27:0: error: token "." is not valid in preprocessor expressions

      #if (defined USE_NETCDF3 .OR. defined USE_NETCDF4)
      ^

Correction

Do not Fortran syntax for logical operators in preprocessor statements.

Use && for .and.

Use || for .or.

 

Bad
    #if (defined USE_NETCDF3 .OR. defined USE_NETCDF4)

Good
    #if (defined USE_NETCDF3 || defined USE_NETCDF4)

Issue 5

Message

    ../metforcing/ecmwf/read_ecmwf.F90:546.16:

               if ( result1 .eq. .true. ) glbdata1(iv,c) = 0.0
                    1

    Error: Logicals at (1) must be compared with .eqv. instead of .eq.

Correction

Compare logicals with .eqv. not with .eq.

 

Bad
    if ( result1 .eq. .true. ) glbdata1(iv,c) = 0.0

Good
    if ( result1 .eqv. .true. ) glbdata1(iv,c) = 0.0

Issue 6

Message

../dataassim/obs/SMOPSsm/SMOPSsm_Mod.F90:508.12:

       if ( SMOPSsm_struc(n)%useRealtime ) then
            1
Error: IF clause at (1) requires a scalar LOGICAL expression

Correction

You cannot use an integer in a boolean context.  Rewrite the conditional statement like this

 

       if ( SMOPSsm_struc(n)%useRealtime == 1 ) then

or change useRealtime to be a logical variable.

Issue 7

Message

../surfacemodels/land/ruc.3.7/ruc_driver_37.F90:532.14:

              zlvl*2.,sfcprs,tabs,q2,qcatm,rho,                &
              1
Error: Rank mismatch in argument 'z3d' at (1) (rank-3 and scalar)

Make sure that the actual arguments that correspond to dummy arrays are arrays not scalars.

0

Introduction

NCCS uses the Environment Modules Project (http://modules.sourceforge.net/), better known as the module command, to manage multiple versions of programs and libraries.  Users use it to manage their environment.  Users are not restricted to only the system level modulefiles provided by NCCS.  Users may create their own.  One specific use of custom modulefiles is to manage various LIS-related environments; e.g., to manage compilers and environment variables for compiling and running LIS 6 vs LIS 7 or for compiling and running with the Intel compilers vs the GCC compilers.  This post explains how to create a custom modulefile.

 

Please refer to the man pages for module(1) and modulefile(4) for more information.

Set up

First create a sub-directory named privatemodules in your home directory.

 

Then add this command to your .profile (or equivalent) login file.

 

module use --append $HOME/privatemodules

 

This will make your custom modulefiles available to the module program every time that you log on.

 

Note

Please note that on-line resources suggest using module load use.own to set up your custom modulefiles.  This command will create, if necessary, a privatemodules sub-directory in your home directory along with providing a sample modulefile named null.  Then this command will add your privatemodules sub-directory to the $MODULEPATH environment variable, making your custom modulefiles available to the module program.

However, that command does not work on all systems.  On some systems, like discover, you have to run module load $MODULESHOME/modulefiles/use.own.

Also note that if you subsequently run the module purge command, then you must rerun the module load use.own command to make your custom modulefiles available again.

Thus, instead of using module load use.own, this post suggests that you add module use --append $HOME/privatemodules to your .profile file.  This command neither relies on the system to find the use.own module nor is affected by a module purge command.

Creating a custom modulefile

Below is a sample custom modulefile.  It loads the modules and sets the environment variables needed to set up an Intel-18-based development environment.  To use it, copy the contents of the listing into $HOME/privatemodules/intel_18_0_3_222.  Then run the command:

 

discover07$ module load intel_18_0_3_222

 

After loading the custom modulefile, a module list command will report something similar to:

 

discover07$ module list
Currently Loaded Modulefiles:
  1) comp/intel-18.0.3.222   5) other/svn-1.9.5
  2) mpi/impi-18.0.3.222     6) other/vim-8.0
  3) tool/tview-2017.1.21    7) intel_18_0_3_222
  4) other/git-2.9.3

Listing of sample custom modulefile intel_18_0_3_222
#%Module1.0###################################################################

proc ModulesHelp { } {
    puts stderr "\t[module-info name] - loads the INTEL_18_0_3_222 env"
    puts stderr ""
    puts stderr "\tThe following env variables are set:"
    puts stderr "\t\tDEV_ENV"
    puts stderr ""
    puts stderr "\tThe following modules are loaded:"
    puts stderr "\t\tcomp/intel-18.0.3.222"
    puts stderr "\t\tmpi/impi-18.0.3.222"
    puts stderr "\t\ttool/tview-2017.1.21"
    puts stderr "\t\tother/git-2.9.3"
    puts stderr "\t\tother/svn-1.9.5"
    puts stderr "\t\tother/vim-8.0"
    puts stderr ""
    puts stderr "\tPython 2.7 is added to the search PATH."
}


conflict comp mpi


module-whatis     "loads the [module-info name] environment"


set modname     [module-info name]
set modmode     [module-info mode]


module load comp/intel-18.0.3.222
module load mpi/impi-18.0.3.222

module load tool/tview-2017.1.21
module load other/git-2.9.3
module load other/svn-1.9.5
module load other/vim-8.0


setenv   DEV_ENV         INTEL_18_0_3_222

prepend-path   PATH   /usr/local/other/SLES11.3/python/2.7.11/gcc-4.3.4/bin

Additional custom modulefiles

It is suggested that you keep your custom modulefiles as lean as possible.  Do not load more modules than are needed for a particular task.  The above sample custom modulefile loads only the modules needed to perform general software development with the Intel 18 compilers.

 

But Jim, I need Matlab, R, IDL, etc., what should I do?  Create additional custom modulefiles targeting the other tools and tasks that you need.  For example, create a custom modulefile for Matlab.  It should load only the modules needed to run Matlab, and it should set only environment variables required and related to running Matlab.

 

Warning
When you load too many unrelated modules into your environment, you run the risk of creating undetected incompatibilites between tools and libraries.  For example, you may end up loading three different NetCDF libraries.  And one day LIS runs, but the next day LIS cannot read its own restart file.

 

When dealing with multiple tasks using multiple custom modulefiles, you may either load and change the custom modulefiles, or you may launch multiple xterm s (one for each task) on discover, and load your different custom modulefiles in each xterm.

 

Example of loading and changing custom modulefiles
discover07$ module load lis_7_intel_14_0_3_174_sp3

# Work, work, work

discover07$ module purge
discover07$ module load my_matlab_mod

# Plot, plot, plot

Sample .profile and .bashrc files

Please refer to the man page for bash(1) for more information regarding the use of the .profile and .bashrc files.

 

Sample .profile file
# This file is read each time a login shell is started.

#
# Set LD_LIBRARY_PATH
#
# non LDT_*, non LIS_*, non LVT_* paths go here
#export LD_LIBRARY_PATH=$ADDITIONAL_PATHS:$LD_LIBRARY_PATH


#
# Set environment
#
export EDITOR=vim
export SVN_EDITOR=vim
export TVDSVRLAUNCHCMD=ssh


#
# Set PATH
#
# non LDT_*, non LIS_*, non LVT_* paths go here
#export PATH=$ADDITIONAL_PATHS:$PATH


#
# Set modules
#
# load modules manually from the command line
# to keep your login environment clean
module use --append $HOME/privatemodules


ulimit -s unlimited

Sample .bashrc file
# This file is for interactive shells.
# On some systems, it is loaded in some non-interactive cases.
# Return if this is not an interactive shell.
[ "$PS1" ] || return


#
# Set aliases
#
unalias -a
alias ls="ls --color=auto"


#
# Set prompt
#
export PS1="\[\e[36m\]\h\[\e[00m\][\!]\$ "


#
# Set environment
#
export LS_COLORS='no=00:fi=00:di=33:ln=36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=32:*.cmd=32:*.exe=32:*.com=32:*.btm=32:*.bat=32:*.sh=32:*.csh=32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35'

export MANPATH=$MANPATH:/usr/slurm/share/man

LIS development environments

Here are several custom modulefiles for LIS development.

LIS 7 with Intel 14 compilers

Note
This is an old environment.  It is here for reference.  Please use the LIS 7 with Intel 18 compilers environment.

Listing of custom modulefile lis_7_intel_14_0_3_174_sp3
#%Module1.0###################################################################

proc ModulesHelp { } {
    puts stderr "\t[module-info name] - loads the LIS_7_INTEL_14_0_3_174_SP3 env"
    puts stderr ""
    puts stderr "\tThe following env variables are set:"
    puts stderr "\t\tDEV_ENV"
    puts stderr "\t\tLIS_ARCH"
    puts stderr "\t\tLIS_SPMD"
    puts stderr "\t\tLIS_FC"
    puts stderr "\t\tLIS_CC"
    puts stderr "\t\tLIS_JASPER"
    puts stderr "\t\tLIS_GRIBAPI"
    puts stderr "\t\tLIS_NETCDF"
    puts stderr "\t\tLIS_HDF4"
    puts stderr "\t\tLIS_HDFEOS"
    puts stderr "\t\tLIS_HDF5"
    puts stderr "\t\tLIS_MODESMF"
    puts stderr "\t\tLIS_LIBESMF"
    puts stderr "\t\tLIS_MINPACK"
    puts stderr "\t\tLIS_CRTM"
    puts stderr "\t\tLIS_CRTM_PROF"
    puts stderr "\t\tLIS_CMEM"
    puts stderr "\t\tLIS_LAPACK"
    puts stderr "\t\tLDT_ARCH"
    puts stderr "\t\tLDT_FC"
    puts stderr "\t\tLDT_CC"
    puts stderr "\t\tLDT_JASPER"
    puts stderr "\t\tLDT_GRIBAPI"
    puts stderr "\t\tLDT_NETCDF"
    puts stderr "\t\tLDT_HDF4"
    puts stderr "\t\tLDT_HDFEOS"
    puts stderr "\t\tLDT_HDF5"
    puts stderr "\t\tLDT_MODESMF"
    puts stderr "\t\tLDT_LIBESMF"
    puts stderr "\t\tLVT_ARCH"
    puts stderr "\t\tLVT_FC"
    puts stderr "\t\tLVT_CC"
    puts stderr "\t\tLVT_JASPER"
    puts stderr "\t\tLVT_GRIBAPI"
    puts stderr "\t\tLVT_NETCDF"
    puts stderr "\t\tLVT_HDF4"
    puts stderr "\t\tLVT_HDFEOS"
    puts stderr "\t\tLVT_HDF5"
    puts stderr "\t\tLVT_MODESMF"
    puts stderr "\t\tLVT_LIBESMF"
    puts stderr "\t\tLVT_GDAL"
    puts stderr "\t\tLVT_FORTRANGIS"
    puts stderr ""
    puts stderr "\tThe following modules are loaded:"
    puts stderr "\t\tcomp/intel-14.0.3.174"
    puts stderr "\t\tmpi/impi-5.0.3.048"
    puts stderr "\t\ttool/tview-8.11.0.0"
    puts stderr "\t\tother/git-2.9.3"
    puts stderr "\t\tother/svn-1.8.13"
    puts stderr "\t\tother/vim-8.0"
    puts stderr ""
    puts stderr "\tPython 2.7 is added to the search PATH."
}


conflict comp mpi


module-whatis     "loads the [module-info name] environment"


set modname     [module-info name]
set modmode     [module-info mode]


module load comp/intel-14.0.3.174
module load mpi/impi-5.0.3.048

module load tool/tview-8.11.0.0
module load other/git-2.9.3
module load other/svn-1.8.13
module load other/vim-8.0


set   def_lis_jasper      /discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3
set   def_lis_gribapi     /discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3
set   def_lis_netcdf      /discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3
set   def_lis_hdf4        /discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3
set   def_lis_hdfeos      /discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3
set   def_lis_hdf5        /discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3
set   def_lis_modesmf     /discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default
set   def_lis_libesmf     /discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default
set   def_lis_minpack     /discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
set   def_lis_crtm        /discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev
set   def_lis_crtm_prof   /discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_11_1_038/
set   def_lis_cmem        /discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117
set   def_lis_lapack      /usr/local/other/SLES11/lapack/3.3.1/intel-12.1.0.233/lib
set   def_lvt_gdal        /discover/nobackup/projects/lis/libs/gdal/2.0.0
set   def_lvt_fortrangis  /discover/nobackup/projects/lis/libs/fortrangis/2.4


setenv   DEV_ENV         LIS_7_INTEL_14_0_3_174_SP3
setenv   LIS_ARCH        linux_ifc
setenv   LIS_SPMD        parallel
setenv   LIS_FC          mpif90
setenv   LIS_CC          mpicc
setenv   LIS_JASPER      $def_lis_jasper
setenv   LIS_GRIBAPI     $def_lis_gribapi
setenv   LIS_NETCDF      $def_lis_netcdf
setenv   LIS_HDF4        $def_lis_hdf4
setenv   LIS_HDFEOS      $def_lis_hdfeos
setenv   LIS_HDF5        $def_lis_hdf5
setenv   LIS_MODESMF     $def_lis_modesmf
setenv   LIS_LIBESMF     $def_lis_libesmf
setenv   LIS_MINPACK     $def_lis_minpack
setenv   LIS_CRTM        $def_lis_crtm
setenv   LIS_CRTM_PROF   $def_lis_crtm_prof
setenv   LIS_CMEM        $def_lis_cmem
setenv   LIS_LAPACK      $def_lis_lapack


setenv   LDT_ARCH      linux_ifc
setenv   LDT_FC        mpif90
setenv   LDT_CC        mpicc
setenv   LDT_JASPER    $def_lis_jasper
setenv   LDT_GRIBAPI   $def_lis_gribapi
setenv   LDT_NETCDF    $def_lis_netcdf
setenv   LDT_HDF4      $def_lis_hdf4
setenv   LDT_HDFEOS    $def_lis_hdfeos
setenv   LDT_HDF5      $def_lis_hdf5
setenv   LDT_MODESMF   $def_lis_modesmf
setenv   LDT_LIBESMF   $def_lis_libesmf


setenv   LVT_ARCH        linux_ifc
setenv   LVT_FC          mpif90
setenv   LVT_CC          mpicc
setenv   LVT_JASPER      $def_lis_jasper
setenv   LVT_GRIBAPI     $def_lis_gribapi
setenv   LVT_NETCDF      $def_lis_netcdf
setenv   LVT_HDF4        $def_lis_hdf4
setenv   LVT_HDFEOS      $def_lis_hdfeos
setenv   LVT_HDF5        $def_lis_hdf5
setenv   LVT_MODESMF     $def_lis_modesmf
setenv   LVT_LIBESMF     $def_lis_libesmf
setenv   LVT_GDAL        $def_lvt_gdal
setenv   LVT_FORTRANGIS  $def_lvt_fortrangis


prepend-path   LD_LIBRARY_PATH   "$def_lis_hdf4/lib:$def_lis_hdf5/lib:$def_lis_libesmf:$def_lis_netcdf/lib:$def_lis_gribapi/lib"
prepend-path   PATH   /usr/local/other/SLES11.3/python/2.7.11/gcc-4.3.4/bin
prepend-path   PATH   /home/jvgeiger/local/nccmp-1.3.0/bin
prepend-path   PATH   "$def_lis_netcdf/bin:$def_lis_gribapi/bin"

LIS 7 with Intel 18 compilers

This is the recommended environment for compiling and running LIS on discover.

 

Warning

The 26 Apr 2018 version of this article provided a custom modulefile for compiling and running LIS with Intel 17 compilers.  This development/running environment produced  bad results for "2d ensemble gridspace" output.  We could not find an error in the LIS code that led to the bad results, and LIS produced good results after upgrading to Intel 18.

Stop using the lis_7_intel_17_0_4_196 modulefile and switch to using the lis_7_intel_18_0_3_222 modulefile.


Listing of the custom modulefile lis_7_intel_18_0_3_222
#%Module1.0###################################################################

proc ModulesHelp { } {
    puts stderr "\t[module-info name] - loads the LIS_7_INTEL_18_0_3_222 env"
    puts stderr ""
    puts stderr "\tThe following env variables are set:"
    puts stderr "\t\tDEV_ENV"
    puts stderr "\t\tLIS_ARCH"
    puts stderr "\t\tLIS_SPMD"
    puts stderr "\t\tLIS_FC"
    puts stderr "\t\tLIS_CC"
    puts stderr "\t\tLIS_JASPER"
    puts stderr "\t\tLIS_ECCODES"
    puts stderr "\t\tLIS_NETCDF"
    puts stderr "\t\tLIS_HDF4"
    puts stderr "\t\tLIS_HDFEOS"
    puts stderr "\t\tLIS_HDF5"
    puts stderr "\t\tLIS_MODESMF"
    puts stderr "\t\tLIS_LIBESMF"
    puts stderr "\t\tLIS_MINPACK"
    puts stderr "\t\tLIS_CRTM"
    puts stderr "\t\tLIS_CRTM_PROF"
    puts stderr "\t\tLIS_CMEM"
    puts stderr "\t\tLIS_LAPACK"
    puts stderr "\t\tLDT_ARCH"
    puts stderr "\t\tLDT_FC"
    puts stderr "\t\tLDT_CC"
    puts stderr "\t\tLDT_JASPER"
    puts stderr "\t\tLDT_ECCODES"
    puts stderr "\t\tLDT_NETCDF"
    puts stderr "\t\tLDT_HDF4"
    puts stderr "\t\tLDT_HDFEOS"
    puts stderr "\t\tLDT_HDF5"
    puts stderr "\t\tLDT_MODESMF"
    puts stderr "\t\tLDT_LIBESMF"
    puts stderr "\t\tLDT_GDAL"
    puts stderr "\t\tLDT_FORTRANGIS"
    puts stderr "\t\tLDT_LIBGEOTIFF"
    puts stderr "\t\tLVT_ARCH"
    puts stderr "\t\tLVT_FC"
    puts stderr "\t\tLVT_CC"
    puts stderr "\t\tLVT_JASPER"
    puts stderr "\t\tLVT_ECCODES"
    puts stderr "\t\tLVT_NETCDF"
    puts stderr "\t\tLVT_HDF4"
    puts stderr "\t\tLVT_HDFEOS"
    puts stderr "\t\tLVT_HDF5"
    puts stderr "\t\tLVT_MODESMF"
    puts stderr "\t\tLVT_LIBESMF"
    puts stderr "\t\tLVT_GDAL"
    puts stderr "\t\tLVT_FORTRANGIS"
    puts stderr ""
    puts stderr "\tThe following modules are loaded:"
    puts stderr "\t\tother/comp/gcc-7.3"
    puts stderr "\t\tcomp/intel-18.0.3.222"
    puts stderr "\t\tmpi/impi-18.0.3.222"
    puts stderr "\t\ttool/tview-2017.1.21"
    puts stderr "\t\tother/git-2.18.0"
    puts stderr "\t\tother/svn-1.9.5"
    puts stderr "\t\tother/vim-8.0"
    puts stderr ""
    puts stderr "\tPython 2.7 is added to the search PATH."
}


conflict comp mpi


module-whatis     "loads the [module-info name] environment"


set modname     [module-info name]
set modmode     [module-info mode]


module load other/comp/gcc-7.3
module load comp/intel-18.0.3.222
module load mpi/impi-18.0.3.222

module load tool/tview-2017.1.21
module load other/git-2.18.0
module load other/svn-1.9.5
module load other/vim-8.0


set   def_lis_jasper      /discover/nobackup/projects/lis/libs/jasper/2.0.14_intel-18.0.3.222
set   def_lis_eccodes     /discover/nobackup/projects/lis/libs/ecCodes/2.7.0_intel-18.0.3.222
set   def_lis_netcdf      /discover/nobackup/projects/lis/libs/netcdf/4.5.0_intel-18.0.3.222
set   def_lis_hdf4        /discover/nobackup/projects/lis/libs/hdf4/4.2.13_intel-18.0.3.222
set   def_lis_hdfeos      /discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-18.0.3.222
set   def_lis_hdf5        /discover/nobackup/projects/lis/libs/hdf5/1.10.1_intel-18.0.3.222
set   def_lis_modesmf     /discover/nobackup/projects/lis/libs/esmf/7.1.0r_intel-18.0.3.222_impi-18.0.3.222/mod/modO/Linux.intel.64.intelmpi.default
set   def_lis_libesmf     /discover/nobackup/projects/lis/libs/esmf/7.1.0r_intel-18.0.3.222_impi-18.0.3.222/lib/libO/Linux.intel.64.intelmpi.default
set   def_lis_minpack     /discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
set   def_lis_crtm        /discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev_intel_18_0_3_222
set   def_lis_crtm_prof   /discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_18_0_3_222
set   def_lis_cmem        /discover/nobackup/projects/lis/libs/LIS-MEM/intel_18_0_3_222
set   def_lis_lapack      /usr/local/other/SLES11/lapack/3.3.1/intel-12.1.0.233/lib
set   def_lvt_gdal        /discover/nobackup/projects/lis/libs/gdal/2.4.1_intel-18.0.3.222
set   def_lvt_fortrangis  /discover/nobackup/projects/lis/libs/fortrangis/2.4_intel-18.0.3.222
set   def_ldt_libgeotiff  /discover/nobackup/projects/lis/libs/geotiff/1.4.3_intel-18.0.3.222

setenv   DEV_ENV         LIS_7_INTEL_18_0_3_222
setenv   LIS_ARCH        linux_ifc
setenv   LIS_SPMD        parallel
setenv   LIS_FC          mpif90
setenv   LIS_CC          mpicc
setenv   LIS_JASPER      $def_lis_jasper
setenv   LIS_ECCODES     $def_lis_eccodes
setenv   LIS_NETCDF      $def_lis_netcdf
setenv   LIS_HDF4        $def_lis_hdf4
setenv   LIS_HDFEOS      $def_lis_hdfeos
setenv   LIS_HDF5        $def_lis_hdf5
setenv   LIS_MODESMF     $def_lis_modesmf
setenv   LIS_LIBESMF     $def_lis_libesmf
setenv   LIS_MINPACK     $def_lis_minpack
setenv   LIS_CRTM        $def_lis_crtm
setenv   LIS_CRTM_PROF   $def_lis_crtm_prof
setenv   LIS_CMEM        $def_lis_cmem
setenv   LIS_LAPACK      $def_lis_lapack


setenv   LDT_ARCH        linux_ifc
setenv   LDT_FC          mpif90
setenv   LDT_CC          mpicc
setenv   LDT_JASPER      $def_lis_jasper
setenv   LDT_ECCODES     $def_lis_eccodes
setenv   LDT_NETCDF      $def_lis_netcdf
setenv   LDT_HDF4        $def_lis_hdf4
setenv   LDT_HDFEOS      $def_lis_hdfeos
setenv   LDT_HDF5        $def_lis_hdf5
setenv   LDT_MODESMF     $def_lis_modesmf
setenv   LDT_LIBESMF     $def_lis_libesmf
setenv   LDT_GDAL        $def_lvt_gdal
setenv   LDT_FORTRANGIS  $def_lvt_fortrangis
setenv   LDT_LIBGEOTIFF  $def_ldt_libgeotiff


setenv   LVT_ARCH        linux_ifc
setenv   LVT_FC          mpif90
setenv   LVT_CC          mpicc
setenv   LVT_JASPER      $def_lis_jasper
setenv   LVT_ECCODES     $def_lis_eccodes
setenv   LVT_NETCDF      $def_lis_netcdf
setenv   LVT_HDF4        $def_lis_hdf4
setenv   LVT_HDFEOS      $def_lis_hdfeos
setenv   LVT_HDF5        $def_lis_hdf5
setenv   LVT_MODESMF     $def_lis_modesmf
setenv   LVT_LIBESMF     $def_lis_libesmf
setenv   LVT_GDAL        $def_lvt_gdal
setenv   LVT_FORTRANGIS  $def_lvt_fortrangis


prepend-path   LD_LIBRARY_PATH   "$def_ldt_libgeotiff/lib:$def_lvt_gdal/lib:$def_lis_hdf4/lib:$def_lis_hdf5/lib:$def_lis_libesmf:$def_lis_netcdf/lib:$def_lis_eccodes/lib:$def_lis_jasper/lib64"
prepend-path   PATH   /usr/local/other/SLES11.3/python/2.7.11/gcc-4.3.4/bin
prepend-path   PATH   /home/jvgeiger/local/nccmp-1.8.2.1/bin
prepend-path   PATH   /discover/nobackup/projects/lis/libs/ncview/2.1.7_intel-18.0.3.222/bin
prepend-path   PATH   "$def_lis_netcdf/bin:$def_lis_eccodes/bin"

1

Using svn+ssh method to access LIS repository

Introduction

The Progress/CVSACL Security Maintenance that was completed on 1 Apr 2015 prompted an upgrade to the Subversion package installed on discover to access the LIS software repository hosted on progress.

 

Please see the notes in the LIS ModelingGuru article "Subversion upgrade 1.8.13" for more details.

https://modelingguru.nasa.gov/community/atmospheric/lis/blog/2015/04/01/subversi on-upgrade-1813

 

This security update raised two issues:

 

  1. Due to security concerns, the password caching feature of Subversion was disabled in this upgrade.  Thus every Subversion command that accesses the LIS software repository (svn update, svn log, etc.) prompts you for your Subversion password.

  2. Any SSL-based communications with the progress server must use TLS 1.2.  Using older versions of Subversion (and OpenSSL) on remote systems to access the LIS software repository results in error messages similar to this:

     

    sh$ svn checkout https://progress.nccs.nasa.gov/svn/lis/7/public7.1svn: OPTIONS of 'https://progress.nccs.nasa.gov/svn/lis/7/public7.1': SSL
    handshake failed: SSL error: sslv3 alert handshake failure
    (https://progress.nccs.nasa.gov)
    

     

    It is often difficult to get Subversion (and OpenSSL) upgraded on remote systems.

 

This article explains how to use Subversion's svn+ssh method along with ssh-agent to resolve these two problems.

 

This article is intended for LIS repository users who are either troubled by issue 1 or stopped by issue 2.  If neither issue affects you, then you may continue to follow the instructions in the above "Subversion upgrade 1.8.13" article.

Set up

This method does require some set up from each user before it can be used.  Please see the instructions on https://progress.nccs.nasa.gov/trac/admin/wiki/QuickStart

svn+ssh method

Since this method does not use SSL-based communications, it is not affected by the above Progress/CVSACL Security Maintenance.

Access from discover

Once you have both your public SSH key (~/.ssh/id_rsa.pub) uploaded to progress via the "keyupload" link and the progressdirect virtual host configured, you may access the LIS software repository via commands like this:

 

sh$ svn checkout svn+ssh://progressdirect/svn/lis/7/public7.1

Access from a remote system

Once you have both your public SSH key (~/.ssh/id_rsa.pub) uploaded to progress via the "keyupload" link and and your SSH tunnel established, you may access the LIS software repository via commands like this:

 

sh$ svn checkout svn+ssh://ptunnel/svn/lis/7/public7.1

 

Please note that you must always establish the SSH tunnel before you try to run any Subversion commands.  But once the SSH tunnel is established, it is good for your entire login session.

ssh-agent

Best practices dictate that you protect your private SSH key (~/.ssh/id_rsa) with a password.  You will be prompted to create a password when you create your SSH keys via the ssh-keygen command.  Of course, a password-protected key requires unlocking each time it is used, so every Subversion command that accesses the LIS software repository (svn update, svn log, etc.) via the svn+ssh method will prompt you for your key's password.  You can eliminate the need to repeatedly enter your key's password by using the ssh-agent command.  The simplest way to use ssh-agent is to run it right after you log onto discover or your remote system:

 

sh$ ssh-agent bash

 

Note that bash is just an example.  You may specify any shell that you prefer.

After you have run ssh-agent, you must load your private SSH key:

 

sh$ ssh-add ~/.ssh/id_rsa

 

Here you will be prompted for the key's password.  After you provide the password, ssh-agent will work with SSH to manage access to your private SSH key.  Thus, you will no longer be prompted by Subversion for a password when you run any commands that access the repository.

Converting from https to svn+ssh

To convert a working directory from using the https method to using the svn+ssh method, you must run the svn relocate command.

 

For example, you have a working copy of the public 7.1 branch obtained via:

 

sh$ svn checkout https://progress.nccs.nasa.gov/svn/lis/7/public7.1

 

To convert it to using svn+ssh, go into the working directory and run:

 

sh$ svn relocate svn+ssh://progressdirect/svn/lis/7/public7.1

 

or:

 

sh$ svn relocate svn+ssh://ptunnel/svn/lis/7/public7.1

 

whichever is appropriate for your situation.

 

Summary

  1. log on
  2. run ssh-agent
  3. run ssh-add
  4. establish tunnel, if neccessary
  5. use Subversion
0

The Progress/CVSACL Security Maintenance that was completed on 1 Apr 2015 prompted an upgrade to the Subversion package installed on discover to access the LIS software repository hosted on progress.

 

Currently, the new Subversion package is available only on the SP3 front-end systems.  Thus you must log onto discover-sp3 to use Subversion.  You may do development and run LIS on either an SP1 or an SP3 system, but you must be on an SP3 system to run svn.

 

To use the new Subversion package, you must load the other/svn-1.8.13 module:

 

discover07$ module load other/svn-1.8.13

 

After you have loaded the new Subversion module.  You must run svn upgrade in any existing working directories.  If you forget, Subversion will remind you, as seen below.

 

discover07$ svn update
svn: E155036: Please see the 'svn upgrade' command
svn: E155036: The working copy at '/gpfsm/dnb12s1/jvgeiger/SRC/7/d7'
is too old (format 10) to work with client version '1.8.13 (r1667537)' (expects format 31). You need to upgrade the working copy first.

 

Simply run the svn upgrade command.

 

discover07$ svn upgrade
Upgraded '.'
Upgraded 'plugins'
Upgraded 'offline'
Upgraded 'params'
Upgraded 'params/gfrac'
Upgraded 'params/gfrac/VIIRSDaily'
Upgraded 'params/gfrac/ALMIPII'
Upgraded 'params/gfrac/NESDISWeekly'
Upgraded 'params/gfrac/SPORTDaily'
.
.
.

 

Please note that a fresh checkout from the LIS software repository does not require that you to run the svn upgrade command.

 

Finally, you must accept the new SSL certificate.  You must run an svncommand that accesses the repository.  When prompted, permanently accept the certificate by selecting p.  For example, run svn update in a working directory.

 

discover07$ svn update
Updating '.':
Error validating server certificate for 'https://progress.nccs.nasa.gov:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: www.nccs.nasa.gov
 - Valid: from Oct 22 12:00:01 2013 GMT until Apr 21 12:00:00 2016 GMT
 - Issuer: www.digicert.com, DigiCert Inc, US
 - Fingerprint: A3:4F:7A:E6:E5:AE:55:BB:13:F2:56:48:63:33:76:EF:CC:9C:D7:DE
(R)eject, accept (t)emporarily or accept (p)ermanently? p
2
Last updated: Wed Sep 14 15:39:40 EDT 2016

Introduction

NCCS is adding new hardware, containing Intel Haswell nodes, running the SUSE Linux Enterprise Server (SLES) 11 SP3 operating system.  These notes describe how to recompile LIS 7 to run on this operating system.

 

For compiling LIS on the SLES 11 SP1 nodes, please see the post "Compiling LIS 6 and LIS 7 --- SLURM migration".

Modules

Use these modules to compile LIS 7:

 

module load comp/intel-14.0.3.174
module load mpi/impi-5.0.3.048

Libraries

Specify these variables/libraries in your .profile (or equivalent) file to configure LIS 7, LDT, and LVT (here written in bash syntax):

 

Note: New libraries for MINPACK, CRTM, CRTM_PROF, and CMEM, have not yet been recompiled.

 

export DEV_ENV=7_INTEL_14_SP3
export LIS_ARCH=linux_ifc
export LIS_SPMD=parallel
export LIS_FC=mpif90
export LIS_CC=mpicc
export LIS_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3
export LIS_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3
export LIS_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3
export LIS_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3
export LIS_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3
export LIS_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3
export LIS_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default
export LIS_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default
export LIS_MINPACK=
export LIS_CRTM=
export LIS_CRTM_PROF=
export LIS_CMEM=
export LIS_LAPACK=/usr/local/other/SLES11/lapack/3.3.1/intel-12.1.0.233/lib

export LVT_ARCH=linux_ifc
export LVT_FC=mpif90
export LVT_CC=mpicc
export LVT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3
export LVT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3
export LVT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3
export LVT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3
export LVT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3
export LVT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3
export LVT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default
export LVT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default

export LVT_GDAL=/discover/nobackup/projects/lis/libs/gdal/2.0.0

export LVT_FORTRANGIS=/discover/nobackup/projects/lis/libs/fortrangis/2.4

export LDT_ARCH=linux_ifc export LDT_FC=mpif90 export LDT_CC=mpicc export LDT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3 export LDT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3 export LDT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3 export LDT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3 export LDT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3 export LDT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3 export LDT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default export LDT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default export LD_LIBRARY_PATH=${LIS_HDF4}/lib:${LIS_HDF5}/lib:${LIS_LIBESMF}:${LIS_NETCDF}/lib:${LIS_GRIBAPI}/lib:$LD_LIBRARY_PATH
0

Last updated: Thu Dec 18 13:40:12 EST 2014

Introduction

NCCS is migrating the queuing system used on discover from PBS to SLURM.

 

NCCS has identified the following libraries as being incompatible with SLURM.

 

  • Intel MPI:
    • mpi/impi-3.1.*
    • mpi/impi-3.2.011
    • mpi/impi-3.2.1.009
    • mpi/impi-4.*-beta
  • MVAPICH2:
    • other/mpi/mvapich2-1.6rc2/intel-11.1.072
    • other/mpi/mvapich2-1.7*
    • other/mpi/mvapich2-1.8/*
    • other/mpi/mvapich2-1.8a2/*
  • OpenMPI:
    • mpi/openmpi-1.2.5/intel-9
    • mpi/openmpi-1.2.5/intel-10
    • other/mpi/openmpi/1.4.*
    • other/mpi/openmpi/1.6-gcc_4.8-20120401_nag-5.3-854
    • other/mpi/openmpi/1.6.0*
    • other/mpi/openmpi/1.6.3*
    • other/mpi/openmpi/1.6.4*

 

On discover, LIS 6 uses mpi/impi-3.2.011, and LIS 7 uses mpi/impi-4.*-beta.  The notes below describe how to recompile LIS with libraries compatible with SLURM.


Compiling LIS 6

Using Intel 11 and IMPI 3

Modules

Use these modules to compile LIS 6:

 

module load comp/intel-11.1.038
module load mpi/impi-3.2.2.006

Libraries

Specify these libraries in the LIS 6 configure.lis file:

 

LIB_ESMF = /discover/nobackup/projects/lis/libs/esmf/3_1_0rp3_intel_11_1_038_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default
MOD_ESMF = /discover/nobackup/projects/lis/libs/esmf/3_1_0rp3_intel_11_1_038_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
INC_JASPER = /usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038/include
LIB_JASPER = /usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038/lib

 

Note that the other libraries specified in the default arch/configure.lis.linux_ifc are correct.

If you experience errors while compiling or linking the LIS 6 executable on discover, then please consult the latest revision of the arch/configure.lis.linux_ifc file.

 


Compiling LIS 7, LDT, and LVT

Using Intel 11 and IMPI 3

Modules

Use these modules to compile LIS 7, LDT, and LVT:

 

module load comp/intel-11.1.069
module load mpi/impi-3.2.2.006
module load lib/mkl-9.1.023

Libraries

Specify these variables/libraries in your .profile (or equivalent) file to configure LIS 7, LDT, and LVT (here written in bash syntax):

 

export LIS_ARCH=linux_ifc
export LIS_SPMD=parallel
export LIS_FC=mpif90
export LIS_CC=mpicc
export LIS_JASPER=/usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038
export LIS_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.9.16_intel_11_1_069
export LIS_NETCDF=/discover/nobackup/projects/lis/libs/netcdf4/4.3.0_intel_11_1_069
export LIS_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.9_intel_11_1_069
export LIS_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.18v1.00_intel_11_1_069
export LIS_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_11_1_069
export LIS_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
export LIS_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default
export LIS_MINPACK=/discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
export LIS_CRTM=/discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev
export LIS_CRTM_PROF=/discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_11_1_038/
export LIS_CMEM=/discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117

export LVT_ARCH=linux_ifc
export LVT_SPMD=parallel
export LVT_FC=mpif90
export LVT_CC=mpicc
export LVT_JASPER=/usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038
export LVT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.9.16_intel_11_1_069
export LVT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf4/4.3.0_intel_11_1_069
export LVT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.9_intel_11_1_069
export LVT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.18v1.00_intel_11_1_069
export LVT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_11_1_069
export LVT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
export LVT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default

export LDT_ARCH=linux_ifc
export LDT_SPMD=parallel
export LDT_FC=mpif90
export LDT_CC=mpicc
export LDT_JASPER=/usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038
export LDT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.9.16_intel_11_1_069
export LDT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf4/4.3.0_intel_11_1_069
export LDT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.9_intel_11_1_069
export LDT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.18v1.00_intel_11_1_069
export LDT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_11_1_069
export LDT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
export LDT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default

Using Intel 13 and IMPI 4

Modules

Use these modules to compile LIS 7, LDT, and LVT:

 

comp/intel-13.1.3.192
mpi/impi-4.0.3.008

Libraries

Specify these variables/libraries in your .profile (or equivalent) file to configure LIS 7, LDT, and LVT (here written in bash syntax):

 

export LIS_ARCH=linux_ifc
export LIS_SPMD=parallel
export LIS_FC=mpif90
export LIS_CC=mpicc
export LIS_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192
export LIS_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117
export LIS_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2r4_noNETCDF_intel_13_0_1_117
export LIS_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos
export LIS_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117
export LIS_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/mod/modO/Linux.intel.64.intelmpi.default
export LIS_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default
export LIS_MINPACK=/discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
export LIS_CRTM=/discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev
export LIS_CRTM_PROF=/discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_11_1_038/
export LIS_CMEM=/discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117
export LIS_MEM=/discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117
export LIS_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192

export LVT_ARCH=linux_ifc
export LVT_FC=mpif90
export LVT_CC=mpicc
export LVT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192
export LVT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117
export LVT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2r4_noNETCDF_intel_13_0_1_117/
export LVT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos/
export LVT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117
export LVT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/mod/modO/Linux.intel.64.intelmpi.default
export LVT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default
export LVT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192

export LDT_ARCH=linux_ifc
export LDT_FC=mpif90
export LDT_CC=mpicc
export LDT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192
export LDT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117
export LDT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2r4_noNETCDF_intel_13_0_1_117/
export LDT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos/
export LDT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117
export LDT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/mod/modO/Linux.intel.64.intelmpi.default
export LDT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default
export LDT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default/:/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192/lib:/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117/lib:/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117/lib:/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192/lib

Submitting a PBS job script into SLURM

The SLURM project provides wrappers for PBS commands, so your existing scripts will still work.  All that you must do is add one line to the top of your script.

 

For example, if you used to run

 

qsub lis.job

 

then first edit the lis.job script.  Add this line to the top of that file:

 

#!/bin/sh

 

Then run

 

qsub lis.job

 

as before.

 


Using native SLURM

Switching from PBS commands to SLURM commands is easy. NCCS read "First Introduction to Using SLURM on Discover" [http://www.nccs.nasa.gov/images/SLURM-intro_3-mod.pdf]. These slides show how to easily convert a PBS job script into a SLURM job script.

 

NCCS has hosted two brown bags (http://www.nccs.nasa.gov/list_brown_bags.html) discussing SLURM.  Please read "Introduction to the SLURM resource manager" [http://www.nccs.nasa.gov/images/intro-to-slurm-20131218.pdf]. These slides provide a good overview of the basic SLURM commands.

 

Please note that the second brown bag on SLURM has not yet been posted.

 


Obtaining an interactive session with X11 fowarding

To obtain an interactive session with X11 fowarding enabled, you run the xsub command as with PBS.  For example:

 

xsub -I -l select=1:ncpus=16,walltime=1:00:00 -W group_list=s0942

 

will give you one node with 16 cpu for 1 hour.

0

Latest DAS in GEOS

Posted by Carlos Cruz Jun 29, 2010

A few months ago Jules published the required steps to setup and run the GEOS5-DAS. Also, additional edits were required (provided at the end of the post) to a handful of files for the DAS to run without any problems. This setup also used 1/2 degree resolution background files thus making portability experiments inefficient. Fortunately the latest DAS tags, GEOSadas-5_5_2, is  the easiest tag I have had to build/run in a long time:

 

cvs co -r GEOSadas-5_5_2 GEOSadas
cd GEOSadas
source g5_modules
make install
cd ../Linux/bin
fvsetup

 

The greatest advance over earlier tags is that fvsetup actually contains defaults that work and the background files used are 2degree resolution making it feasible for portability experiments. Note that the default settings may not be optimal but they will allow you to get observational and background files. (Getting the latter is perhaps the biggest challenge in getting the DAS setup.)

0

Previously I have discussed the importance of build time for developer productivity and how parallelism and optimization flags can significantly improve performance of the build.   Today I will examine other techniques within the context of a motivating example from modelE.

 

One of the developers had pointed out that for the chemistry+aerosol configuration of the model, just one particular file was taking a disproportionate amount of time to compile.   Worse, due to various dependencies, a majority of changes made to the model ultimately force recompilation of this file.     I decided to investigate to see what could be done to ameliorate the situation.

 

The first step is of course to instrument the build so that we have some hard data to analyze.   Since I wanted to do this a bit more systematically than I had in the past, I actually went to read the manual on the Unix "time" command so that I could produce a clean profile without the hand editing I have done in the past.    I settled with the following lines in the Makefile:

 

 

COMPILATION_PERF =~/compilationPerf.dat
%.o:%.f:
     /usr/bin/time -f %e, $< -a -o $(COMPILATION_PERF) $(F90)...
 
It is important to give the path to the time command because some shells provide their own time command with a different syntax.   The build will now append 1 line to my dat file for each file that is compiled.  The line will have elapsed time (%e) in seconds and the file name separated by a comma for easy import into a spreadsheet.

Baseline Performance

 

With the changes to the Makefile accomplished, I made a fresh build of the model for the desired configuration and sorted the results:
File                                               

Compilation Time (sec)

TRACERS_DRV.f155.58

TRCHEM_master.f

28.73

DIAG.f25.70
DEFACC.f20.47
RAD_DRV.f18.56
RADIATION.f18.50
CLOUDS2.f13.39
CLOUDS2_DRV.f  8.03
GHY.f  7.14
DIAG_PRT.f  6.89
Total (139 files)427.65
Thus, just one file, TRACERS_DRV.f, was taking over 35% of the compilation time!

Drilling deeper

Taking a closer look at the expensive file, I found that it was just a collection of independent external subroutines.   Since there is no containing F90 module, it is a simple matter to split the file into ~10 separate files with one procedure in each one and then re-instrument.   I found the following:
Procedure                                     Compilation Time (sec)
INIT_IJTS_DIAG76.71
INIT_JLS_DIAG39.09
INIT_TRACER_CONS_DIAG

18.42

TRACER_IC13.88
INIT_TRACER  2.66
SET_TRACER_2DSOURCE  2.01
TRACER_3DSOURCE  1.98
SET_DIAG_RAD  0.36
GET_COND_FACTOR  0.31
DAILY_TRACER  0.23
INIT_IJLTS_DIAG  0.20
GET_LATLON_MASK  0.17
TOTAL160.92
We can make several interesting observations from this data.   First, unsurprisingly a small minority of procedures is consuming the bulk of the compilation.    Second, simply by splitting the procedures into separate files, parallel compilation could in theory reduce the build time for this set of procedures by about 50%, though we could have hoped for more.   And finally, from the names, we might guess that the first 5 files in the list are all only involved during the  initialization phase.   (Not entirely obvious for TRACER_IC, but it is also an initialization procedure.)
Why does it matter that the expensive procedures are "only" initialization procedures?   The performance of initialization procedures is unimportant for practical purposes since they are only executed once whereas most procedures are executed once per iteration (or more).    As it turns out the amount of computational work in these procedures is quite small anyway with virtually no 3D loops - just lots and lots of logic.

Easing off on optimization

Thus, if we simply crank down the optimization for these top 5 offenders to -O0, we might significantly reduce the compilation time with virtually no consequence to the run performance.  And indeed we see the following:
Procedure                                     Compilation Time (sec)
INIT_IJTS_DIAG0.36
INIT_JLS_DIAG0.29
INIT_TRACER_CONS_DIAG

0.24

TRACER_IC1.16
INIT_TRACER0.22
As these results show, the performance bottleneck is completely obliterated by this minor change.   As it turns out, -O0 can actually be applied to the original file and still preserve bitwise identical results with the fully optimized version.   Although applying lower optimizations to the non-initializing procedures is perhaps suboptimal, we avoid introducing new files in the repository and various other complicating factors.

Analysis

But why were those routines taking so much time to compile?   If we look closely at their structure, these procedures largely consist of large SELECT CASE blocks (10's -100's of CASE statements) embedded within a loop.   By commenting out various sections of code I was able to show that the compilation time was superlinear with the number of CASE blocks.   However, splitting the original SELECT CASE into two statements had no impact. My theory is that because the various variables accessed in these structures are largely compile time parameters including the loop bounds, the compiler has a great deal of potential to completely unroll everything while looking for opportunities to optimize.    Intuitively I can see that no real simplification is possible, but the compiler is trying very hard.
Many of these bits of code will soon be replaced by a more general algorithm based upon associative arrays. That change will effectively eliminate this compilation bottleneck, though that was not the intended purpose.

Consequences?

And what impact does reduced optimization have on the overall execution performance of the model?     The baseline configuration required 138 seconds per simulated day.   The unoptimized configuration required only 136.1 seconds - a hair faster.   (No, we did not speed up the code, but merely have demonstrated that the noise in the performance is larger than the effect of the changes we are making.)

 

 

Conclusions

 

No doubt a wide variety of things can produce compilation bottlenecks.   However, as this article shows, one does not need to be an expert about compiler implementations to be able to successfully tackle these challenges.   Rather a simple systematic exploration of the situation will often lead to simple solutions.

 

For the particular case discussed here, I am a bit dissatisfied with the use of optimization flags as a permanent solution because future developers may inadvertently (or even purposefully!) alter the compilation flags for this file.   I can try to protect the situation with a comment in the Makefile, but a better solution would be a structural change to the code which eliminates the bottleneck while also improving other characteristics of software quality.   In this particular case such a change is already planned, though it is only coincidental for the purposes of this blog entry.

 

0

There is a bug with respect to alarms in ESMF 3.1.0rp2.  The bug is such that if you create an alarm that rings at the same frequency as the clock, then the alarm will not ring properly.  For example, you create a clock with a timestep of 15 minutes, then you create an alarm that rings every 15 minutes. The alarm will not ring every 15 minutes as expected.

 

This bug has been fixed.  Please use ESMF 3.1.0rp3.  On discover, this version is located in /discover/nobackup/projects/lis/Code/esmf_3_1_0rp3_intel_11_1_038_impi_3_2_011.

 

0

When an unsupported situation (e.g. illegal value) arises in a scientific model, the usual implementation is to stop the model.   In more sophisticated cases, some possibly-useful message about why the model stopped is generated.  However, the logic for handling these rare cases is often buggy - precisely because the situation in rarely encountered!   At any rate, this report-and-abort approach is problematic for TDD because a test which encounters the unsupported situation fails to complete, as opposed to returning with an error condition.   (Note: frameworks in languages such as python and java can exit gracefully from these types of situations, but Fortran has no true exception handing mechanism.)

 

But there is another path.

 

If instead of halting the execution, the routine in question were to "throw" an exception, the framework would be back-in-business.   Even though Fortran does not support true exceptions, the various Fortran unit testing frameworks (and pFUnit in particular) provide means to throw pseudo-exceptions that can then be handled at some other level in the software.   (For the curious, the bit of exception handling that is missing is the ability to pop back up multiple levels of the procedure stack.)

 

So how do we drive this approach with TDD?  Simple.  We write a test which intentionally creates the "unsupported" condition, and then checks that the desired exception is thrown.  "Unsupported" is in quotes, because technically we are now supporting it, albeit in a limited fashion.    In order to pass this test, the implementation must throw the desired exception, but only under the appropriate circumstances.  (Otherwise the other more-typical tests would then signal failure due to the unexpected exception.)

 

The frustrating downside of this approach is that the source code (as opposed to the test code) must now link to the testing framework.   One can use CPP, to minimize the impact on the other developers, but CPP is its own evil.

1

There are several ways of outlining the methodology used for scientific inquiry but it generally consists of the following basic steps:

 

1) Observe, collect data and ask critical questions

2) Formulate hypothetical explanations to answer tose critical questions

3) Make predictions from the hypotheses, i.e. develop a theory

4) Experiments (tests) of all of the above

 

In practice this method works as follows: given a theory of some phenomenon in the world we test it with experiments checking if the responses of the world correspond to the predictions of the current theory. Discrepancies give rise to an iterative cycle of theory refinement as new hypotheses are formulated and new experiments are performed.

 

With TDD [1], frequent micro-iterations are used to develop software:

 

1) Choose an area of the software design to work on.
2) A concrete example, as an executable test to drive the development, is designed, written and shown to fail
3) The code is extended to pass the test (while continuing to pass all other existing tests)

4) The code and tests are refactored to eliminate redundancy

 

Where I have made an effort to provide an analogy between the "four steps" of the scientific method and the "four steps" in TDD.

 

Note that tests in TDD take the role of experiments, while design takes the role of theory. Experimental reproducibility is managed through continued use of automated tests to ensure that the theory has not been broken. The program theory is driven through experimentation with tests. The theory is refined to fit the tests, with refactoring to ensure suitable generality.

 

It would seem that at first glance that the scientific method fits TDD rather well and, perhaps, there is value in using the scientific method as a methaphor to understand some aspects of TDD.

-----

 

References

 

[1] K. Beck. Test Driven Development: By Example. Addison Wesley, 2002.

[2] R. Mugridge, Test Driven Development and the Scientific Method, procs. XP2003, Genova, Italy, 2003

0

I would like to elaborate on the question of what constitutes the simplest implementation, or as Tom puts it "how big of a step?".

 

Recall our test

! test condition that if k=0, yf=y0

   timeStep = 3600
   kappa = 0.0
   numSteps = 86400/timeStep
   allocate(y(numSteps))
   y0 = 1.0
   call Euler(y, y0, timeStep, numSteps)
   call assertEqual(y0, y(numSteps), tolerance=1e-8)

 

Let's start with the simplest implementation

   yf(numSteps) = y0

I would argue that this implementation  does not do very much (if anything at all).  This simply sets an output variable equals to an input variable. It does not test the algorithm we are trying to implement.

 

The other implementation

 

  yy0 = y0
   do i = 1,numSteps
      time = time + timeStep
      yf(i) = yy0*(1.0 - timeStep*kappa)
      yy0 = yf(i)
   end do

perhaps does too much. How can we be so confident that after numSteps our code will do what we intend it to do? Or that the timeStep will not violate the stability condition (this one is probably ok because we must choose timeStep < 1/kappa)?

 

Anyway, I would conclude that the implementation is as small as it can be and that it is the test that is not "properly" formulated. We should, at first, test what happens after one time step (and then two or three?). So, test and implement in small steps but let the smallness be determined by the task at hand.

USAGov logo NASA Logo - nasa.gov