Skip navigation
NASA Logo, National Aeronautics and Space Administration
Modeling Guru Banner

Manage categories

Close

Create and manage categories in Land Information System. Removing a category will not remove content.

Categories in Land Information System
Add a new category (0 remaining)

Manage Announcements

Close

Create and manage announcements in Land Information System. Try to limit the announcements to keep them useful.

Announcements in Land Information System
Subject Author Date Actions

Blog Posts

8 Posts
0

Running TotalView

Posted by jvgeiger Dec 6, 2017

1. Introduction

This document describes how to configure your environment and how to launch TotalView on discover.

 

Please see https://www.roguewave.com/products-services/totalview for documentation on using TotalView.

 

Note

The instructions that follow assume that you are using the bash shell on discover.  If you are using another POSIX-compatible shell (like zsh) the following shell-related statements/commands may be used as is.  If you are using a non-POSIX-compatible shell (like csh or tcsh), then you must modify the statements/commands accordingly.

2. Logging onto discover

Log onto discover with X11 forwarding enabled.  This allows the TotalView windows to pop up onto your local computer.

 

On your Linux or MacOS local computer, add this host block to your $HOME/.ssh/config file.

 

Host discover-sp3
   User your_username_here
   ProxyCommand ssh login.nccs.nasa.gov direct %h
   ForwardX11 yes
   ForwardX11Trusted yes
   Protocol 2

 

Then log onto discover via:

 

% ssh discover-sp3

 

The following sections assume that you are now on discover.


3. Authorized keys

To run multi-node jobs on the discover compute nodes, you must generate password-less SSH public/private keys.  If you do not have id_dsa, id_dsa.pub, and authorized_keys files in your in your $HOME/.ssh directory on discover, then run

 

% ssh-keygen -t dsa % cat id_dsa.pub >> authorized_keys

 

Important

When you run ssh-keygen, you will be prompted for a passphrase to protect the private key.  In general, this is good security practice; however, to run on multiple compute nodes, your private key must be unprotected.  So, when prompted for a passphrase, leave it blank; i.e., hit enter two times.

4. Compiling

Compile your program with '-g'.

 

For example, when configuring LIS, at this prompt:

 

Optimization level (-2=strict checks, -1=debug, 0,1,2,3, default=2):

 

enter either -1 or -2.

5. Interactive session

Small serial/non-MPI programs can be run in TotalView right on the discover login node; however, if your job is large or your program runs in parallel, then you must request an interactive session on discover.

 

% xsalloc --ntasks=<ntasks> --time=<walltime> --account=<account> --constraint=hasw

 

For example,

 

% xsalloc --ntasks=252 --time=6:00:00 --account=s0942 --constraint=hasw

 

For more information on these and other command line switches, see man salloc.

 

Note

When you are logged onto an interactive session on a compute node, your $HOME/.profile will not be sourced.  Depending on how you use your .profile and .bashrc scripts, you may need to explicitly source your .profile.

 

% source ~/.profile
Note

When you are logged onto an interactive session on a compute node, your environment may be empty.  You may need to reset your LIS environment variables and your required modules:

 

% module load lis_7_intel_14_0_3_174_sp3

 

Please see the blog post Creating a custom modulefile for more information.

Note

When you are logged onto an interactive session on a compute node, you will be placed in your $HOME directory.  You must change directory (cd) to your running directory.

 

For example:

 

% cd /discover/nobackup/jvgeiger/experiment_4

6. Loading TotalView

There are many TotalView modules available on discover.

 

I use either tool/tview-2017.1.21 or tool/tview-8.11.0.0.

 

NCCS-recommended version of TotalView
% module load tool/tview-2017.1.21

 

or

 

Old version of TotalView
% module load tool/tview-8.11.0.0

7. Running TotalView

7.1. Serial/non-MPI programs

For serial/non-MPI programs, for example ./LDT ldt.config, you run:

 

% totalview ./LDT

 

You will be presented with a 'Startup Parameters' dialog box:

startup.png

 

Select the 'Arguments' tab and fill in any command line arguments.

arguments.png

7.2. MPI programs

For MPI programs, for example mpirun -np 252 ./LIS  --file lis.config.exp4, you run:

 

% totalview ./LIS

 

Provide any command line arguments, in this example '--file lis.config.exp4' into the 'Arguments' tab (see above section).

 

Then you must specify how to run the program in parallel.  Select the 'Parallel' tab.

parallel.png

 

Click on the 'Parallel system' drop-down menu and select the appropriate parallel system.  For discover, this will be 'Intel MPI-Hydra'.

parallel_dropdown.png

 

Then fill in the number of nodes and tasks for the job.  In this example, I am running 252 processes, and each discover compute node has 28 cores, so I am using 9 nodes (252 / 28 = 9).

tasks.png

 

Note

Older versions of TotalView do not have 'Intel MPI-Hydra'.  For those versions of TotalView, select 'Intel MPI-MPD'.

 

If you are using an older version of TotalView, then you must run

 

% mpdboot -r sshmpi -n $SLURM_NNODES -f $PBS_NODEFILE

 

before you run TotalView.

8. Quick-list

8.1. Older TotalView

  • request interactive session

    xsalloc --ntasks=<ntasks> --time=<walltime> --account=<account> --constraint=hasw
  • source .profile

  • load modules

  • cd to running directory

  • mpdboot -r sshmpi -n $SLURM_NNODES -f $PBS_NODEFILE

  • totalview ./LIS

  • request interactive session

    xsalloc --ntasks=<ntasks> --time=<walltime> --account=<account> --constraint=hasw
  • source .profile

  • load modules

  • cd to running directory

  • totalview ./LIS

0

Below is a list of issues we encounter when switching from the Intel compilers to the GCC (gcc and gfortran) compilers.

Issue 1

Logical variables should be set using .false. or .true. values. Do not use 0 or 1.

 

Bad
  gCalcSOSmode = 0

Good
  gCalcSOSmode = .false.

Issue 2

Message

    ../core/LIS_histDataMod.F90:1523.33:


              n,2,ntiles,(/"kg/m2","mm"/),1,(/"-"/),1,1,1,&
                                   1

    Error: Different CHARACTER lengths (5/2) in array constructor at (1)

Correction

Strings within an array constructor must have the same length.  Use spaces to pad the shorter strings.

 

Bad
          call register_dataEntry(LIS_MOC_LSM_COUNT,LIS_MOC_SNOWICE,&
               LIS_histData(n)%head_lsm_list,&
               n,2,ntiles,(/"kg/m2","mm"/),1,(/"-"/),1,1,1,&
               model_patch=.true.)

Good
          call register_dataEntry(LIS_MOC_LSM_COUNT,LIS_MOC_SNOWICE,&
               LIS_histData(n)%head_lsm_list,&
               n,2,ntiles,(/"kg/m2","mm   "/),1,(/"-"/),1,1,1,&
               model_patch=.true.)

Issue 3

Message

    ../rtms/LIS-CMEM3/pe/CMEM3_set_pedecvars.F90:88.39:

      real          :: vdata(LIS_rc%npatch(n,LIS_rc%lsm_index))
                                           1

    Error: Symbol 'n' is used before it is typed at (1)

Correction

Variables must be declared before they are used.  Here n must be declared before vdata because vdata references n.

 

Bad
    implicit none
    type(ESMF_State)       :: DEC_State
    character*100          :: vname
    real          :: vdata(LIS_rc%npatch(n,LIS_rc%lsm_index))
    integer                :: n

Good
    implicit none
    integer                :: n
    type(ESMF_State)       :: DEC_State
    character*100          :: vname
    real          :: vdata(LIS_rc%npatch(n,LIS_rc%lsm_index))

Issue 4

Message

    ../surfacemodels/land/noahmp.3.6/NoahMP36_readrst.F90:27:0: error: token "." is not valid in preprocessor expressions

      #if (defined USE_NETCDF3 .OR. defined USE_NETCDF4)
      ^

Correction

Do not Fortran syntax for logical operators in preprocessor statements.

Use && for .and.

Use || for .or.

 

Bad
    #if (defined USE_NETCDF3 .OR. defined USE_NETCDF4)

Good
    #if (defined USE_NETCDF3 || defined USE_NETCDF4)

Issue 5

Message

    ../metforcing/ecmwf/read_ecmwf.F90:546.16:

               if ( result1 .eq. .true. ) glbdata1(iv,c) = 0.0
                    1

    Error: Logicals at (1) must be compared with .eqv. instead of .eq.

Correction

Compare logicals with .eqv. not with .eq.

 

Bad
    if ( result1 .eq. .true. ) glbdata1(iv,c) = 0.0

Good
    if ( result1 .eqv. .true. ) glbdata1(iv,c) = 0.0

Issue 6

Message

../dataassim/obs/SMOPSsm/SMOPSsm_Mod.F90:508.12:

       if ( SMOPSsm_struc(n)%useRealtime ) then
            1
Error: IF clause at (1) requires a scalar LOGICAL expression

Correction

You cannot use an integer in a boolean context.  Rewrite the conditional statement like this

 

       if ( SMOPSsm_struc(n)%useRealtime == 1 ) then

or change useRealtime to be a logical variable.

Issue 7

Message

../surfacemodels/land/ruc.3.7/ruc_driver_37.F90:532.14:

              zlvl*2.,sfcprs,tabs,q2,qcatm,rho,                &
              1
Error: Rank mismatch in argument 'z3d' at (1) (rank-3 and scalar)

Make sure that the actual arguments that correspond to dummy arrays are arrays not scalars.

0

Creating a custom modulefile

Posted by jvgeiger Aug 26, 2015

Introduction

NCCS uses the Environment Modules Project (http://modules.sourceforge.net/), better known as the module command, to manage multiple versions of programs and libraries.  Users use it to manage their environment.  Users are not restricted to only the system level modulefiles provided by NCCS.  Users may create their own.  One specific use of custom modulefiles is to manage various LIS-related environments; e.g., to manage compilers and environment variables for compiling and running LIS 6 vs LIS 7 or for compiling and running with the Intel compilers vs the GCC compilers.  This post explains how to create a custom modulefile.

 

Please refer to the man pages for module(1) and modulefile(4) for more information.

Set up

First create a sub-directory named privatemodules in your home directory.

 

Then add this command to your .profile (or equivalent) login file.

 

module use --append $HOME/privatemodules

 

This will make your custom modulefiles available to the module program every time that you log on.

 

Note

Please note that on-line resources suggest using module load use.own to set up your custom modulefiles.  This command will create, if necessary, a privatemodules sub-directory in your home directory along with providing a sample modulefile named null.  Then this command will add your privatemodules sub-directory to the $MODULEPATH environment variable, making your custom modulefiles available to the module program.

However, that command does not work on all systems.  On some systems, like discover, you have to run module load $MODULESHOME/modulefiles/use.own.

Also note that if you subsequently run the module purge command, then you must rerun the module load use.own command to make your custom modulefiles available again.

Thus, instead of using module load use.own, this post suggests that you add module use --append $HOME/privatemodules to your .profile file.  This command neither relies on the system to find the use.own module nor is affected by a module purge command.

Creating a custom modulefile

Below is a sample custom modulefile.  It loads the modules and sets the environment variables needed to set up an Intel-18-based development environment.  To use it, copy the contents of the listing into $HOME/privatemodules/intel_18_0_3_222.  Then run the command:

 

discover07$ module load intel_18_0_3_222

 

After loading the custom modulefile, a module list command will report something similar to:

 

discover07$ module list
Currently Loaded Modulefiles:
  1) comp/intel-18.0.3.222   5) other/svn-1.9.5
  2) mpi/impi-18.0.3.222     6) other/vim-8.0
  3) tool/tview-2017.1.21    7) intel_18_0_3_222
  4) other/git-2.9.3

Listing of sample custom modulefile intel_18_0_3_222
#%Module1.0###################################################################

proc ModulesHelp { } {
    puts stderr "\t[module-info name] - loads the INTEL_18_0_3_222 env"
    puts stderr ""
    puts stderr "\tThe following env variables are set:"
    puts stderr "\t\tDEV_ENV"
    puts stderr ""
    puts stderr "\tThe following modules are loaded:"
    puts stderr "\t\tcomp/intel-18.0.3.222"
    puts stderr "\t\tmpi/impi-18.0.3.222"
    puts stderr "\t\ttool/tview-2017.1.21"
    puts stderr "\t\tother/git-2.9.3"
    puts stderr "\t\tother/svn-1.9.5"
    puts stderr "\t\tother/vim-8.0"
    puts stderr ""
    puts stderr "\tPython 2.7 is added to the search PATH."
}


conflict comp mpi


module-whatis     "loads the [module-info name] environment"


set modname     [module-info name]
set modmode     [module-info mode]


module load comp/intel-18.0.3.222
module load mpi/impi-18.0.3.222

module load tool/tview-2017.1.21
module load other/git-2.9.3
module load other/svn-1.9.5
module load other/vim-8.0


setenv   DEV_ENV         INTEL_18_0_3_222

prepend-path   PATH   /usr/local/other/SLES11.3/python/2.7.11/gcc-4.3.4/bin

Additional custom modulefiles

It is suggested that you keep your custom modulefiles as lean as possible.  Do not load more modules than are needed for a particular task.  The above sample custom modulefile loads only the modules needed to perform general software development with the Intel 18 compilers.

 

But Jim, I need Matlab, R, IDL, etc., what should I do?  Create additional custom modulefiles targeting the other tools and tasks that you need.  For example, create a custom modulefile for Matlab.  It should load only the modules needed to run Matlab, and it should set only environment variables required and related to running Matlab.

 

Warning
When you load too many unrelated modules into your environment, you run the risk of creating undetected incompatibilites between tools and libraries.  For example, you may end up loading three different NetCDF libraries.  And one day LIS runs, but the next day LIS cannot read its own restart file.

 

When dealing with multiple tasks using multiple custom modulefiles, you may either load and change the custom modulefiles, or you may launch multiple xterm s (one for each task) on discover, and load your different custom modulefiles in each xterm.

 

Example of loading and changing custom modulefiles
discover07$ module load lis_7_intel_14_0_3_174_sp3

# Work, work, work

discover07$ module purge
discover07$ module load my_matlab_mod

# Plot, plot, plot

Sample .profile and .bashrc files

Please refer to the man page for bash(1) for more information regarding the use of the .profile and .bashrc files.

 

Sample .profile file
# This file is read each time a login shell is started.

#
# Set LD_LIBRARY_PATH
#
# non LDT_*, non LIS_*, non LVT_* paths go here
#export LD_LIBRARY_PATH=$ADDITIONAL_PATHS:$LD_LIBRARY_PATH


#
# Set environment
#
export EDITOR=vim
export SVN_EDITOR=vim
export TVDSVRLAUNCHCMD=ssh


#
# Set PATH
#
# non LDT_*, non LIS_*, non LVT_* paths go here
#export PATH=$ADDITIONAL_PATHS:$PATH


#
# Set modules
#
# load modules manually from the command line
# to keep your login environment clean
module use --append $HOME/privatemodules


ulimit -s unlimited

Sample .bashrc file
# This file is for interactive shells.
# On some systems, it is loaded in some non-interactive cases.
# Return if this is not an interactive shell.
[ "$PS1" ] || return


#
# Set aliases
#
unalias -a
alias ls="ls --color=auto"


#
# Set prompt
#
export PS1="\[\e[36m\]\h\[\e[00m\][\!]\$ "


#
# Set environment
#
export LS_COLORS='no=00:fi=00:di=33:ln=36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=32:*.cmd=32:*.exe=32:*.com=32:*.btm=32:*.bat=32:*.sh=32:*.csh=32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35'

export MANPATH=$MANPATH:/usr/slurm/share/man

LIS development environments

Here are several custom modulefiles for LIS development.

LIS 7 with Intel 14 compilers

Note
This is an old environment.  It is here for reference.  Please use the LIS 7 with Intel 18 compilers environment.

Listing of custom modulefile lis_7_intel_14_0_3_174_sp3
#%Module1.0###################################################################

proc ModulesHelp { } {
    puts stderr "\t[module-info name] - loads the LIS_7_INTEL_14_0_3_174_SP3 env"
    puts stderr ""
    puts stderr "\tThe following env variables are set:"
    puts stderr "\t\tDEV_ENV"
    puts stderr "\t\tLIS_ARCH"
    puts stderr "\t\tLIS_SPMD"
    puts stderr "\t\tLIS_FC"
    puts stderr "\t\tLIS_CC"
    puts stderr "\t\tLIS_JASPER"
    puts stderr "\t\tLIS_GRIBAPI"
    puts stderr "\t\tLIS_NETCDF"
    puts stderr "\t\tLIS_HDF4"
    puts stderr "\t\tLIS_HDFEOS"
    puts stderr "\t\tLIS_HDF5"
    puts stderr "\t\tLIS_MODESMF"
    puts stderr "\t\tLIS_LIBESMF"
    puts stderr "\t\tLIS_MINPACK"
    puts stderr "\t\tLIS_CRTM"
    puts stderr "\t\tLIS_CRTM_PROF"
    puts stderr "\t\tLIS_CMEM"
    puts stderr "\t\tLIS_LAPACK"
    puts stderr "\t\tLDT_ARCH"
    puts stderr "\t\tLDT_FC"
    puts stderr "\t\tLDT_CC"
    puts stderr "\t\tLDT_JASPER"
    puts stderr "\t\tLDT_GRIBAPI"
    puts stderr "\t\tLDT_NETCDF"
    puts stderr "\t\tLDT_HDF4"
    puts stderr "\t\tLDT_HDFEOS"
    puts stderr "\t\tLDT_HDF5"
    puts stderr "\t\tLDT_MODESMF"
    puts stderr "\t\tLDT_LIBESMF"
    puts stderr "\t\tLVT_ARCH"
    puts stderr "\t\tLVT_FC"
    puts stderr "\t\tLVT_CC"
    puts stderr "\t\tLVT_JASPER"
    puts stderr "\t\tLVT_GRIBAPI"
    puts stderr "\t\tLVT_NETCDF"
    puts stderr "\t\tLVT_HDF4"
    puts stderr "\t\tLVT_HDFEOS"
    puts stderr "\t\tLVT_HDF5"
    puts stderr "\t\tLVT_MODESMF"
    puts stderr "\t\tLVT_LIBESMF"
    puts stderr "\t\tLVT_GDAL"
    puts stderr "\t\tLVT_FORTRANGIS"
    puts stderr ""
    puts stderr "\tThe following modules are loaded:"
    puts stderr "\t\tcomp/intel-14.0.3.174"
    puts stderr "\t\tmpi/impi-5.0.3.048"
    puts stderr "\t\ttool/tview-8.11.0.0"
    puts stderr "\t\tother/git-2.9.3"
    puts stderr "\t\tother/svn-1.8.13"
    puts stderr "\t\tother/vim-8.0"
    puts stderr ""
    puts stderr "\tPython 2.7 is added to the search PATH."
}


conflict comp mpi


module-whatis     "loads the [module-info name] environment"


set modname     [module-info name]
set modmode     [module-info mode]


module load comp/intel-14.0.3.174
module load mpi/impi-5.0.3.048

module load tool/tview-8.11.0.0
module load other/git-2.9.3
module load other/svn-1.8.13
module load other/vim-8.0


set   def_lis_jasper      /discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3
set   def_lis_gribapi     /discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3
set   def_lis_netcdf      /discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3
set   def_lis_hdf4        /discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3
set   def_lis_hdfeos      /discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3
set   def_lis_hdf5        /discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3
set   def_lis_modesmf     /discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default
set   def_lis_libesmf     /discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default
set   def_lis_minpack     /discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
set   def_lis_crtm        /discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev
set   def_lis_crtm_prof   /discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_11_1_038/
set   def_lis_cmem        /discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117
set   def_lis_lapack      /usr/local/other/SLES11/lapack/3.3.1/intel-12.1.0.233/lib
set   def_lvt_gdal        /discover/nobackup/projects/lis/libs/gdal/2.0.0
set   def_lvt_fortrangis  /discover/nobackup/projects/lis/libs/fortrangis/2.4


setenv   DEV_ENV         LIS_7_INTEL_14_0_3_174_SP3
setenv   LIS_ARCH        linux_ifc
setenv   LIS_SPMD        parallel
setenv   LIS_FC          mpif90
setenv   LIS_CC          mpicc
setenv   LIS_JASPER      $def_lis_jasper
setenv   LIS_GRIBAPI     $def_lis_gribapi
setenv   LIS_NETCDF      $def_lis_netcdf
setenv   LIS_HDF4        $def_lis_hdf4
setenv   LIS_HDFEOS      $def_lis_hdfeos
setenv   LIS_HDF5        $def_lis_hdf5
setenv   LIS_MODESMF     $def_lis_modesmf
setenv   LIS_LIBESMF     $def_lis_libesmf
setenv   LIS_MINPACK     $def_lis_minpack
setenv   LIS_CRTM        $def_lis_crtm
setenv   LIS_CRTM_PROF   $def_lis_crtm_prof
setenv   LIS_CMEM        $def_lis_cmem
setenv   LIS_LAPACK      $def_lis_lapack


setenv   LDT_ARCH      linux_ifc
setenv   LDT_FC        mpif90
setenv   LDT_CC        mpicc
setenv   LDT_JASPER    $def_lis_jasper
setenv   LDT_GRIBAPI   $def_lis_gribapi
setenv   LDT_NETCDF    $def_lis_netcdf
setenv   LDT_HDF4      $def_lis_hdf4
setenv   LDT_HDFEOS    $def_lis_hdfeos
setenv   LDT_HDF5      $def_lis_hdf5
setenv   LDT_MODESMF   $def_lis_modesmf
setenv   LDT_LIBESMF   $def_lis_libesmf


setenv   LVT_ARCH        linux_ifc
setenv   LVT_FC          mpif90
setenv   LVT_CC          mpicc
setenv   LVT_JASPER      $def_lis_jasper
setenv   LVT_GRIBAPI     $def_lis_gribapi
setenv   LVT_NETCDF      $def_lis_netcdf
setenv   LVT_HDF4        $def_lis_hdf4
setenv   LVT_HDFEOS      $def_lis_hdfeos
setenv   LVT_HDF5        $def_lis_hdf5
setenv   LVT_MODESMF     $def_lis_modesmf
setenv   LVT_LIBESMF     $def_lis_libesmf
setenv   LVT_GDAL        $def_lvt_gdal
setenv   LVT_FORTRANGIS  $def_lvt_fortrangis


prepend-path   LD_LIBRARY_PATH   "$def_lis_hdf4/lib:$def_lis_hdf5/lib:$def_lis_libesmf:$def_lis_netcdf/lib:$def_lis_gribapi/lib"
prepend-path   PATH   /usr/local/other/SLES11.3/python/2.7.11/gcc-4.3.4/bin
prepend-path   PATH   /home/jvgeiger/local/nccmp-1.3.0/bin
prepend-path   PATH   "$def_lis_netcdf/bin:$def_lis_gribapi/bin"

LIS 7 with Intel 18 compilers

This is the recommended environment for compiling and running LIS on discover.

 

Warning

The 26 Apr 2018 version of this article provided a custom modulefile for compiling and running LIS with Intel 17 compilers.  This development/running environment produced  bad results for "2d ensemble gridspace" output.  We could not find an error in the LIS code that led to the bad results, and LIS produced good results after upgrading to Intel 18.

Stop using the lis_7_intel_17_0_4_196 modulefile and switch to using the lis_7_intel_18_0_3_222 modulefile.


Listing of the custom modulefile lis_7_intel_18_0_3_222
#%Module1.0###################################################################

proc ModulesHelp { } {
    puts stderr "\t[module-info name] - loads the LIS_7_INTEL_18_0_3_222 env"
    puts stderr ""
    puts stderr "\tThe following env variables are set:"
    puts stderr "\t\tDEV_ENV"
    puts stderr "\t\tLIS_ARCH"
    puts stderr "\t\tLIS_SPMD"
    puts stderr "\t\tLIS_FC"
    puts stderr "\t\tLIS_CC"
    puts stderr "\t\tLIS_JASPER"
    puts stderr "\t\tLIS_ECCODES"
    puts stderr "\t\tLIS_NETCDF"
    puts stderr "\t\tLIS_HDF4"
    puts stderr "\t\tLIS_HDFEOS"
    puts stderr "\t\tLIS_HDF5"
    puts stderr "\t\tLIS_MODESMF"
    puts stderr "\t\tLIS_LIBESMF"
    puts stderr "\t\tLIS_MINPACK"
    puts stderr "\t\tLIS_CRTM"
    puts stderr "\t\tLIS_CRTM_PROF"
    puts stderr "\t\tLIS_CMEM"
    puts stderr "\t\tLIS_LAPACK"
    puts stderr "\t\tLDT_ARCH"
    puts stderr "\t\tLDT_FC"
    puts stderr "\t\tLDT_CC"
    puts stderr "\t\tLDT_JASPER"
    puts stderr "\t\tLDT_ECCODES"
    puts stderr "\t\tLDT_NETCDF"
    puts stderr "\t\tLDT_HDF4"
    puts stderr "\t\tLDT_HDFEOS"
    puts stderr "\t\tLDT_HDF5"
    puts stderr "\t\tLDT_MODESMF"
    puts stderr "\t\tLDT_LIBESMF"
    puts stderr "\t\tLDT_GDAL"
    puts stderr "\t\tLDT_FORTRANGIS"
    puts stderr "\t\tLDT_LIBGEOTIFF"
    puts stderr "\t\tLVT_ARCH"
    puts stderr "\t\tLVT_FC"
    puts stderr "\t\tLVT_CC"
    puts stderr "\t\tLVT_JASPER"
    puts stderr "\t\tLVT_ECCODES"
    puts stderr "\t\tLVT_NETCDF"
    puts stderr "\t\tLVT_HDF4"
    puts stderr "\t\tLVT_HDFEOS"
    puts stderr "\t\tLVT_HDF5"
    puts stderr "\t\tLVT_MODESMF"
    puts stderr "\t\tLVT_LIBESMF"
    puts stderr "\t\tLVT_GDAL"
    puts stderr "\t\tLVT_FORTRANGIS"
    puts stderr ""
    puts stderr "\tThe following modules are loaded:"
    puts stderr "\t\tother/comp/gcc-7.3"
    puts stderr "\t\tcomp/intel-18.0.3.222"
    puts stderr "\t\tmpi/impi-18.0.3.222"
    puts stderr "\t\ttool/tview-2017.1.21"
    puts stderr "\t\tother/git-2.18.0"
    puts stderr "\t\tother/svn-1.9.5"
    puts stderr "\t\tother/vim-8.0"
    puts stderr ""
    puts stderr "\tPython 2.7 is added to the search PATH."
}


conflict comp mpi


module-whatis     "loads the [module-info name] environment"


set modname     [module-info name]
set modmode     [module-info mode]


module load other/comp/gcc-7.3
module load comp/intel-18.0.3.222
module load mpi/impi-18.0.3.222

module load tool/tview-2017.1.21
module load other/git-2.18.0
module load other/svn-1.9.5
module load other/vim-8.0


set   def_lis_jasper      /discover/nobackup/projects/lis/libs/jasper/2.0.14_intel-18.0.3.222
set   def_lis_eccodes     /discover/nobackup/projects/lis/libs/ecCodes/2.7.0_intel-18.0.3.222
set   def_lis_netcdf      /discover/nobackup/projects/lis/libs/netcdf/4.5.0_intel-18.0.3.222
set   def_lis_hdf4        /discover/nobackup/projects/lis/libs/hdf4/4.2.13_intel-18.0.3.222
set   def_lis_hdfeos      /discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-18.0.3.222
set   def_lis_hdf5        /discover/nobackup/projects/lis/libs/hdf5/1.10.1_intel-18.0.3.222
set   def_lis_modesmf     /discover/nobackup/projects/lis/libs/esmf/7.1.0r_intel-18.0.3.222_impi-18.0.3.222/mod/modO/Linux.intel.64.intelmpi.default
set   def_lis_libesmf     /discover/nobackup/projects/lis/libs/esmf/7.1.0r_intel-18.0.3.222_impi-18.0.3.222/lib/libO/Linux.intel.64.intelmpi.default
set   def_lis_minpack     /discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
set   def_lis_crtm        /discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev_intel_18_0_3_222
set   def_lis_crtm_prof   /discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_18_0_3_222
set   def_lis_cmem        /discover/nobackup/projects/lis/libs/LIS-MEM/intel_18_0_3_222
set   def_lis_lapack      /usr/local/other/SLES11/lapack/3.3.1/intel-12.1.0.233/lib
set   def_lvt_gdal        /discover/nobackup/projects/lis/libs/gdal/2.4.1_intel-18.0.3.222
set   def_lvt_fortrangis  /discover/nobackup/projects/lis/libs/fortrangis/2.4_intel-18.0.3.222
set   def_ldt_libgeotiff  /discover/nobackup/projects/lis/libs/geotiff/1.4.3_intel-18.0.3.222

setenv   DEV_ENV         LIS_7_INTEL_18_0_3_222
setenv   LIS_ARCH        linux_ifc
setenv   LIS_SPMD        parallel
setenv   LIS_FC          mpif90
setenv   LIS_CC          mpicc
setenv   LIS_JASPER      $def_lis_jasper
setenv   LIS_ECCODES     $def_lis_eccodes
setenv   LIS_NETCDF      $def_lis_netcdf
setenv   LIS_HDF4        $def_lis_hdf4
setenv   LIS_HDFEOS      $def_lis_hdfeos
setenv   LIS_HDF5        $def_lis_hdf5
setenv   LIS_MODESMF     $def_lis_modesmf
setenv   LIS_LIBESMF     $def_lis_libesmf
setenv   LIS_MINPACK     $def_lis_minpack
setenv   LIS_CRTM        $def_lis_crtm
setenv   LIS_CRTM_PROF   $def_lis_crtm_prof
setenv   LIS_CMEM        $def_lis_cmem
setenv   LIS_LAPACK      $def_lis_lapack


setenv   LDT_ARCH        linux_ifc
setenv   LDT_FC          mpif90
setenv   LDT_CC          mpicc
setenv   LDT_JASPER      $def_lis_jasper
setenv   LDT_ECCODES     $def_lis_eccodes
setenv   LDT_NETCDF      $def_lis_netcdf
setenv   LDT_HDF4        $def_lis_hdf4
setenv   LDT_HDFEOS      $def_lis_hdfeos
setenv   LDT_HDF5        $def_lis_hdf5
setenv   LDT_MODESMF     $def_lis_modesmf
setenv   LDT_LIBESMF     $def_lis_libesmf
setenv   LDT_GDAL        $def_lvt_gdal
setenv   LDT_FORTRANGIS  $def_lvt_fortrangis
setenv   LDT_LIBGEOTIFF  $def_ldt_libgeotiff


setenv   LVT_ARCH        linux_ifc
setenv   LVT_FC          mpif90
setenv   LVT_CC          mpicc
setenv   LVT_JASPER      $def_lis_jasper
setenv   LVT_ECCODES     $def_lis_eccodes
setenv   LVT_NETCDF      $def_lis_netcdf
setenv   LVT_HDF4        $def_lis_hdf4
setenv   LVT_HDFEOS      $def_lis_hdfeos
setenv   LVT_HDF5        $def_lis_hdf5
setenv   LVT_MODESMF     $def_lis_modesmf
setenv   LVT_LIBESMF     $def_lis_libesmf
setenv   LVT_GDAL        $def_lvt_gdal
setenv   LVT_FORTRANGIS  $def_lvt_fortrangis


prepend-path   LD_LIBRARY_PATH   "$def_ldt_libgeotiff/lib:$def_lvt_gdal/lib:$def_lis_hdf4/lib:$def_lis_hdf5/lib:$def_lis_libesmf:$def_lis_netcdf/lib:$def_lis_eccodes/lib:$def_lis_jasper/lib64"
prepend-path   PATH   /usr/local/other/SLES11.3/python/2.7.11/gcc-4.3.4/bin
prepend-path   PATH   /home/jvgeiger/local/nccmp-1.8.2.1/bin
prepend-path   PATH   /discover/nobackup/projects/lis/libs/ncview/2.1.7_intel-18.0.3.222/bin
prepend-path   PATH   "$def_lis_netcdf/bin:$def_lis_eccodes/bin"

1

Using svn+ssh method to access LIS repository

Introduction

The Progress/CVSACL Security Maintenance that was completed on 1 Apr 2015 prompted an upgrade to the Subversion package installed on discover to access the LIS software repository hosted on progress.

 

Please see the notes in the LIS ModelingGuru article "Subversion upgrade 1.8.13" for more details.

https://modelingguru.nasa.gov/community/atmospheric/lis/blog/2015/04/01/subversi on-upgrade-1813

 

This security update raised two issues:

 

  1. Due to security concerns, the password caching feature of Subversion was disabled in this upgrade.  Thus every Subversion command that accesses the LIS software repository (svn update, svn log, etc.) prompts you for your Subversion password.

  2. Any SSL-based communications with the progress server must use TLS 1.2.  Using older versions of Subversion (and OpenSSL) on remote systems to access the LIS software repository results in error messages similar to this:

     

    sh$ svn checkout https://progress.nccs.nasa.gov/svn/lis/7/public7.1svn: OPTIONS of 'https://progress.nccs.nasa.gov/svn/lis/7/public7.1': SSL
    handshake failed: SSL error: sslv3 alert handshake failure
    (https://progress.nccs.nasa.gov)
    

     

    It is often difficult to get Subversion (and OpenSSL) upgraded on remote systems.

 

This article explains how to use Subversion's svn+ssh method along with ssh-agent to resolve these two problems.

 

This article is intended for LIS repository users who are either troubled by issue 1 or stopped by issue 2.  If neither issue affects you, then you may continue to follow the instructions in the above "Subversion upgrade 1.8.13" article.

Set up

This method does require some set up from each user before it can be used.  Please see the instructions on https://progress.nccs.nasa.gov/trac/admin/wiki/QuickStart

svn+ssh method

Since this method does not use SSL-based communications, it is not affected by the above Progress/CVSACL Security Maintenance.

Access from discover

Once you have both your public SSH key (~/.ssh/id_rsa.pub) uploaded to progress via the "keyupload" link and the progressdirect virtual host configured, you may access the LIS software repository via commands like this:

 

sh$ svn checkout svn+ssh://progressdirect/svn/lis/7/public7.1

Access from a remote system

Once you have both your public SSH key (~/.ssh/id_rsa.pub) uploaded to progress via the "keyupload" link and and your SSH tunnel established, you may access the LIS software repository via commands like this:

 

sh$ svn checkout svn+ssh://ptunnel/svn/lis/7/public7.1

 

Please note that you must always establish the SSH tunnel before you try to run any Subversion commands.  But once the SSH tunnel is established, it is good for your entire login session.

ssh-agent

Best practices dictate that you protect your private SSH key (~/.ssh/id_rsa) with a password.  You will be prompted to create a password when you create your SSH keys via the ssh-keygen command.  Of course, a password-protected key requires unlocking each time it is used, so every Subversion command that accesses the LIS software repository (svn update, svn log, etc.) via the svn+ssh method will prompt you for your key's password.  You can eliminate the need to repeatedly enter your key's password by using the ssh-agent command.  The simplest way to use ssh-agent is to run it right after you log onto discover or your remote system:

 

sh$ ssh-agent bash

 

Note that bash is just an example.  You may specify any shell that you prefer.

After you have run ssh-agent, you must load your private SSH key:

 

sh$ ssh-add ~/.ssh/id_rsa

 

Here you will be prompted for the key's password.  After you provide the password, ssh-agent will work with SSH to manage access to your private SSH key.  Thus, you will no longer be prompted by Subversion for a password when you run any commands that access the repository.

Converting from https to svn+ssh

To convert a working directory from using the https method to using the svn+ssh method, you must run the svn relocate command.

 

For example, you have a working copy of the public 7.1 branch obtained via:

 

sh$ svn checkout https://progress.nccs.nasa.gov/svn/lis/7/public7.1

 

To convert it to using svn+ssh, go into the working directory and run:

 

sh$ svn relocate svn+ssh://progressdirect/svn/lis/7/public7.1

 

or:

 

sh$ svn relocate svn+ssh://ptunnel/svn/lis/7/public7.1

 

whichever is appropriate for your situation.

 

Summary

  1. log on
  2. run ssh-agent
  3. run ssh-add
  4. establish tunnel, if neccessary
  5. use Subversion
0

Subversion upgrade 1.8.13

Posted by jvgeiger Apr 1, 2015

The Progress/CVSACL Security Maintenance that was completed on 1 Apr 2015 prompted an upgrade to the Subversion package installed on discover to access the LIS software repository hosted on progress.

 

Currently, the new Subversion package is available only on the SP3 front-end systems.  Thus you must log onto discover-sp3 to use Subversion.  You may do development and run LIS on either an SP1 or an SP3 system, but you must be on an SP3 system to run svn.

 

To use the new Subversion package, you must load the other/svn-1.8.13 module:

 

discover07$ module load other/svn-1.8.13

 

After you have loaded the new Subversion module.  You must run svn upgrade in any existing working directories.  If you forget, Subversion will remind you, as seen below.

 

discover07$ svn update
svn: E155036: Please see the 'svn upgrade' command
svn: E155036: The working copy at '/gpfsm/dnb12s1/jvgeiger/SRC/7/d7'
is too old (format 10) to work with client version '1.8.13 (r1667537)' (expects format 31). You need to upgrade the working copy first.

 

Simply run the svn upgrade command.

 

discover07$ svn upgrade
Upgraded '.'
Upgraded 'plugins'
Upgraded 'offline'
Upgraded 'params'
Upgraded 'params/gfrac'
Upgraded 'params/gfrac/VIIRSDaily'
Upgraded 'params/gfrac/ALMIPII'
Upgraded 'params/gfrac/NESDISWeekly'
Upgraded 'params/gfrac/SPORTDaily'
.
.
.

 

Please note that a fresh checkout from the LIS software repository does not require that you to run the svn upgrade command.

 

Finally, you must accept the new SSL certificate.  You must run an svncommand that accesses the repository.  When prompted, permanently accept the certificate by selecting p.  For example, run svn update in a working directory.

 

discover07$ svn update
Updating '.':
Error validating server certificate for 'https://progress.nccs.nasa.gov:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: www.nccs.nasa.gov
 - Valid: from Oct 22 12:00:01 2013 GMT until Apr 21 12:00:00 2016 GMT
 - Issuer: www.digicert.com, DigiCert Inc, US
 - Fingerprint: A3:4F:7A:E6:E5:AE:55:BB:13:F2:56:48:63:33:76:EF:CC:9C:D7:DE
(R)eject, accept (t)emporarily or accept (p)ermanently? p
2

Compiling LIS 7 on SLES11 SP3

Posted by jvgeiger Mar 9, 2015
Last updated: Wed Sep 14 15:39:40 EDT 2016

Introduction

NCCS is adding new hardware, containing Intel Haswell nodes, running the SUSE Linux Enterprise Server (SLES) 11 SP3 operating system.  These notes describe how to recompile LIS 7 to run on this operating system.

 

For compiling LIS on the SLES 11 SP1 nodes, please see the post "Compiling LIS 6 and LIS 7 --- SLURM migration".

Modules

Use these modules to compile LIS 7:

 

module load comp/intel-14.0.3.174
module load mpi/impi-5.0.3.048

Libraries

Specify these variables/libraries in your .profile (or equivalent) file to configure LIS 7, LDT, and LVT (here written in bash syntax):

 

Note: New libraries for MINPACK, CRTM, CRTM_PROF, and CMEM, have not yet been recompiled.

 

export DEV_ENV=7_INTEL_14_SP3
export LIS_ARCH=linux_ifc
export LIS_SPMD=parallel
export LIS_FC=mpif90
export LIS_CC=mpicc
export LIS_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3
export LIS_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3
export LIS_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3
export LIS_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3
export LIS_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3
export LIS_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3
export LIS_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default
export LIS_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default
export LIS_MINPACK=
export LIS_CRTM=
export LIS_CRTM_PROF=
export LIS_CMEM=
export LIS_LAPACK=/usr/local/other/SLES11/lapack/3.3.1/intel-12.1.0.233/lib

export LVT_ARCH=linux_ifc
export LVT_FC=mpif90
export LVT_CC=mpicc
export LVT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3
export LVT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3
export LVT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3
export LVT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3
export LVT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3
export LVT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3
export LVT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default
export LVT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default

export LVT_GDAL=/discover/nobackup/projects/lis/libs/gdal/2.0.0

export LVT_FORTRANGIS=/discover/nobackup/projects/lis/libs/fortrangis/2.4

export LDT_ARCH=linux_ifc export LDT_FC=mpif90 export LDT_CC=mpicc export LDT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel-14.0.3.174_sp3 export LDT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel-14.0.3.174_sp3 export LDT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.3.3.1_intel-14.0.3.174_sp3 export LDT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.11_intel-14.0.3.174_sp3 export LDT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.19v1.00_intel-14.0.3.174_sp3 export LDT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.14_intel-14.0.3.174_sp3 export LDT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/mod/modO/Linux.intel.64.intelmpi.default export LDT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5.2.0rp3_intel-14.0.3.174_impi-5.0.3.048_sp3/lib/libO/Linux.intel.64.intelmpi.default export LD_LIBRARY_PATH=${LIS_HDF4}/lib:${LIS_HDF5}/lib:${LIS_LIBESMF}:${LIS_NETCDF}/lib:${LIS_GRIBAPI}/lib:$LD_LIBRARY_PATH
0

Last updated: Thu Dec 18 13:40:12 EST 2014

Introduction

NCCS is migrating the queuing system used on discover from PBS to SLURM.

 

NCCS has identified the following libraries as being incompatible with SLURM.

 

  • Intel MPI:
    • mpi/impi-3.1.*
    • mpi/impi-3.2.011
    • mpi/impi-3.2.1.009
    • mpi/impi-4.*-beta
  • MVAPICH2:
    • other/mpi/mvapich2-1.6rc2/intel-11.1.072
    • other/mpi/mvapich2-1.7*
    • other/mpi/mvapich2-1.8/*
    • other/mpi/mvapich2-1.8a2/*
  • OpenMPI:
    • mpi/openmpi-1.2.5/intel-9
    • mpi/openmpi-1.2.5/intel-10
    • other/mpi/openmpi/1.4.*
    • other/mpi/openmpi/1.6-gcc_4.8-20120401_nag-5.3-854
    • other/mpi/openmpi/1.6.0*
    • other/mpi/openmpi/1.6.3*
    • other/mpi/openmpi/1.6.4*

 

On discover, LIS 6 uses mpi/impi-3.2.011, and LIS 7 uses mpi/impi-4.*-beta.  The notes below describe how to recompile LIS with libraries compatible with SLURM.


Compiling LIS 6

Using Intel 11 and IMPI 3

Modules

Use these modules to compile LIS 6:

 

module load comp/intel-11.1.038
module load mpi/impi-3.2.2.006

Libraries

Specify these libraries in the LIS 6 configure.lis file:

 

LIB_ESMF = /discover/nobackup/projects/lis/libs/esmf/3_1_0rp3_intel_11_1_038_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default
MOD_ESMF = /discover/nobackup/projects/lis/libs/esmf/3_1_0rp3_intel_11_1_038_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
INC_JASPER = /usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038/include
LIB_JASPER = /usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038/lib

 

Note that the other libraries specified in the default arch/configure.lis.linux_ifc are correct.

If you experience errors while compiling or linking the LIS 6 executable on discover, then please consult the latest revision of the arch/configure.lis.linux_ifc file.

 


Compiling LIS 7, LDT, and LVT

Using Intel 11 and IMPI 3

Modules

Use these modules to compile LIS 7, LDT, and LVT:

 

module load comp/intel-11.1.069
module load mpi/impi-3.2.2.006
module load lib/mkl-9.1.023

Libraries

Specify these variables/libraries in your .profile (or equivalent) file to configure LIS 7, LDT, and LVT (here written in bash syntax):

 

export LIS_ARCH=linux_ifc
export LIS_SPMD=parallel
export LIS_FC=mpif90
export LIS_CC=mpicc
export LIS_JASPER=/usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038
export LIS_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.9.16_intel_11_1_069
export LIS_NETCDF=/discover/nobackup/projects/lis/libs/netcdf4/4.3.0_intel_11_1_069
export LIS_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.9_intel_11_1_069
export LIS_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.18v1.00_intel_11_1_069
export LIS_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_11_1_069
export LIS_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
export LIS_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default
export LIS_MINPACK=/discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
export LIS_CRTM=/discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev
export LIS_CRTM_PROF=/discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_11_1_038/
export LIS_CMEM=/discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117

export LVT_ARCH=linux_ifc
export LVT_SPMD=parallel
export LVT_FC=mpif90
export LVT_CC=mpicc
export LVT_JASPER=/usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038
export LVT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.9.16_intel_11_1_069
export LVT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf4/4.3.0_intel_11_1_069
export LVT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.9_intel_11_1_069
export LVT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.18v1.00_intel_11_1_069
export LVT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_11_1_069
export LVT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
export LVT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default

export LDT_ARCH=linux_ifc
export LDT_SPMD=parallel
export LDT_FC=mpif90
export LDT_CC=mpicc
export LDT_JASPER=/usr/local/other/SLES11/jasper/1.900.1/intel-11.1.038
export LDT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.9.16_intel_11_1_069
export LDT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf4/4.3.0_intel_11_1_069
export LDT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2.9_intel_11_1_069
export LDT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos2/2.18v1.00_intel_11_1_069
export LDT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_11_1_069
export LDT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/mod/modO/Linux.intel.64.intelmpi.default
export LDT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp2_intel_11_1_069_impi_3_2_2_006/lib/libO/Linux.intel.64.intelmpi.default

Using Intel 13 and IMPI 4

Modules

Use these modules to compile LIS 7, LDT, and LVT:

 

comp/intel-13.1.3.192
mpi/impi-4.0.3.008

Libraries

Specify these variables/libraries in your .profile (or equivalent) file to configure LIS 7, LDT, and LVT (here written in bash syntax):

 

export LIS_ARCH=linux_ifc
export LIS_SPMD=parallel
export LIS_FC=mpif90
export LIS_CC=mpicc
export LIS_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192
export LIS_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117
export LIS_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2r4_noNETCDF_intel_13_0_1_117
export LIS_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos
export LIS_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117
export LIS_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/mod/modO/Linux.intel.64.intelmpi.default
export LIS_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default
export LIS_MINPACK=/discover/nobackup/projects/lis/libs/minpack/intel_11_1_038
export LIS_CRTM=/discover/nobackup/projects/lis/libs/JCSDA_CRTM/REL-2.0.2.Surface-rev
export LIS_CRTM_PROF=/discover/nobackup/projects/lis/libs/CRTM_Profile_Utility/intel_11_1_038/
export LIS_CMEM=/discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117
export LIS_MEM=/discover/nobackup/projects/lis/libs/LIS-MEM/intel_13.0.1.117
export LIS_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192

export LVT_ARCH=linux_ifc
export LVT_FC=mpif90
export LVT_CC=mpicc
export LVT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192
export LVT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117
export LVT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2r4_noNETCDF_intel_13_0_1_117/
export LVT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos/
export LVT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117
export LVT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/mod/modO/Linux.intel.64.intelmpi.default
export LVT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default
export LVT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192

export LDT_ARCH=linux_ifc
export LDT_FC=mpif90
export LDT_CC=mpicc
export LDT_GRIBAPI=/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192
export LDT_NETCDF=/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117
export LDT_HDF4=/discover/nobackup/projects/lis/libs/hdf4/4.2r4_noNETCDF_intel_13_0_1_117/
export LDT_HDFEOS=/discover/nobackup/projects/lis/libs/hdfeos/
export LDT_HDF5=/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117
export LDT_MODESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/mod/modO/Linux.intel.64.intelmpi.default
export LDT_LIBESMF=/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default
export LDT_JASPER=/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/discover/nobackup/projects/lis/libs/esmf/5_2_0rp3_intel_13_0_1_117_impi_4_0_3_008/lib/libO/Linux.intel.64.intelmpi.default/:/discover/nobackup/projects/lis/libs/grib_api/1.12.3_intel_13_1_3_192/lib:/discover/nobackup/projects/lis/libs/hdf5/1.8.8_intel_13_0_1_117/lib:/discover/nobackup/projects/lis/libs/netcdf/4.1.3_intel_13_0_1_117/lib:/discover/nobackup/projects/lis/libs/jasper/1.900.1_intel_13_1_3_192/lib

Submitting a PBS job script into SLURM

The SLURM project provides wrappers for PBS commands, so your existing scripts will still work.  All that you must do is add one line to the top of your script.

 

For example, if you used to run

 

qsub lis.job

 

then first edit the lis.job script.  Add this line to the top of that file:

 

#!/bin/sh

 

Then run

 

qsub lis.job

 

as before.

 


Using native SLURM

Switching from PBS commands to SLURM commands is easy. NCCS read "First Introduction to Using SLURM on Discover" [http://www.nccs.nasa.gov/images/SLURM-intro_3-mod.pdf]. These slides show how to easily convert a PBS job script into a SLURM job script.

 

NCCS has hosted two brown bags (http://www.nccs.nasa.gov/list_brown_bags.html) discussing SLURM.  Please read "Introduction to the SLURM resource manager" [http://www.nccs.nasa.gov/images/intro-to-slurm-20131218.pdf]. These slides provide a good overview of the basic SLURM commands.

 

Please note that the second brown bag on SLURM has not yet been posted.

 


Obtaining an interactive session with X11 fowarding

To obtain an interactive session with X11 fowarding enabled, you run the xsub command as with PBS.  For example:

 

xsub -I -l select=1:ncpus=16,walltime=1:00:00 -W group_list=s0942

 

will give you one node with 16 cpu for 1 hour.

0

ESMF 3.1.0rp2 Alarm Issue

Posted by jvgeiger May 18, 2010

There is a bug with respect to alarms in ESMF 3.1.0rp2.  The bug is such that if you create an alarm that rings at the same frequency as the clock, then the alarm will not ring properly.  For example, you create a clock with a timestep of 15 minutes, then you create an alarm that rings every 15 minutes. The alarm will not ring every 15 minutes as expected.

 

This bug has been fixed.  Please use ESMF 3.1.0rp3.  On discover, this version is located in /discover/nobackup/projects/lis/Code/esmf_3_1_0rp3_intel_11_1_038_impi_3_2_011.

 

Actions

Notifications

Popular Tags

USAGov logo NASA Logo - nasa.gov