Skip navigation
NASA Logo, National Aeronautics and Space Administration
Modeling Guru Banner
Currently Being Moderated

Some Hints about Open MPI

VERSION 11  Click to view document history
Created on: Mar 10, 2011 4:31 PM by Amidu Oloso - Last Modified:  Feb 25, 2014 10:42 AM by Amidu Oloso

OpenMPI Version 1.4.3 has been experimented with in the process of improving the performance of Enzo. The exercise was motivated by Enzo performing sub-optimally on discover at NCCS. The following Open MPI environmental variable was found to reduce the communication time for the test case benchmarked from about 2600 seconds to about 800 seconds thereby bringing the overall wall time down to about 1800 seconds from about 3600 seconds. The environmental variable and its value was:

export OMPI_MCA_btl_openib_rd_num=128
export OMPI_MCA_btl_openib_rd_low=75
export OMPI_MCA_btl=self,openib
export OMPI_MCA_btl_max_eager_rdma=2
export OMPI_MCA_btl_eager_rdma_threshold=0
export OMPI_MCA_btl_use_eager_rdma=1
export OMPI_MCA_btl_openib_rdma_num=128


OMPI_MCA_btl self,openib


The particular build of Open MPI used for testing was configured as follows:


./configure --enable-openib-ibcm --with-openib --prefix=<Installation_Directory>


where <Installation_Directory> is the location of "bin", "lib", and "include" directories.


A build is presently available at /discover/nobackup/aoloso/openmpi-1.4.3. This build was done with compiler module comp/intel-11.1.072


To use the above build of OpenMPI, do the following:

  • add /discover/nobackup/aoloso/openmpi-1.4.3/bin to PATH
  • add /discover/nobackup/aoloso/openmpi-1.4.3/lib to LD_LIBRARY_PATH
  • explicitly define TMPDIR environmental variable


For example for bash family of shells, you can do:


export TMPDIR=/tmp
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/discover/nobackup/aoloso/openmpi-1.4.3/lib"
export PATH="/discover/nobackup/aoloso/openmpi-1.4.3/bin:${PATH}"


After testing is complete, it is possible that a build will be available in /usr/local/other on discover with an associated module created for that build. In that case, all a user should need to do is load the module and will then not need to explicitly modify the "PATH" and the "LD_LIBRARY_PATH" variables.

Comments (0)
USAGov logo NASA Logo -