Skip navigation
NASA Logo, National Aeronautics and Space Administration
Currently Being Moderated

Create your Own Python Distribution through Virtual Environments

VERSION 8  Click to view document history
Created on: Feb 22, 2018 2:34 PM by Jules Kouatchou - Last Modified:  Mar 25, 2020 5:15 PM by Jules Kouatchou

There are several Python distribution modules available on discover. They have been installed over time to meet various users/groups’ needs. It is at times very difficult to keep up with users’ demands to add new packages to existing Python distributions. At several occasions, we have ran into dependency conflicts and have ended up creating new Python modules on discover because we were not able to install two groups of packages within the same distribution.

 

With the growing number of Python users and the desire to have all their requests met, we believe that the best way to move forward (in the creation of Python distributions on discover) is to do the following:

 

  • Create a core Python distribution with some specific packages such Python, Ipython, Jupyter, Numpy, SciPy, Matplotlib, Virtualenv, Pip, Pipenv, Pandas, etc. We will use as starting point the most recent version of the Anaconda Python distribution and add few other packages. As soon as a new distribution is created, it will remain frozen.
  • Allow users to create their own Python environment (though Virtualenv or Pipenv) if they wish to add more packages. They will then have their own Python distributions that they could manage as they wish.

 

The advantage of such an approach is that users:

  1. Will have full control of what they want (in terms of Python related packages)
  2. Will no longer have to wait for the NCCS Application Team for a package to be installed.
  3. Will be able to create as many Python distributions (with different versions of existing packages for instance) as they wish.

 

Throughout the process, the NCCS Application Team will be available to provide guidance as needed.

 

We describe below the basic command to use Virtualenv and Pipenv on discover.

 

As a starting point, you need to load (from a SLESS 12 node) the module (no dependency):

 

python/GEOSpyD/Ana2019.10_py3.7

 

The above Python distribution contains the Python core packages we listed above. You should then have access of the Virtualenv and Pipenv tools.

 

From your NOBACKUP space (do not do it from your home directory) on discover, you need to create an new directory (say myProjectFolder):

 

mkdir myProjectFolder

 

 

Virtualenv

virtualenv is a tool to create isolated Python environment. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either). To create a Python virtual environment, issue the commands:

 

cd myProjectFolder
virtualenv --system-site-packages myVirtualPythonEnv

 

 

Here myProjectFolder is the directory (if possible not in your home directory) where you want to start from and myVirtualPythonEnv is the name of the Python virtrual envirionment you are creating. The above virtualenv command will create a full image Python distribution (contains in the above module) in the directory myVirtualPythonEnv.

 

You now have a Python distribution in your own space. You will be free to add/remove any package you wish. Now how do you use the distribution? You need to activate:

 

Activate with C-Shell

 

setenv PATH Full_Path_to_your_Python_Distribution/bin:${PATH}
setenv LD_LIBRARY_PATH Full_Path_to_your_Python_Distribution/lib:${LD_LIBRARY_PATH}

 

Activate with Bash-Shell

source Full_Path_to_your_Python_Distribution/bin/activate

 

From there, you can add packages using pip:

 

pip install packageName

 

or from source:

 

python setup.py install

 

At any time, you can activate your environment (as shown above) and use your Python distribution.

 

Pipenv

Pipenv is a dependency manager for Python projects. With pipenv:

  • You no longer need to use pip and virtualenv separately as above. They work together.
  • You have insight into your dependency graph (e.g. “pipenv graph”)

 

You first need to initiate Pipenv:

 

cd myProjectFolder
pipenv install

 

This will create two new files, Pipfile and Pipfile.lock, in your project directory, and a new virtual environment for your project if it doesn’t exist already.

 

To install a Python package for your project, use the install keyword. For example

 

pipenv install packageName

 

will install the current version of packageName. A package can be removed in a similar way with the uninstall keyword.

 

To activate the virtual environment associated with your Python project, use the command:

 

pipenv shell

 

To run you python script in your environment:

 

pipenv run python myPythonScript.py

 

 

References:

 

  1. Pipenv & Virtual Environments
  2. Pipenv: Python Development Workflow for Humans
  3. How to Mange your Python Projects with Pipenv
  4. VIDEO: Python Tutorial: virtualenv and why you should use virtual environments
  5. Installing Pakages
  6. Pip Quick Start
Comments (0)
USAGov logo NASA Logo - nasa.gov