Add all scripts within a repository to $PATH using envmaster

I have a couple of general scripts repositories for myself and shared with colleagues. These are for scripts which are useful but don’t fit into existing projects or justify having their own repository. An example is rsgis_scripts on bitbucket. The scripts are split into different directories which aren’t available on the main path. To make them available when needed I use EnvMaster (described in a previous post). I have an envmaster module which will search the repository for folders containing executables and add them to $PATH. It will also add folders containing ‘__init__.py’ to $PYTHONPATH.

#%EnvMaster1.0
import os
import glob
 
REPO_PATH = "/home/dan/Documents/Development/rsgis_scripts"
 
# Walk through directory
for d_name, sd_name, f_list in os.walk(REPO_PATH):
    # Ignore hidden directories
    if not d_name.startswith(".") and not ".git" in d_name and not ".hg" in d_name:
        file_list = glob.glob(os.path.join(d_name,"*"))
        # Check if a directory contains executable files
        if True in [(os.path.isfile(f) and os.access(f, os.X_OK)) for f in file_list]:
            module.setBin(d_name)
        # Check for Python libraries
        if len(glob.glob(os.path.join(d_name,"*","__init__.py"))) > 0:
            module.setPython(d_name)

The script is saved as ‘rsgis_scripts’ in ‘$ENVMASTERPATH’.
To load the module and prepend all the folders to ‘$PATH’ use:

envmaster load rsgis_scripts

They will remain on $PATH until you close the terminal or unload the module using:

envmaster unload rsgis_scripts

This script is just an example, it would be easy to modify for different use cases. If you have multiple repositories a module file can be created for each one.

Managing Software & Libraries with EnvMaster

If you’ve compiled software from source you’re probably used to the following sequence:

./configure
make
sudo make install

Which will configure the software to look for libraries and install itself to the default location (normally /usr/local), make and install. As root privileges are required to install to /usr/local, sudo is required.

This is fine unless:

1. You’re not in the sudo’ers list (e.g., on a shared computer you’re not an administrator for).
2. You want to have different versions of things (e.g., stable and development versions).

then things start to get complicated. Installing all the software to a single folder you have permission to write to, e.g., ~/software, by passing the –prefix=~/software flag to configure, then adding ~/software/bin to your $PATH and ~/software/lib to $LD_LIBRARY_PATH would solve the first problem but will still cause problems if you want different versions of things.

Ideally you’d install everything into it’s own directory something like:

~/software/
          rsgislib/
                  2.0.0
                  20131019
          gdal/
              1.10.1
              1.9.0

which means you’re going to be spending a lot of time hacking around with environmental variables!

This is where EnvMaster comes in, it allows you to install different versions of software and libraries, in a nicely organised directory structure, and sorts out all the paths for you. When it’s properly set up you can load and swap software / libraries round using:

# Load in RSGISLib
envmaster load rsgislib

# Swap to use the developement version of RSGISLib
envmaster swap rsgislib/2.0.0 rsgislib/20131019

To install EnvMaster clone the source from the EnvMaster Bitbucket page and install:

hg clone https://bitbucket.org/chchrsc/envmaster envmaster

export ENVMASTER_ROOT=~/software/envmaster
python setup.py install --prefix=$ENVMASTER_ROOT

Note: as with the rest of the examples, I’m installing to ~/software (where ~ is your home directory). You can use anywhere you have permission to write, just do a mental find and replace, wherever you see ~/software with the path your using. We normally install things to /share/osgeo/fw/

EnvMaster uses a corresponding text file for each library, these need to be stored in a separate directory (ideally on there own). We’ll make one called modules.

mkdir -p ~/software/modules

Once this is set up, create a text file (lets call it ~/software/setupenvmaster) containing the following:

export PATH=$ENVMASTER_ROOT/bin:$PATH
export PYTHONPATH=$ENVMASTER_ROOT/lib/python2.7/site-packages
# Set up path to modules files
export ENVMASTERPATH=~/software/modules
# Initialize EnvMaster
. $ENVMASTER_ROOT/init/bash

And source it using:

. ~/software/setupenvmaster

If you add this line to .bashrc / .bash_profile envmaster will be available in every new terminal.

Note, if you used a different version of python to install envmaster (e.g., python3.3) you need to change pythonX.X to reflect this.

Now if you type

envmaster avail

You should see ‘~/software/modules’ and ‘No module files found’.

EnvMaster is now all set up and it’s time to start installing things.

Let’s install GDAL, first download from here and untar.

./configure --prefix=/home/dan/software/gdal/1.10.1
make
make install

Note, you probably want to install GDAL with other options (e.g., HDF5), see the RSGISLib documentation for the options recommended if building for RSGISLib.

You then need to set up the files for EnvMaster. Within ~/software/modules make a directory called gdal and create two text files, one called 1.10.1 (the version of gdal) and the other called version.py. In 1.10.1 put the following:

#%EnvMaster1.0

module.setAll('/home/dan/software/gdal/1.10.1')

The first line tells EnvMaster this is an EnvMaster file, module.SetAll() sets environmental variables (PATH etc.,) based on the contents of ~/software/gdal/1.10.1.
In version.py put the following:

#%EnvMaster1.0

version = '1.10.1'

This tells EnvMaster the default version of GDAL is 1.10.1

If you run envmaster avail again it should list gdal. You can load GDAL using:

envmaster load gdal

To unload GDAL (and unset paths) use:

envmaster unload gdal

You can try running gdal_translate to check.
To see the environmental variables GDAL set use:

envmaster disp gdal

As well adding the path to general variables (e.g., PATH), EnvMaster has created variables specific to GDAL (e.g., GDAL_LIB_PATH), these are useful when linking from other software.

To see the envmaster modules you have loaded you can use:

envmaster list

There are loads more options available in EnvMaster than shown here. Whilst it does take a bit of time to set up, it allows you to build a highly organised and very flexible system. The user guide for EnvMaster, in LyX format, is available with the source code from here.

A comprehensive list of instructions for building software with EnvMaster under Linux is available from Landcare (here). Note: These were developed for their system and have very generously been made available. Use at your own risk.
Pete Bunting’s instructions for building under OS X are also available here, these have been tested under OS X 10.9.