Installing RSGISLib on Windows through a Virtual Machine

Although some efforts have been made to create windows versions of the software in the stack we use, the main platform we’re using and developing on is Linux (or OS X). Therefore, the advice we normally give people wanting to use RSGISLib or SPDLib on Windows is to install Linux in a virtual machine and use this.

Below are the steps needed to get everything working on a virtual machine. All these steps assume a 64-bit version of Windows (as you’ll need to install a 64-bit version of Linux). There is lots on information on the internet which provides more detail on the specifics of setting up a virtual machine and installing Linux, so I’ve just provided an overview of these.

1. Download and install VirtualBox

Download the free VirtualBox from https://www.virtualbox.org/ and install it.

2. Download a Linux installation image

There are a number of Linux distributions available. The recommenced distribution is Ubuntu, which is available to download from http://www.ubuntu.com/download. If you’re computer has limited RAM (less than 4 GB), you might want to consider a more lightweight version such as lubunbu

Whichever distribution you chose make sure you download a 64-bit desktop CD / DVD image. Note: you don’t actually have to burn the image to a CD, the iso file is fine.

3. Set up a new virtual machine

The VirtualBox documentation has a good step-by-step guide to setting up a virtual machine (http://www.virtualbox.org/manual/ch01.html) and there is lots of documentation available on the internet. Important considerations are to allocate enough space for the virtual machine hard drive, if you select a dynamically allocated hard drive it won’t use the full capacity at once but will be able to grow as you need it. Also give the virtual machine as much RAM as you can spare.

4. Install Linux

When you first start the virtual machine you will be prompted to install an operating system, select the Linux .iso file you have downloaded and it should boot off this.

The installation procedure will depend on the Linux distribution you have. Instructions for Ubuntu are here. At some point in the installation it might warn you about erasing all data on the disk, don’t worry about this it’s only the virtual disk you have just created.

5. Install guest additions

Once Linux has been installed you need to install the ‘Guest additions’, to do this select Devices, ‘Install Guest Additions CD Image’ from the VirtualBox menu, then follow the prompts within Linux.

6. Setup for copying data

The easiest way to copy data to and from the machine is to enable drag and drop. Once the machine is turned on go to Devices, Drag and Drop, and select ‘Bidirectional’. This will allow you to drag and drop files to the virtual machine. You can also set up a shared folder, there are some details on this here.

7. Install the software stack

As detailed in earlier posts, we make Linux binaries of the software we develop available through conda. To get everything installed, follow the steps below within the virtual machine:

• Install conda

Download the latest version from here, grab the 64-bit Python 3.5 installer. Once the file has downloaded open a terminal window (in ubuntu, click on the icon in the top left and start typing ‘Terminal’, click on the application which pops up).

Within a terminal window type the following:

cd ~/Downloads
bash Miniconda3-*-Linux-x86_64.sh

• Install software from binstar

Open a new terminal window and type the following:

# Install anaconda client
conda install anaconda-client

# Install software to separate environment
conda install anaconda-client
conda env create au-eoed/au-osgeo-lnx

# Activate environment
source activate au-osgeo-lnx

# Use environment for all new terminals (optional)
echo source activate au-osgeo-lnx >> ~/.bashrc

• Test

Open a new terminal window, you should be able to type:

tuiview

To open TuiView

As you’ll be interacting with the software from the command line, you will need to learn some commands. The Software Carpentry shell course (http://swcarpentry.github.io/shell-novice/) is a good starting point. The Linux Command Line book is also recommended.

Version control with Mercurial

I’ve been quite enthusiastically advocating version control software (specifically mercurial) recently amongst some colleagues for managing code. Whilst the principles of version control are pretty familiar to most people, there are better ways than creating multiple copies with different names such as _final_final_verson2!

Version control software provides a better way of not only retaining different versions of files but allowing you to compare changes between them so you can easily find the lines that have been changed, particularly useful if you’ve just broken something! Mercurial and Git are two popular examples of version controls software. What’s nice about these (compared to other version control software such as subversion) is that all the changes are kept locally so you just need the software installed on your computer to have a folder under version control. For small projects, with only a couple of files, I just have local mercurial repositories set up.

When multiple people are working on the same code, these software work best when combined with hosting such as that provided by Bitbucket or github. My personal preference is Bitbucket as it offers mercurial (whereas github uses only git), allows private repositories for free (with unlimited users if you’re in academia) and I think it has a nicer interface. A lot of the software we’re developing / using is on Bitbucket (see bitbucket.org/petebunting/ and bitbucket.org/chchrsc/).

The command line interface for mercurial can be downloaded for Windows and OS X from http://mercurial.selenic.com, or can be installed from the package manager if you’re using linux:

sudo apt-get install mercurial

If you’d rather use a GUI than the command line TortoiseHG is a good option for Windows or sourcetree for Windows or OS X.

There are loads of good resources for learning mercurial on the internet, hginit.com is a particularly well written one and a good place to start. I’ve also put together a quick ‘cheat sheet’ with the commands I use most frequently.

Mercurial Cheat Sheet

 hg clone REMOTE LOCAL # Create local copy.
 hg pull -u # Pull changes from remote server (bitbucket) and update local copy.
 hg merge # Merge changes from remote server with local changes (sometimes required after pulling).
 hg commit # Commit changes locally (see setting up hgrc).
 hg push # Push changes to remote server (bitbucket).
 hg add # Add new file to version control (need to commit after).
 hg remove # Remove a file from filesystem and version control (need to commit after).
 hg mv # Move a file in the filesystem and version control (need to commit after).
 hg tag # Create name for revision (e.g., version number).
 hg update -r REV/TAG # Revert to revision or tag.
 hg revert FILE # Revert file to last commit (can also revert to revision)
 hg log # List all commits (good idea to pipe to less under UNIX).
 hg status # Show status of all files, can use -m to only show modified files.
 hg init # Initialise mercurial repository in directory (for new repositories only).

Setting up hgrc
The first time you commit you will likely an error ‘no username supplied’ to fix this you need to add the following lines to .hg/hgrc

 [ui]
 username = Firstname Lastname <email>
 verbose = True
 [paths]
 default = PATH TO REPOSITORY

The tag command is particularly useful for creating version numbers (e.g., version1.1) or noting the version of code used for publications etc. It does take some getting into but you soon get used to running ‘hg commit’ whenever you make changes.

Managing Software & Libraries with EnvMaster

If you’ve compiled software from source you’re probably used to the following sequence:

./configure
make
sudo make install

Which will configure the software to look for libraries and install itself to the default location (normally /usr/local), make and install. As root privileges are required to install to /usr/local, sudo is required.

This is fine unless:

1. You’re not in the sudo’ers list (e.g., on a shared computer you’re not an administrator for).
2. You want to have different versions of things (e.g., stable and development versions).

then things start to get complicated. Installing all the software to a single folder you have permission to write to, e.g., ~/software, by passing the –prefix=~/software flag to configure, then adding ~/software/bin to your $PATH and ~/software/lib to $LD_LIBRARY_PATH would solve the first problem but will still cause problems if you want different versions of things.

Ideally you’d install everything into it’s own directory something like:

~/software/
          rsgislib/
                  2.0.0
                  20131019
          gdal/
              1.10.1
              1.9.0

which means you’re going to be spending a lot of time hacking around with environmental variables!

This is where EnvMaster comes in, it allows you to install different versions of software and libraries, in a nicely organised directory structure, and sorts out all the paths for you. When it’s properly set up you can load and swap software / libraries round using:

# Load in RSGISLib
envmaster load rsgislib

# Swap to use the developement version of RSGISLib
envmaster swap rsgislib/2.0.0 rsgislib/20131019

To install EnvMaster clone the source from the EnvMaster Bitbucket page and install:

hg clone https://bitbucket.org/chchrsc/envmaster envmaster

export ENVMASTER_ROOT=~/software/envmaster
python setup.py install --prefix=$ENVMASTER_ROOT

Note: as with the rest of the examples, I’m installing to ~/software (where ~ is your home directory). You can use anywhere you have permission to write, just do a mental find and replace, wherever you see ~/software with the path your using. We normally install things to /share/osgeo/fw/

EnvMaster uses a corresponding text file for each library, these need to be stored in a separate directory (ideally on there own). We’ll make one called modules.

mkdir -p ~/software/modules

Once this is set up, create a text file (lets call it ~/software/setupenvmaster) containing the following:

export PATH=$ENVMASTER_ROOT/bin:$PATH
export PYTHONPATH=$ENVMASTER_ROOT/lib/python2.7/site-packages
# Set up path to modules files
export ENVMASTERPATH=~/software/modules
# Initialize EnvMaster
. $ENVMASTER_ROOT/init/bash

And source it using:

. ~/software/setupenvmaster

If you add this line to .bashrc / .bash_profile envmaster will be available in every new terminal.

Note, if you used a different version of python to install envmaster (e.g., python3.3) you need to change pythonX.X to reflect this.

Now if you type

envmaster avail

You should see ‘~/software/modules’ and ‘No module files found’.

EnvMaster is now all set up and it’s time to start installing things.

Let’s install GDAL, first download from here and untar.

./configure --prefix=/home/dan/software/gdal/1.10.1
make
make install

Note, you probably want to install GDAL with other options (e.g., HDF5), see the RSGISLib documentation for the options recommended if building for RSGISLib.

You then need to set up the files for EnvMaster. Within ~/software/modules make a directory called gdal and create two text files, one called 1.10.1 (the version of gdal) and the other called version.py. In 1.10.1 put the following:

#%EnvMaster1.0

module.setAll('/home/dan/software/gdal/1.10.1')

The first line tells EnvMaster this is an EnvMaster file, module.SetAll() sets environmental variables (PATH etc.,) based on the contents of ~/software/gdal/1.10.1.
In version.py put the following:

#%EnvMaster1.0

version = '1.10.1'

This tells EnvMaster the default version of GDAL is 1.10.1

If you run envmaster avail again it should list gdal. You can load GDAL using:

envmaster load gdal

To unload GDAL (and unset paths) use:

envmaster unload gdal

You can try running gdal_translate to check.
To see the environmental variables GDAL set use:

envmaster disp gdal

As well adding the path to general variables (e.g., PATH), EnvMaster has created variables specific to GDAL (e.g., GDAL_LIB_PATH), these are useful when linking from other software.

To see the envmaster modules you have loaded you can use:

envmaster list

There are loads more options available in EnvMaster than shown here. Whilst it does take a bit of time to set up, it allows you to build a highly organised and very flexible system. The user guide for EnvMaster, in LyX format, is available with the source code from here.

A comprehensive list of instructions for building software with EnvMaster under Linux is available from Landcare (here). Note: These were developed for their system and have very generously been made available. Use at your own risk.
Pete Bunting’s instructions for building under OS X are also available here, these have been tested under OS X 10.9.

TuiView

TuiView is an open source viewer for remote sensing data, named after the New Zealand bird (Tui). It’s primarily for viewing raster data but it will also display vectors. I’ve been using it as my main remote sensing viewer for over a year and I’d highly recommend giving it a try.

It works on windows, OS X and Linux, if you’re on windows, you can grab an installer from here and you should be good to go (don’t worry about there only being 32 bit installers it’s pretty light on memory).

If you’re on Linux / OS X you binaries are available through conda from binstar.org/osgeo or you can install from source.

To install from source you need to have GDAL, numpy and PyQt installed. I’d recommend installing GDAL from source and making sure you build with python bindings (–with-python). For Linux you can install numpy and PyQt from the package manager. Under OS X, assuming you already have numpy, you need to install Qt4, then install sip and PyQt4. If you want to view vectors you also need turbogdal.

After the prerequisites have been installed you can install tuiview using:

hg clone https://bitbucket.org/chchrsc/tuiview tuiview
cd tuiview
sudo python setup.py install

You should then be able to bring up TuiView by typing tuiview in a terminal window. You can load in an image by clicking on the first icon on the left.

TuiView can handle very large (> 50 GB) datasets, very quickly if you build pyramids for the dataset and pre-calculate statistics. You can use gdalcalcstats from gdalutils, or the popimgstats option in RSGISLib

gdalcalcstats image.kea -ignore 0
# or
rsgiscmd --algor imageutils --option popimgstats \
   --image image.kea --ignore 0 \
   --pyramids yes

Note: The rsgiscmd interface is shown here, you could also use XML or the Python bindings.
It’s a good habit to create pyramids and calculate statistics for outputs at the end of your scripts as other programs (such as ArcGIS and ERDAS Imagine) can also use them.

TuiView has lots of options, including the ability to load multiple images in the same or linked windows, and excellent support for raster attribute tables. A comprehensive guide to the functionality is available on the TuiView wiki.