Apply the same stretch to multiple files in TuiView

There are some cases when it is necessary to play around with the stretch used to display data in TuiView (edit -> stretch). For example, if the scene contains very bright or dark areas (e.g., clouds/shadows) this can skew the statistics and make the images display too light/dark. One option is to use a local stretch (round house icon), which will only use values from the current display to calculate statistics. You can also manually set the minimum and maximum values.

Once a good stretch has been obtained this can be saved to the image for formats which support it (e.g., KEA) using the ‘Save Stretch and Lookup Table’ button (disk icon). If multiple files need to be opened with the same stretch applied you can pass in a file with a saved stretch when opening TuiView e.g.,

tuiview --stretchfromgdal image_with_saved_stretch.kea *kea

This will open all files matching the pattern ‘*bil’, applying the stretch saved in ‘image_with_saved_stretch.kea’

You can also save a stretch to a text file using the ‘Export Stretch and Lookup Table to Text File’ button (disk with ABC icon), this can be passed in when opening TuiView using

tuiview --stretchfromtext casi.stretch *bil

This will open all files matching the pattern ‘*bil’, applying the stretch saved in the text file ‘casi.stretch’

Viewing hyperspectral data in TuiView

As TuiView is based on GDAL, it has always been able to open the ENVI files commonly used for hyperspectral data. However, there have been some recent enhancements to TuiView which have improved handling of hyperspectral data.

TuiView can be installed for Linux, OS X and Windows through conda using:

conda install -c rios tuiview

The source is available to download from https://bitbucket.org/chchrsc/tuiview/.

As hyperspectral data contain many bands, specifying the ones to use for display when you load the file in makes things easier. You can do this from the command line using the ‘–bands’ argument. For example:

tuiview --rgb --stddev --bands 280,460,160 f159223b_mapped_osng.bil

For the Specim Fenix dataset used as an example the command will display a colour composite of NIR, SWIR and Red at approximately the centre wavelengths of the Landsat 8 bands with a standard deviation stretch applied to the pixel values. Opening the image may take a while as the statistics need to be calculated for the the three bands selected for display in order to apply a standard deviation stretch.

Once the file has opened you can use the ‘Query tool’ to display spectral profiles for each pixel.

If you get an error that TuiView is unable to open the file it could be that ‘data ignore value = 0’ is set in the header and the GDAL driver is unable to handle it. You can export the environmental variable:

export GDAL_PAM_ENABLED=ON

on Linux/OS X, or:

set GDAL_PAM_ENABLED=ON

on Windows. This will create a ‘*.aux.xml’ metadata file with additional features the driver doesn’t support when TuiView opens the file. More information is available in the GDALPamDataset documentation.

If you have hyperspectral data which isn’t mapped (e.g., level1b / level2 data in the NASA Data processing levels) TuiView will not open these by default and will give an error about only allowing north-up images. If you know the data don’t have geospatial information you can force TuiView to open them by setting:

export TUIVIEW_ALLOW_NOGEO=YES

on Linux/OS X, or:

set TUIVIEW_ALLOW_NOGEO=YES

on Windows.

More details about available environmental variables used by TuiView are available on the TuiView Wiki.

Hyperspectral data from the NERC ARSF used for this post were provided courtesy of NERC via the NERC Earth Observation Data Centre (NEODC).

A Python-Based Open Source System for Geographic Object-Based Image Analysis (GEOBIA) Utilizing Raster Attribute Tables – Bonus Features

This is the first post, of what I hope will be a regular feature, with some ‘Bonus Features’ of recently published papers I have been involved with. The idea is to provide some of the details considered too technical (or trivial) for an academic publication, code snippets, a bit of backstory and other things I think might be of interest.

This first post is on:

Clewley, D.; Bunting, P.; Shepherd, J.; Gillingham, S.; Flood, N.; Dymond, J.; Lucas, R.; Armston, J.; Moghaddam, M. A Python-Based Open Source System for Geographic Object-Based Image Analysis (GEOBIA) Utilizing Raster Attribute Tables. Remote Sensing 2014, 6, 6111-6135.

The article is open access and is available to download from:

http://www.mdpi.com/2072-4292/6/7/6111

1. Installing the software.

   Binaries for OS X and Linux are made available through conda. See the software page for more instructions.

2. RFC40 and RSGISLib 2.1 / 2.2 differences

   Whilst we were writing the paper there were some major changes in RSGISLib due to the release of GDAL 1.11 which included the RFC40 changes proposed and implemented by Sam and Pete. These changes meant that rather than loading the entire RAT to memory (which was fast but the size of the RAT which could be processed was limited by the amount of available RAM), rows were accessed from disk (which was slower but removed the RAM limitation). For RSGISLib Pete and myself removed all the RAT functions after the 2.1 release and started adding them back to take advantage of the new RAT interface in GDAL 1.11. For some functions this required reworking the algorithm. After a pretty intense couple of days coding we managed to port most of the main functions across, and we’re gradually adding the rest. Therefore, some of the functions listed in the paper as available in 2.1 aren’t available in the latest release yet.

   In RIOS, Neil and Sam added a ‘ratapplier’ interface to process large RATs chunks at a time, similar to the ‘applier’ interface for processing images. More information is in the RIOS documentation. The ratapplier interface is backwards compatible with pre-RFC40 versions of GDAL (although all rows are loaded at once).

   The RFC40 changes in TuiView allowed massive RATs (e.g., segmentations for Australia and Alaska) to be easily visualised on moderate specification machines. Being able to open and navigate a RAT with 10s of millions of rows on a laptop really is an impressive feat!

3. Software Comparison

   To run the segmentation in RSGISLib and OTB the following scripts were used (note OTB isn’t currently available in conda, for the paper we installed through the ubuntugis-unstable package repository).

   RSGISLib (Python):

   from rsgislib.segmentation import segutils
    
   inputImage = '../Data/naip_newhogansouth_2012_sub.tif'
   clumpsFile = 'naip_newhogansouth_2012_clumps_elim_final.kea'
     
   # Run segmentation
   segutils.runShepherdSegmentation(inputImage, clumpsFile,
                              numClusters=60, minPxls=100, 
                              distThres=100, bands=None, 
                              sampling=100, kmMaxIter=200)
   

   OTB (Bash):

   otbcli_Segmentation -in ../Data/naip_newhogansouth_2012_sub.tif \
       -mode.vector.out naip_newhogansouth_2012_otb_seg.shp
   

   To time the segmentation the UNIX ‘time’ command was used.

   There are lots of algorithms in OTB for segmentation so if you find the algorithm in RSGSILib doesn’t quite fit your requirements I’d highly recommend trying OTB. Although OTB can produce a raster output the vector output algorithm is able to process larger datasets and utilise multiple cores. To convert to a raster gdal_rasterize can be used. For example:

   gdal_rasterize -a DN -tr 30 30 -ot UInt32 \
      -of KEA segmentation_vector.shp \
      segmentation_raster.kea
   

   To convert to a RAT the following Python function can be used:

   from rsgislib import rastergis
   clumps='segmentation_raster.kea'
   pyramids=True
   colourtable=True
   rastergis.populateStats(clumps, colourtable, pyramids)
   

   This can then be used exactly the same as the segmentation produced in RSGISLib. We think the ability to use different segmentation algorithms, from different packages, is a real benefit of the system.

4. Examples

   The ‘Change in Mangroves Extent’ example was part of a course Pete and Myself gave at JAXA’s 20th Kyoto and Carbon Meeting (agenda and presentations available here, full workshop available to download from SourceForge). For segmenting the image and attributing objects there are two utility functions available in RSGISLib, described in an earlier post. For the course and paper we used the old RIOS RAT interface. However, it is recommended to use the new ratapplier interface. As an example the classification of water would be:

   from rios import ratapplier
   import numpy
   
   def classifyWater(info, inputs, outputs):
       # Read the 1996 JERS-1 Mean dB values for the clumps
       # The column is represented as a numpy array of size 
       # block length x 1.
       HH96MeandB = inputs.inrat.HH96MeandB
       # Create a new numpy array with the same dimensions (i.e., length)
       # as the 'HH96MeandB' array. The data type has been defined as
       # an 8 bit integer (i.e., values from -128 to 128).
       # All pixel values will be initialised to zero
       Water96 = numpy.zeros_like(HH96MeandB, dtype=numpy.int8)
       # Similar to an SQL where selection the where numpy where function
       # allows a selection to be made. In this case all array elements 
       # with a 1996 HH value less than -12 dB are being selected and 
       # the corresponding elements in the Water96 array will be set to 1.
       Water96 = numpy.where((HH96MeandB < -12), 1, Water96)
       # Save out to column 'Water96'
       outputs.outrat.Water96 = Water96
   
   # Set up inputs and outputs for ratapplier
   inRats = ratapplier.RatAssociations()
   outRats = ratapplier.RatAssociations()
   
   inRats.inrat = ratapplier.RatHandle('N06W053_96-10_segs.kea')
   outRats.outrat = ratapplier.RatHandle('N06W053_96-10_segs.kea')
   
   print('Classifying water')
   ratapplier.apply(classifyWater, inRats, outRats)
   

If you have any questions or comments about the system described in the paper email the RSGISLib support group (rsgislib-support@googlegroups.com).

Export quicklooks with vector overlay using TuiView

Often when you have a large stack of images (e.g., a time series) it’s useful to create quick looks so you can flick through them. Normally I use gdal_translate with the ‘-scale’ and ‘-outsize’ flags, as described in a previous post.

Recently I wanted to quickly export all bands of an image for a presentation, with some points overlain. The TuiView manual describes how it’s possible to automate the ‘Save Current Display’ menu option using Python, which sounded ideal for this task.

Following the instructions to export a single image in the manual, I added a loop to export each band and name the output file with the band name, which produced exactly the result I needed. As this is something I’m likely to need to do again, I tidied the script up, made it a bit more general and added some more of the options available. The script can be downloaded from RSGIS Scripts on Bitbucket.

To export all bands in an image (original task):

python export_quicklook_tuiview.py --inimage tonzi_data_layers.kea \
       --outimage tonzi_data_layers_ql.png \
       --invector sensor_locations.shp \
       --allbands

I also added the options to export an image using the default stretch:

python export_quicklook_tuiview.py --inimage tonzi_data_layers.kea \
       --invector sensor_locations.shp \
       --outimage tonzi_data_layers_ql.png

Or if the data has already been stretched and RGB mapped to bands 1,2,3

python export_quicklook_tuiview.py --inimage tonzi_data_2sd_rgb.kea \
       --invector sensor_locations.shp \
       --outimage tonzi_data_rgb_2sd_ql.png --rgb --nostretch

To add text to the quick looks (e.g., the filename) imagemagick can be used:

convert in_image.png -gravity north -strokewidth 10 \
-pointsize 100 -annotate 0 "Title" out_image_with_title.png

To run for all images in a directory GNU Parallel can be used:

ls *kea | parallel \
python export_quicklook_tuiview.py --inimage {} \
       --invector sensor_locations.shp \
       --outimage {.}_ql.png

TuiView

TuiView is an open source viewer for remote sensing data, named after the New Zealand bird (Tui). It’s primarily for viewing raster data but it will also display vectors. I’ve been using it as my main remote sensing viewer for over a year and I’d highly recommend giving it a try.

It works on windows, OS X and Linux, if you’re on windows, you can grab an installer from here and you should be good to go (don’t worry about there only being 32 bit installers it’s pretty light on memory).

If you’re on Linux / OS X you binaries are available through conda from binstar.org/osgeo or you can install from source.

To install from source you need to have GDAL, numpy and PyQt installed. I’d recommend installing GDAL from source and making sure you build with python bindings (–with-python). For Linux you can install numpy and PyQt from the package manager. Under OS X, assuming you already have numpy, you need to install Qt4, then install sip and PyQt4. If you want to view vectors you also need turbogdal.

After the prerequisites have been installed you can install tuiview using:

hg clone https://bitbucket.org/chchrsc/tuiview tuiview
cd tuiview
sudo python setup.py install

You should then be able to bring up TuiView by typing tuiview in a terminal window. You can load in an image by clicking on the first icon on the left.

TuiView can handle very large (> 50 GB) datasets, very quickly if you build pyramids for the dataset and pre-calculate statistics. You can use gdalcalcstats from gdalutils, or the popimgstats option in RSGISLib

gdalcalcstats image.kea -ignore 0
# or
rsgiscmd --algor imageutils --option popimgstats \
   --image image.kea --ignore 0 \
   --pyramids yes

Note: The rsgiscmd interface is shown here, you could also use XML or the Python bindings.
It’s a good habit to create pyramids and calculate statistics for outputs at the end of your scripts as other programs (such as ArcGIS and ERDAS Imagine) can also use them.

TuiView has lots of options, including the ability to load multiple images in the same or linked windows, and excellent support for raster attribute tables. A comprehensive guide to the functionality is available on the TuiView wiki.