Writing unit tests for QGIS Python plugins

Testing your code is a good idea. Testing can help you find bugs early and spot regressions where you didn’t expect them. The tests themselves can form a kind of “living” documentation, and can even drive the design of your project. A unit test is a test written for a single module of code; it might check to see that a drop down menu has been populated correctly, or that a layer was added to the map canvas once an algorithm has finished processing. By testing small parts of code in isolation you can quickly identify which components in a much larger system are failing when things start to go wrong. The Python Guide provides some useful guidance for writing unit tests.

Update 2017-03-26: Capabilities for testing plugins in QGIS have improved greatly since this post was written. Instead, see this answer on GIS.SE and this post by Boundless.

This post describes some of the specifics you need to know for writing unit tests for QGIS Python plugins. It assumes an existing understanding of writing plugins for QGIS using Python.

Writing tests using Python’s unittest module

The Python standard library includes a module for writing unit tests called unittest. A test suite written for using unittest will look something like the code below.

import unittest

def power_of_two(n):
    return n**2

class TestPowerFunction(unittest.TestCase):
    def test_power(self):
        self.assertEqual(power_of_two(2), 4)
    def test_negative(self):
        self.assertEqual(power_of_two(-3), 9)
    def test_float(self):
        self.assertAlmostEqual(power_of_two(1.3), 1.69, places=2)

if __name__ == '__main__':
    unittest.main()

As this post only focuses on the specifics of writing unit tests for QGIS plugins, the reader is directed to the Python manual page for unittest for more information.

Using PyQGIS with unittest

To use the unittest module for testing a QGIS plugin we need to load the qgis modules in a Python script outside of the main QGIS application. The PyQGIS cookbook describes how to do this. The instructions are reiterated here as this step is one which new users frequently struggle with. In this section and later I have assumed that on Linux QGIS is installed to /usr/local, and on Windows QGIS is installed using OSGeo4W to C:\OSGeo4W.

To import the qgis modules they need to be in your PYTHONPATH environment variable.

Configuring PYTHONPATH on Linux:

export PYTHONPATH="${PYTHONPATH}:/usr/local/share/qgis/python"

Configuring PYTHONPATH on Windows:

set PYTHONPATH="%PYTHONPATH%;C:\OSGeo4W\apps\qgis\python"

The qgis module depends on the QGIS shared libraries (libqgis_core, libqgis_gui, etc.).

On Linux these need to be in your LD_LIBRARY_PATH:

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"

On Windows these need to be in your PATH:

set PATH="%PATH;C:\OSGeo4W\bin" # OSGeo4W binaries
set PATH="%PATH;C:\OSGeo4W\apps\qgis\bin" # QGIS binaries

The qgis modules should be loaded before PyQt4 modules, so that correct version of the SIP API is loaded. In the QGIS Python console you don’t have to worry about this, as the qgis modules have already been loaded for you, but when you’re running a standalone application the order matters.

Although it’s possible to set environment variables using os.environ and the Python module search path using sys.path, it’s better to leave this up to the user as it makes the code more portable.

Once the modules have been imported you need to tell QGIS where it should look for it’s resources (projections, data providers, etc.) using QgsApplication.setPrefixPath(), then initalise the data providers using QgsApplication.initQgis(). After doing this it’s useful to check if any data providers are available; an empty list usually indicates an issue with the prefix specified. When you’re finished using the QGIS libraries you should call QgsApplication.exitQgis() to clean up.

from qgis.core import *
from qgis.gui import *

from PyQt4 import QtCore, QtGui, QtTest

import unittest

QgsApplication.setPrefixPath("/usr/local", True)

QgsApplication.initQgis()

if len(QgsProviderRegistry.instance().providerList()) == 0:
    raise RuntimeError('No data providers available.')

QgsApplication.exitQgis()

If you’re running multiple tests using nose (discussed later) you should only call exitQgis() once, at the end of the tests. This can be done using the atexit module:

import atexit
atexit.register(QgsApplication.exitQgis)

The debug messages that QGIS sometimes prints to the console can be disabled by setting the QGIS_DEBUG environment variable.

On Linux:

export QGIS_DEBUG="-1"

On Windows:

set QGIS_DEBUG="-1"

Importing plugins

To import QGIS plugins the plugin directories need to be in your PYTHONPATH.

Configuring PYTHONPATH on Linux:

# default QGIS plugins
export PYTHONPATH="${PYTHONPATH}:/usr/local/share/qgis/python/plugins"
# user installed plugins
export PYTHONPATH="${PYTHONPATH}:${HOME}/.qgis2/python/plugins"

Configuring PYTHONPATH on Windows:

# default QGIS plugins
set PYTHONPATH="%PYTHONPATH%;C:\OSGeo4W\apps\qgis\python\plugins"
# user installed plugins
set PYTHONPATH="%PYTHONPATH%;%USERPROFILE%\.qgis2\python\plugins"

QGIS plugins depend on the qgis.gui.QgsInterface (often simply iface) to interact with the QGIS application. This object is a required argument of the classFactory() function which initalises plugins, but it isn’t available in custom applications. Instead we create a ‘dummy’ iface object which will accept any method call, which simply returns itself. We make an exception for the layers method, which should return a list of QgsMapLayers.

class DummyInterface(object):
    def __getattr__(self, *args, **kwargs):
        def dummy(*args, **kwargs):
            return self
        return dummy
    def __iter__(self):
        return self
    def next(self):
        raise StopIteration
    def layers(self):
        # simulate iface.legendInterface().layers()
        return QgsMapLayerRegistry.instance().mapLayers().values()
iface = DummyInterface()

The default organization name and application name need to be configured so that QSettings() behaves as if it was running from within QGIS.

QtCore.QCoreApplication.setOrganizationName('QGIS')
QtCore.QCoreApplication.setApplicationName('QGIS2')

We are now ready to import the plugin.

import MyPlugin
self.plugin = MyPlugin.classFactory(iface)
self.ui = self.plugin.dlg.ui # useful for shorthand later

Not all plugins will work in custom applications. Notably, the processing plugin will import and allow you to use the QGIS algorithms, but not the algorithms of any other providers (e.g. GRASS GIS or SAGA GIS) (see feature request #8955).

Testing geoprocessing algorithms

This section describes a number of approaches used in testing the output from (or input to) a geoprocessing algorithm.

When loading layers for use in tests it is useful to check if the layer isValid() before using it; this can highlight errors (such as an incorrect filename) early on. Failure to catch a broken layer can result in tests failing later on when the layer is accessed, which can produce confusing tracebacks and (on Windows at least) seems to crash the interpreter.

layer = QgsVectorLayer(os.path.join(test_data_directory, 'input.shp'), 'input', 'ogr')
self.assertTrue(layer.isValid(), 'Failed to load "{}".'.format(layer.source()))
QgsMapLayerRegistry().instance().addMapLayer(layer)

To count the number of features in a layer, use layer.dataProvider().featureCount() in preference to layer.featureCount() in case the changes have not been propagated.

self.assertEqual(layer.dataProvider().featureCount(), 42)

Sometimes counting the number of features is not enough. Here we calculate the total area of all the features in a layer.

total_area = 0.0
for feature in layer.getFeatures():
    total_area += feature.geometry().area()
self.assertAlmostEqual(total_area, 42.000, places=3)

We can compare the exact geometry of a feature using WKT.

features = layer.getFeatures()
feature = features.next()
self.assertEqual(feature.geometry().exportToWkt(), u'POINT(194023 229400)')

For large data sets it is more practical to compare a checksum of a layer to a known value, rather than checking each individual feature. We could run a checksum against the data as it is stored on disk, but checking all of the parts of a shapefile (.shp, .dbf, .shx, etc.) is a hassle and would not work for layers using the memory data provider. Instead we can use a short function which will calculate the checksum of a layer’s projection, field names, and the geometry and attributes of each feature. This method has the added bonus that if at some point the test data is migrated from one format to another (e.g. shapefile to SQLite), the checksums will still match (provided that the underlying data is otherwise unmodified).

import hashlib

def checksum(layer):
    m = hashlib.md5()
    m.update(layer.crs().toProj4())
    field_names = [f.name() for f in layer.dataProvider().fields().toList()].__repr__()
    m.update(field_names)
    features = layer.getFeatures()
    for feature in features:
        m.update(feature.geometry().exportToWkt())
        m.update(feature.attributes().__repr__())
    return m.hexdigest()

The checksum of a layer can then be tested against a known value.

layer = QgsVectorLayer('output.shp', 'output', 'ogr')
self.assertEqual(checksum(layer), '61617f919678503c996d38fa31aa3d12')

Working with signals

In addition to checking the output layers of an algorithm, it’s also useful to check it emits the correct signals. For instance, that an error signal is emitted when an invalid input is given. To do this we define a simple helper class. The idea for this came from GUI Programming with Python: QT Edition - Testing signals and slots.

class SignalBox(QtCore.QObject):
    def __init__(self, *args, **kwargs):
        QtCore.QObject.__init__(self, *args, **kwargs)
        self.received = {}
    def __getattr__(self, key):
        return partial(self.__slot, key)
    def __slot(self, attr=None, *args):
        sig = tuple([self.sender()] + list(args))
        if attr not in self.received:
            self.received[attr] = []
        self.received[attr].append(sig)

To use SignalBox we need to create an instance and connect some signals to it. The code below assumes that alg is a QObject with error and finished signals as suggested in Multithreading in QGIS Python Plugins.

signalbox = SignalBox()

alg.error.connect(signalbox.error)
alg.finished.connect(signalbox.finished)

if 'error' in signalbox.received:
    sender, err, message = signalbox.received['error'][0]
    raise(Exception(message))

if 'finished' not in signalbox.received:
    raise(KeyError('Algorithm never fired "finished" signal'))

Testing the Graphical User Interface (GUI)

Testing the GUI involves two steps: configuring the inputs (e.g. selecting layers from drop down menus, writing text in boxes, etc.) and running the algorithm itself.

The example below demonstrates selecting a layer from a combo box by it’s index, then testing if the correct layer has been selected and that the ‘OK’ button has been enabled. A post by John McGehee provides several examples of testing a GUI using PyQt4, QTest and unittest.

comboBox = self.ui.comboBox_InputLayer
okButton = self.ui.buttonBox.buttonRole(self.ui.buttonBox.Ok)
index = comboBox.findText('Woodland')
comboBox.setCurrentIndex(index)
self.assertEqual(comboBox.currentText(), 'Woodland', 'Unexpected layer selected')
self.assertTrue(okButton.isEnabled(), "OK button isn't enabled.")

Once the inputs have been configured we can use QtTest.QTest.mouseClick to press the ‘OK’ button that will start the geoprocessing algorithm in the background. Once the button has been pressed, we use QtTest.QTest.qWait to wait a reasonable amount of time for a new layer to be added to the map canvas. If a layer is not added in this time a timeout error is raised.

If the GUI doesn’t provide easy access to any signals fired from the geoprocessing algorithm we can hook into the QGIS message log and listen for any critical errors with the plugin’s tag. Obviously this requires the algorithm to log useful error messages with QgsMessageLog.logMessage if something goes wrong.

# get a list of the existing layers
existing_layers = QgsMapLayerRegistry().instance().mapLayers().values()

# listen to the QGIS message log
message_log = {}
def log(message, tag, level):
    message_log.setdefault(tag, [])
    message_log[tag].append((message, level,))
QgsMessageLog.instance().messageReceived.connect(log)

# locate the ok button, then click it
okButton = self.ui.buttonBox.buttonRole(self.ui.buttonBox.Ok)
QtTest.QTest.mouseClick(okButton, QtCore.Qt.LeftButton)

# wait for a new layer to be added
timeout = 20 # seconds
count = 0
while count < timeout:
    QtTest.QTest.qWait(1000) # wait 1000ms
    # parse message log for critical errors
    if 'MyPluginTag' in message_log:
        while message_log['MyPluginTag']:
            message, level = message_log['MyPluginTag'].pop()
            self.assertNotEqual(level, QgsMessageLog.CRITICAL, \
                'Critical error in message log:\n{}'.format(message))
    # check if any layers have been added
    if len(QgsMapLayerRegistry().instance().mapLayers()) > len(layers):
        break
    count += 1

# test if process timed out
self.assertTrue(count < timeout, 'Timed out waiting for layer to be added.')

# everything went OK, get the new layer(s)
current_layers = QgsMapLayerRegistry().instance().mapLayers().values()
new_layers = list(set(current_layers).difference(set(existing_layers)))

Taking screenshots

Although screenshots aren’t useful for testing to see if something is displaying correctly, due to differences in appearance between systems, they can be useful as a debugging tool when running ‘headless’ tests. When a test is failing a screenshot can be used to check that all of the inputs to the UI have been entered in the expected way.

The code below captures a screenshot of a QDialog, then saves it as a PNG file in the current directory. The QDialog doesn’t need to be visible for this to work.

screenshot = QtGui.QPixmap.grabWidget(self.plugin.dlg)
f = QtCore.QFile('screenshot.png')
screenshot.save(f, 'PNG')

The PyQGIS Cookbook describes how to export the map canvas as an image.

Nicer testing with nose

nose extents the unittest module to make testing easier. It can collect all the tests in a directory tree and run them one after another.

$ nosetest -v ../tests
Testing concentric ring buffer algorithm ... ok
Testing GUI defaults ... ok
Testing output is added to canvas at end of processing ... ok

I find it useful to keep all of the tests together in a single folder. Don’t forget to add the folder that your plugin is in to PYTHONPATH.

.
├── myplugin
│   ├── __init__.py
│   ├── Makefile
│   ├── myplugin.py
│   └── etc.
└── tests
    ├── point_tests.py
    ├── polygon_tests.py
    └── gui_tests.py

If you’re using a Makefile to build and deploy your plugin it’s trivial to add a test target, so that you only need to type make test to run every test in your project. The example below assumes your tests are saved in a ’tests’ directory in the directory above the Makefile.

# test everything
make test:
    nosetests -v ../tests

# test GUI only
make testgui:
    nosetests -v ../tests/*gui*.py