snorfalorpagus dot net

Multithreading in QGIS Python plugins

Posted on

The Python bindings in QGIS allow plugins to be written in Python. Writing a plugin for QGIS involves two things: writing the geoprocessing algorithm itself, and writing a graphical user interface (GUI) that the user can interact with without writing any code.

Using a GIS often requires patience; geoprocessing tasks can take a long time to complete when working with large datasets. If you run a time consuming task and the UI in the same thread, the UI will become unresponsive; it will appear to the user as if the application has crashed, even if it is still working on the task. The solution is to do all the heavy lifting of the algorithm in another thread. The UI is responsible for passing the input layers and any parameters to the ‘worker’ thread, and handling the output when it’s ready. In the time between these events it can continue doing what it does (updating progress bars, allowing the user to press buttons, etc.).

This post assumes some existing experience with the QGIS Python bindings, but no experience with threading in Python. If you’ve not heard of threads before the Wikipedia page on threads isn’t a bad place to start.

In this example we will create a worker process that calculates the total area of all of the features in a layer. I’ve added a call to time.sleep() in order to simulate a more time consuming task (e.g. buffering or intersection) and keep the example reasonably generic.

# import some modules used in the example
from qgis.core import *
from PyQt4 import QtCore, QtGui
import traceback
import time

class Worker(QtCore.QObject):
    '''Example worker for calculating the total area of all features in a layer'''
    def __init__(self, layer):
        QtCore.QObject.__init__(self)
        if isinstance(layer, QgsVectorLayer) is False:
            raise TypeError('Worker expected a QgsVectorLayer, got a {} instead'.format(type(layer)))
        self.layer = layer
        self.killed = False
    def run(self):
        ret = None
        try:
            # calculate the total area of all of the features in a layer
            total_area = 0.0
            features = self.layer.getFeatures()
            feature_count = self.layer.featureCount()
            progress_count = 0
            step = feature_count // 1000
            for feature in features:
                if self.killed is True:
                    # kill request received, exit loop early
                    break
                geom = feature.geometry()
                total_area += geom.area()
                time.sleep(0.1) # simulate a more time consuming task
                # increment progress
                progress_count += 1
                if step == 0 or progress_count % step == 0:
                    self.progress.emit(progress_count / float(feature_count))
            if self.killed is False:
                self.progress.emit(100)
                ret = (self.layer, total_area,)
        except Exception, e:
            # forward the exception upstream
            self.error.emit(e, traceback.format_exc())
        self.finished.emit(ret)
    def kill(self):
        self.killed = True
    finished = QtCore.pyqtSignal(object)
    error = QtCore.pyqtSignal(Exception, basestring)
    progress = QtCore.pyqtSignal(float)

Everything within the run() method is wrapped in a try-except block, except for the emission of the finished signal. If anything goes wrong in during processing the exception will be caught and emitted by the error signal along with a traceback string. This ensures that the finished signal is emitted, which is important as it will be responsible for cleaning up the thread, as well as executing any post-processing (e.g. reporting results and adding new layers). Failing the clean up after a crashed thread will result in a memory leak.

The worker implements a kill switch; if the kill() method is called, a flag is set which causes the main processing loop to exit early. This approach is preferable to calling the terminate() method of the QThread class, which doesn’t give the thread a chance to clean up and can stop the thread while modifying data.

The progress signal is only emitted in increments of 0.1%. This avoids sending too many signals too quickly to a progress bar, which can cause the UI to become less responsive while it tries to keep up. Increments smaller than this aren’t usuaully visible anyway, due to the size of progress bars.

To start the run method we need to create an instance of Worker, move it to a thread and then start the thread (which in turn calls run()). To give the user some feedback we’ll also show a message, progress bar and a button to cancel the process in the QGIS message bar. This function and the two that follow should be methods on your QtGui.QDialog instance.

def startWorker(self, layer):
    # create a new worker instance
    worker = Worker(layer)

    # configure the QgsMessageBar
    messageBar = self.iface.messageBar().createMessage('Doing something time consuming...', )
    progressBar = QtGui.QProgressBar()
    progressBar.setAlignment(QtCore.Qt.AlignLeft|QtCore.Qt.AlignVCenter)
    cancelButton = QtGui.QPushButton()
    cancelButton.setText('Cancel')
    cancelButton.clicked.connect(worker.kill)
    messageBar.layout().addWidget(progressBar)
    messageBar.layout().addWidget(cancelButton)
    self.iface.messageBar().pushWidget(messageBar, self.iface.messageBar().INFO)
    self.messageBar = messageBar

    # start the worker in a new thread
    thread = QtCore.QThread(self)
    worker.moveToThread(thread)
    worker.finished.connect(workerFinished)
    worker.error.connect(workerError)
    worker.progress.connect(progressBar.setValue)
    thread.started.connect(worker.run)
    thread.start()
    self.thread = thread
    self.worker = worker

Once the worker has been moved to another thread, all communication between the UI and the worker should be done using signals. The worker sends signals back to the UI reporting its progress and any exceptions that occur. The UI can send a signal to the kill slot to request that the process finishes early. Any input data required by the worker should be passed when it is initalised before it is moved to another thread.

When the worker finishes (and emits it’s finished signal) we need to clean up: flag the worker instance for deletion, quit the thread (and wait for it to quit), then flag the thread for deletion. Once this has been dealt with we can remove the progress bar, report success or failure to the user, and add any new layers to the canvas.

def workerFinished(self, ret):
    # clean up the worker and thread
    self.worker.deleteLater()
    self.thread.quit()
    self.thread.wait()
    self.thread.deleteLater()
    # remove widget from message bar
    self.iface.messageBar().popWidget(self.messageBar)
    if ret is not None:
        # report the result
        layer, total_area = ret
        self.iface.messageBar().pushMessage('The total area of {name} is {area}.'.format(name=layer.name(), area=total_area))
    else:
        # notify the user that something went wrong
        self.iface.messageBar().pushMessage('Something went wrong! See the message log for more information.', level=QgsMessageBar.CRITICAL, duration=3)

If you need to run more than one thread at a time make sure you keep track of the worker and thread instances so that the memory they use can be freed once the task finishes.

If something goes wrong during the execution of the run() method our Worker instance emits the exception and traceback via the error signal. If we log this error message to the QGIS message log we can read it and work out what happened.

def workerError(self, e, exception_string):
    QgsMessageLog.logMessage('Worker thread raised an exception:\n'.format(exception_string), level=QgsMessageLog.CRITICAL)

A lot of the code above is boilerplate. If you’re defining more than one algorithm to be run you might consider subclassing from a more generic Worker class, so that you can spend more time thinking about the problem at hand rather than about threads.

References and futher reading