Roadtest Proposed As A New Driver Testing Framework For Linux

Axis Communications on Friday published "Roadtest" as their initial patch-set for this new Linux driver testing framework.

Roadtest is a device driver testing framework whereby drivers are tested under User-Mode Linux against mocked/modelled hardware. Both the tests and hardware models are written in Python. With Roadtest's hardware modelling it hopes to allow for at least some basic test coverage in the lack of hardware being present within continuous integration (CI) setups. For now at least though Roadtest only works with rather simple drivers -- such as those within the industrial I/O (IIO), regulator, and real-time clock (RTC) subsystems.

Here's some more from Axis on this Roadtest Linux driver testing framework:

This patchset proposes roadtest, a device-driver testing framework. Drivers are tested under User Mode Linux (UML) and interact with mocked/modelled hardware. The tests and hardware models are written in Python, the former using Python's built-in unittest framework.

Drivers are tested via their userspace interfaces. The hardware models allow tests to inject values into registers and assert that drivers control the hardware in the right way and react as expected to stimuli.

Roadtest is meant to be used for relatively simple drivers, such as the ones part of the IIO, regulator and RTC subsystems.
...
There are a large amount of these kind of drivers in the kernel. Most of the hardware is not available in current CI systems so most drivers can only, at best, be build-tested there. Even basic soundness such as a driver successfully probing and binding to the devices it tries to be support cannot be tested. Drivers cannot be easily regression-tested to ensure that bugs fixed once do not get reintroduced.

Many drivers support multiple related hardware variants, and far from all patch submitters have access to all the variants which the driver that they are patching supports, so there is no way for them to easily verify that they haven't broken something basic on a variant which they do not own.

Furthermore, hardware can be used in many different configurations with drivers supporting many different devicetree properties, so even just having access to all the variants would be insufficient.

On top of that, some of the chips measure environmental conditions such as temperature, so testing extreme cases may not be simple even if one has access to the hardware.

All this makes development, modification, maintenance, and reviewing of these drivers harder than it necessarily needs to be. Roadtest hopes to make some of these things slightly easier by providing a framework to create hardware models/mocks and to write testcases which exercise drivers using these models.

Right now Roadtest was sent out as an initial "request for comments" patch series on the kernel mailing list. We'll see what all comes of Roadtest for facilitating more Linux driver testing and whether it can end up supporting more advanced hardware/subsystems where regressions and other problems tend to be much more pressing.