Pre-trunk Testing

Twisted uses BuildBot for continuous integration - for testing every revision to ensure it passes the full test suite. BuildBot automatically runs the test suite on each version of Python and each operating system supported by Twisted. As of today, that's 22 different configurations, plus another 16 configurations where Twisted is known not to work completely but where we are workings towards full support.

This works very well for identifying regressions and for finding problems with new functionality on platforms a developer may not have considered. However, because Twisted prefers to keep trunk@HEAD completely working and the test suite fully passing on all platforms, if BuildBot identifies a problem with a trunk revision, we revert it completely. Then the problems with it can be addressed and another attempt made. This also works, but it has a couple shortcomings. Someone actually has to do the revert, which is a little bit of work. Also, someone might check out trunk@HEAD before the change is reverted, giving them a non-working version of Twisted. It's nice to be able to avoid both of these things.

The solution generally applied is to test the change before applying it to trunk. BuildBot can do this too, in a couple different ways. The way we use for Twisted development is to have BuildBot run the test suite on all those supported configurations on a branch of Twisted, rather than on trunk. Since the change is checked in to a branch, instead of to trunk, it avoids the needed to revert anything and it avoids offering broken code to people.

This approach involves a tiny custom tool to convince BuildBot to run the necessary builds. Normally BuildBot just watches for trunk commits and starts builds automatically. Since in this case we want the trunk commit to happen after we see that all the builds will succeed, we use force-builds.py to have BuildBot perform the builds before it sees the trunk commit.

For a long time, force-builds.py needed a hard-coded list of supported platforms on which to trigger builds. Recently I fixed this by having it retrieve that list from the BuildBot master, which already knows which platforms are supported. Publishing this information is done with a minor hack to master.cfg:

    supportedList = list(
        builder['name']
        for builder
        in builders
        if builder['category'] == 'supported')
    path = 'public_html/supported-builders.txt'
    supportedListFile = file(path, 'w')
    supportedListFile.write('\n'.join(supportedList) + '\n')
    supportedListFile.close()

The resulting file is publicly available from the master's web server, so force-builds.py can download it on-demand and then force a build on each supported builder. When the developer sees all of these builds have succeeded, he or she can be more confident that there aren't going to be any surprises when the change is actually applied to trunk.