December Sprint Report

Twisted sprint? Twisted sprint! Here's the final Twisted sprint report of 2011, from our December 10th event at Smarterer in Boston.

David Sturgis:

• was Sprint Host and Food Wrangler
• discussed miscellaneous twisted.web feature

JP Calderone:

• investigated 64 bit Windows 7 IPv6 problems
• finished #5383: Provide a library for simple valued named constants
• finished #5084: Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.listenTCP
• reviewed #3420: twisted.web.client persistent connections
• reviewed #1902: compatibility work-around for commercial SSH 2.0.12 misbehaviours
• reviewed #5400: Change UDP port to have an explicit state machine, and no FileDescriptor dependency
• reviewed #3648: twisted.cred.credentials.UsernameHashedPassword doesn't hash password strings when checkPassword

This was the last sprint for JP as a Bostonian. We will miss you!

Itamar:

• worked on #5427: Improve core documentation index page
• reviewed #5383: Provide a library for simple valued named constants

Alex Levy:

• made headway on several website and documentation improvements

Glyph:

• discussed and then worked on #1956: Make a less sucky producer/consumer API

I (Jessica McKellar):

• reviewed #5427: Improve core documentation index page
• reviewed #5429: Documentation index
• reviewed #5422: pbgtk2.py example is excessively complex

Thank you David for organizing this, and Smarterer for hosting.

Thank you to everyone who closed out 2011 with contributions to Twisted!

November Sprint Report

November saw another successful Sprint at the Smarterer.com offices, thanks to a smaller but extremely busy group of developers:

Reviewmeister(ess) Jessica McKellar secured second place in the Twisted High Score list with an impressive number of reviews, including:

• #3037: "loseWriteConnection breaks loseConnection" (now resolved)
• #4262: "Multicast documentation is misleading" (now resolved)
• #538: "Website-template.tpl is out of date"

She also resolved #1247: ("kqreactor timeout error (traceback)"), and opened #5385, #5386, and #5387, while resolving #5060: ("Correct python requirement in twisted/topfiles/setup.py").

Not to mention all the tickets reviewed at recent Sprints which have since been resolved! (Though, to mention them anyway, #1247, #2115, #2447, #2498, #2507, and #5370)

Eternal High Score Overlord JP Calderone made a fair showing as well, finishing #5371 ("In the developer guide, forbid new global reactor unit tests"), and splitting #4671 ("sometimes I want to have some constant values that are part of a set") into more managable tickets:

• #5382: "Provide a library for valueless named constants"
• #5383: "Provide a library for simple valued named constants"
• #5384: "Provide a library for bitvector-like valued named constants"

Of which #5382 and #5383 saw immediate progress.

Returning Champion Sprinter Paul Swartz reviewed and otherwise worked on several tickets, including:

• #2997: "Conch and getPeer"
• #5352: "Do not register overlapping old DH key exchange messages"
• #1902: "Compatibility work-around for commercial SSH 2.0.12 misbehaviours"
• #3242: "Use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available"

One-word Twisted celebrity Glyph was also on hand to assist with the reviews of #3242, #5382, and #1918

With considerably more words in his name, Itamar Turner-Trauring started porting a Nevow branch with IRI support (internationalized URLs) to Twisted as per ticket #5388, as well as helping resolve #3037 with Jessica.

Finally, Glyph, Itamar, and Dave Sturgis conferred on #288, and the related cluster of web-resource-model issues it involves; It is hoped that this will eventually result in a new Web Server URL Traversal API, or WSURLTA, not to be confused with the nordic kraken-headed demigod of lore.

Twisted 11.1.0 has been released!

Twisted Matrix Laboratories is proud to announce the release of Twisted 11.1, thanks to the efforts of our release manager Thomas Hervé.

Highlights of the 185 tickets closed include:

• The poll() reactor as default where applicable, instead of select() everywhere.
• A new SSL implementation only relying on OpenSSL for cryptography, (not I/O) making it more robust.
• Several improvements to the fresh HTTP/1.1 client implementation, including proxy and cookie support.
• Thomas' personal favorite: a new howto has been published on test-driven development with Twisted.
• A new abortConnection() method for immediately closing TCP and SSL connections, the oldest ticket closed by this release.

For more information, see the NEWS file. Note that this is the last release supporting Python 2.4 (the support on Windows stopped with 11.0).

Thanks to the supporters who have donated to Twisted via the Software Freedom Conservancy, and to the many contributors for this release! Download it now:

• Twisted-11.1.0.tar.bz2
• Twisted-11.1.0.win32-py2.5.msi
• Twisted-11.1.0.win32-py2.6.msi
• Twisted-11.1.0.win32-py2.7.msi

Sponsored Development, October 2011

I'm happy to report that I've just completed another two weeks of sponsored Twisted development.  As usual, the issue tracker directed most of my work.  I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development.  The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on.  Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software.  Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure.  I:

• cleaned up our BuildBot master configuration and published it
• implemented improved error reporting for our API documentation build automation
• converted the subversion repository for the Twisted website to bzr and made it public
• moved some Windows build infrastructure to a new host with better network connectivity to try to reduce spurious build failures

For more details about a ticket, see <http://tm.tl/NNNN>.

The tickets I reviewed, with tickets now closed in bold:

#78 - abortConnection() method for transports

#2674 - t.i.defer.inlineCallbacks documentation issues

#2675 - Test timeout policy should be decided by runner

#3077 - twistd --help plugin list isn't alphabetized

#3078 - Dynamic ZSH tab completion for any commands using t.p.usage

#3350 - make each peer register as a producer in portforward

#3453 - twisted.web._auth.digest mostly ignores the value of the uri field

#3896 - Passing a unicode object to request.write corrupts the entire response

#3926 - twisted.positioning -- a better positioning framework

#4159 - Add pre-exec hook to spawnProcess

#4519 - Delay parsing of request body until Request.args is accessed

#4751 - t.c.telnet.ITelnetProtocol.unhandledSubnegotiation doesn't reflect reality

#4849 - Add stream compression to twisted.words.jabber

#5024 - MSI buildbot patches version to avoid breaking distutils MSI builder, but does it in the wrong place

#5040 - twisted.web.template is lacking patterns

#5044 - Client documentation should use explicit buildProtocol()s rather than magic "protocol = MyProtocol" on factories

#5085 - Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.connectTCP

#5139 - Replace usage of os.path.walk in favor of twisted.python.filepath.FilePath.walk

#5156 - no ambiguous antecedents (plugins documentation edition)

#5192 - 100-continue support for twisted.web.client.Agent

#5194 - log.msg can fail if someone removes an observer at the wrong time

#5250 - finger15.tac patch

#5252 - Manhole should support CTRL-A and CTRL-E for home/end

#5253 - Add support for input history persistence in twisted.conch.manhole

#5263 - add multipart/form-data support to twisted.web.client.Agent for easy form posts

#5267 - allow twisted.internet.ssl.[Private]Certificate (and friends) to be backed with TLSLite rather than pyOpenSSL

#5271 - trial test method timeout support is fragile

#5275 - t.w.template does not always properly escape comments

#5279 - pass mode to FilePath.createDirectory()

#5282 - ILogObserver, log.msg, log.err, and logging.html are insufficiently stern about log observer thread safety

#5283 - opt_user unused in manhole twistd plugin

#5285 - win32eventreactor misses disconnect events that happen too soon

#5286 - twisted.web.template documentation doesn't cover very basic use-case of rendering more than one of something

#5288 - twisted.web.template.Tag.fillSlots documentation doesn't mention it returns self

#5301 - test_tcp_internals.PlatformAssumptionsTestCase fails when we leak fds

#5312 - Create some release automation for building Sphinx documentation

And tickets I developed:

#581 - CopiedFailure.getTraceback should include the result of CopiedFailure.getErrorMessage even if unsafeTracebacks is not set

#745 - writeSequence not well-tested

#1946 - ErrorHolder is basically untested

#2838 - _dumbwin32proc.Process does not implement IProcessTransport completely

#4603 - names should handle unknown records

#4666 - t.i.task.Cooperator scheduler objects must return an IDelayedCall

#4671 - sometimes I want to have some constant values that are part of a set

#5040 - twisted.web.template is lacking patterns

#5062 - ProtocolWrapper masks the name of the wrapped protocol in the logs

#5075 - IntNStringReceiver copies too much data in dataReceived, causing AMP to be slow

#5140 - Deprecate twisted.scripts.tkunzip

#5158 - Installing on Pypy fails because of CPython-specific extension modules

#5233 - win32eventreactor doesn't notice TCP connection is lost in certain (hopefully rare) cases

#5237 - Add SFTP example

#5273 - Improve the API documentation for MultiService.addService to clarify its use in comparison to Service.setServiceParent

#5278 - When connecting with endpoints, the application factory used is neither started nor stopped, and irrelevant start/stop messages are logged instead

#5285 - win32eventreactor misses disconnect events that happen too soon

#5291 - twisted.trial TestCase should support assertDictEqual from Python 2.7 unittest

#5292 - Listening ports used with wrapping factories could log better start messages

#5293 - Remove mktap and corresponding twistd options (mktap is broken in trunk)

#5299 - udp.Port schedules connectionLost call in the global reactor

#5303 - Default transport implementation doesn't write out bytes immediately

#5304 - Win32Reactor.spawnProcess is redundant

#5308 - Make UDP ports identify themselves when stopping

#5316 - Incorrect API link in twisted.web.proxy.Proxy docstring

October Sprint Report

After a brief rest, the Boston Twisted crew is back on the sprinting treadmill. Dave Sturgis came through for us once again for the venue, working with Smarterer to let us use their office space and eat their snacks last Saturday afternoon. We also took advantage of the Python Software Foundation's sprint funding program to buy all the sprinters dinner.

Eight sprinters turned out to fix bugs, add features, and review changes.

Chris (radix) (triumphantly returned from the inhospitable southwest) worked on adding a history-tracking mode to Deferreds to aid in debugging. This feature will let a developer see everything that has happened to a Deferred - what callbacks it has run, what values have passed through it, and what other Deferreds it has been associated with.

Itamar picked up a ticket relating to HTTP client support for persistent connections. He also finished up the work on adding a new transport method, abortConnection, fixed a logging bug in the new TLS implementation, and did some ticket triage and a review.

Allister (amacleod) heroically tackled the problem of our insufficient Windows testing infrastructure, getting introduced to our build farm and setting up a new Windows 7 virtual machine on it.

Jessica (jesstess) did a ton of follow-up on old tickets (1247 2115 2447 2498 2507 2513 2861), trying to learn if they are still valid and elicit further information from the original reporters. She also managed to close one which was no longer valid (but she also filed a new one for a documentation bug, in case you were worried we might run out of tickets).

Glyph knuckled down on a review for our conversion to Sphinx. We're so close I can practically taste it.

Ying (cyli) took a look at improving the authentication options for some of Twisted's built-in servers. She started with the FTP server and made great headway.

As for myself, I reviewed that abortConnection feature as well as a ticket for HTML5 support in twisted.web.template and the TLS logging bug that Itamar fixed.

Thanks to all the sprinters and to Smarterer and the PSF for their support of the sprint! We're going to do another one of these real soon, watch out for an announcement!

Twisted's BuildBot master configuration

Twisted has a somewhat extensively customized BuildBot master configuration which has evolved over the last ten years. We've kept it private only because it contains some passwords which it would be better not to share, but over the years we have wanted to make it public - because it might serve as a good example to others, because it would let more people hack on it and relieve us of some work, etc. At last this has changed: I spent a couple hours today filtering out the sensitive parts. As a result, you can now take a look at lp:twisted-buildbot-configuration. Add some features! Do some refactoring! Write some unit tests!

Pre-trunk Testing

Twisted uses BuildBot for continuous integration - for testing every revision to ensure it passes the full test suite. BuildBot automatically runs the test suite on each version of Python and each operating system supported by Twisted. As of today, that's 22 different configurations, plus another 16 configurations where Twisted is known not to work completely but where we are workings towards full support.

This works very well for identifying regressions and for finding problems with new functionality on platforms a developer may not have considered. However, because Twisted prefers to keep trunk@HEAD completely working and the test suite fully passing on all platforms, if BuildBot identifies a problem with a trunk revision, we revert it completely. Then the problems with it can be addressed and another attempt made. This also works, but it has a couple shortcomings. Someone actually has to do the revert, which is a little bit of work. Also, someone might check out trunk@HEAD before the change is reverted, giving them a non-working version of Twisted. It's nice to be able to avoid both of these things.

The solution generally applied is to test the change before applying it to trunk. BuildBot can do this too, in a couple different ways. The way we use for Twisted development is to have BuildBot run the test suite on all those supported configurations on a branch of Twisted, rather than on trunk. Since the change is checked in to a branch, instead of to trunk, it avoids the needed to revert anything and it avoids offering broken code to people.

This approach involves a tiny custom tool to convince BuildBot to run the necessary builds. Normally BuildBot just watches for trunk commits and starts builds automatically. Since in this case we want the trunk commit to happen after we see that all the builds will succeed, we use force-builds.py to have BuildBot perform the builds before it sees the trunk commit.

For a long time, force-builds.py needed a hard-coded list of supported platforms on which to trigger builds. Recently I fixed this by having it retrieve that list from the BuildBot master, which already knows which platforms are supported. Publishing this information is done with a minor hack to master.cfg:

    supportedList = list(
        builder['name']
        for builder
        in builders
        if builder['category'] == 'supported')
    path = 'public_html/supported-builders.txt'
    supportedListFile = file(path, 'w')
    supportedListFile.write('\n'.join(supportedList) + '\n')
    supportedListFile.close()

The resulting file is publicly available from the master's web server, so force-builds.py can download it on-demand and then force a build on each supported builder. When the developer sees all of these builds have succeeded, he or she can be more confident that there aren't going to be any surprises when the change is actually applied to trunk.