November Sprint Report

November saw another successful Sprint at the Smarterer.com offices, thanks to a smaller but extremely busy group of developers:

Reviewmeister(ess) Jessica McKellar secured second place in the Twisted High Score list with an impressive number of reviews, including:

• #3037: "loseWriteConnection breaks loseConnection" (now resolved)
• #4262: "Multicast documentation is misleading" (now resolved)
• #538: "Website-template.tpl is out of date"

She also resolved #1247: ("kqreactor timeout error (traceback)"), and opened #5385, #5386, and #5387, while resolving #5060: ("Correct python requirement in twisted/topfiles/setup.py").

Not to mention all the tickets reviewed at recent Sprints which have since been resolved! (Though, to mention them anyway, #1247, #2115, #2447, #2498, #2507, and #5370)

Eternal High Score Overlord JP Calderone made a fair showing as well, finishing #5371 ("In the developer guide, forbid new global reactor unit tests"), and splitting #4671 ("sometimes I want to have some constant values that are part of a set") into more managable tickets:

• #5382: "Provide a library for valueless named constants"
• #5383: "Provide a library for simple valued named constants"
• #5384: "Provide a library for bitvector-like valued named constants"

Of which #5382 and #5383 saw immediate progress.

Returning Champion Sprinter Paul Swartz reviewed and otherwise worked on several tickets, including:

• #2997: "Conch and getPeer"
• #5352: "Do not register overlapping old DH key exchange messages"
• #1902: "Compatibility work-around for commercial SSH 2.0.12 misbehaviours"
• #3242: "Use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available"

One-word Twisted celebrity Glyph was also on hand to assist with the reviews of #3242, #5382, and #1918

With considerably more words in his name, Itamar Turner-Trauring started porting a Nevow branch with IRI support (internationalized URLs) to Twisted as per ticket #5388, as well as helping resolve #3037 with Jessica.

Finally, Glyph, Itamar, and Dave Sturgis conferred on #288, and the related cluster of web-resource-model issues it involves; It is hoped that this will eventually result in a new Web Server URL Traversal API, or WSURLTA, not to be confused with the nordic kraken-headed demigod of lore.

Twisted 11.1.0 has been released!

Twisted Matrix Laboratories is proud to announce the release of Twisted 11.1, thanks to the efforts of our release manager Thomas Hervé.

Highlights of the 185 tickets closed include:

• The poll() reactor as default where applicable, instead of select() everywhere.
• A new SSL implementation only relying on OpenSSL for cryptography, (not I/O) making it more robust.
• Several improvements to the fresh HTTP/1.1 client implementation, including proxy and cookie support.
• Thomas' personal favorite: a new howto has been published on test-driven development with Twisted.
• A new abortConnection() method for immediately closing TCP and SSL connections, the oldest ticket closed by this release.

For more information, see the NEWS file. Note that this is the last release supporting Python 2.4 (the support on Windows stopped with 11.0).

Thanks to the supporters who have donated to Twisted via the Software Freedom Conservancy, and to the many contributors for this release! Download it now:

• Twisted-11.1.0.tar.bz2
• Twisted-11.1.0.win32-py2.5.msi
• Twisted-11.1.0.win32-py2.6.msi
• Twisted-11.1.0.win32-py2.7.msi

Sponsored Development, October 2011

I'm happy to report that I've just completed another two weeks of sponsored Twisted development.  As usual, the issue tracker directed most of my work.  I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development.  The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on.  Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software.  Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure.  I:

• cleaned up our BuildBot master configuration and published it
• implemented improved error reporting for our API documentation build automation
• converted the subversion repository for the Twisted website to bzr and made it public
• moved some Windows build infrastructure to a new host with better network connectivity to try to reduce spurious build failures

For more details about a ticket, see <http://tm.tl/NNNN>.

The tickets I reviewed, with tickets now closed in bold:

#78 - abortConnection() method for transports

#2674 - t.i.defer.inlineCallbacks documentation issues

#2675 - Test timeout policy should be decided by runner

#3077 - twistd --help plugin list isn't alphabetized

#3078 - Dynamic ZSH tab completion for any commands using t.p.usage

#3350 - make each peer register as a producer in portforward

#3453 - twisted.web._auth.digest mostly ignores the value of the uri field

#3896 - Passing a unicode object to request.write corrupts the entire response

#3926 - twisted.positioning -- a better positioning framework

#4159 - Add pre-exec hook to spawnProcess

#4519 - Delay parsing of request body until Request.args is accessed

#4751 - t.c.telnet.ITelnetProtocol.unhandledSubnegotiation doesn't reflect reality

#4849 - Add stream compression to twisted.words.jabber

#5024 - MSI buildbot patches version to avoid breaking distutils MSI builder, but does it in the wrong place

#5040 - twisted.web.template is lacking patterns

#5044 - Client documentation should use explicit buildProtocol()s rather than magic "protocol = MyProtocol" on factories

#5085 - Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.connectTCP

#5139 - Replace usage of os.path.walk in favor of twisted.python.filepath.FilePath.walk

#5156 - no ambiguous antecedents (plugins documentation edition)

#5192 - 100-continue support for twisted.web.client.Agent

#5194 - log.msg can fail if someone removes an observer at the wrong time

#5250 - finger15.tac patch

#5252 - Manhole should support CTRL-A and CTRL-E for home/end

#5253 - Add support for input history persistence in twisted.conch.manhole

#5263 - add multipart/form-data support to twisted.web.client.Agent for easy form posts

#5267 - allow twisted.internet.ssl.[Private]Certificate (and friends) to be backed with TLSLite rather than pyOpenSSL

#5271 - trial test method timeout support is fragile

#5275 - t.w.template does not always properly escape comments

#5279 - pass mode to FilePath.createDirectory()

#5282 - ILogObserver, log.msg, log.err, and logging.html are insufficiently stern about log observer thread safety

#5283 - opt_user unused in manhole twistd plugin

#5285 - win32eventreactor misses disconnect events that happen too soon

#5286 - twisted.web.template documentation doesn't cover very basic use-case of rendering more than one of something

#5288 - twisted.web.template.Tag.fillSlots documentation doesn't mention it returns self

#5301 - test_tcp_internals.PlatformAssumptionsTestCase fails when we leak fds

#5312 - Create some release automation for building Sphinx documentation

And tickets I developed:

#581 - CopiedFailure.getTraceback should include the result of CopiedFailure.getErrorMessage even if unsafeTracebacks is not set

#745 - writeSequence not well-tested

#1946 - ErrorHolder is basically untested

#2838 - _dumbwin32proc.Process does not implement IProcessTransport completely

#4603 - names should handle unknown records

#4666 - t.i.task.Cooperator scheduler objects must return an IDelayedCall

#4671 - sometimes I want to have some constant values that are part of a set

#5040 - twisted.web.template is lacking patterns

#5062 - ProtocolWrapper masks the name of the wrapped protocol in the logs

#5075 - IntNStringReceiver copies too much data in dataReceived, causing AMP to be slow

#5140 - Deprecate twisted.scripts.tkunzip

#5158 - Installing on Pypy fails because of CPython-specific extension modules

#5233 - win32eventreactor doesn't notice TCP connection is lost in certain (hopefully rare) cases

#5237 - Add SFTP example

#5273 - Improve the API documentation for MultiService.addService to clarify its use in comparison to Service.setServiceParent

#5278 - When connecting with endpoints, the application factory used is neither started nor stopped, and irrelevant start/stop messages are logged instead

#5285 - win32eventreactor misses disconnect events that happen too soon

#5291 - twisted.trial TestCase should support assertDictEqual from Python 2.7 unittest

#5292 - Listening ports used with wrapping factories could log better start messages

#5293 - Remove mktap and corresponding twistd options (mktap is broken in trunk)

#5299 - udp.Port schedules connectionLost call in the global reactor

#5303 - Default transport implementation doesn't write out bytes immediately

#5304 - Win32Reactor.spawnProcess is redundant

#5308 - Make UDP ports identify themselves when stopping

#5316 - Incorrect API link in twisted.web.proxy.Proxy docstring

October Sprint Report

After a brief rest, the Boston Twisted crew is back on the sprinting treadmill. Dave Sturgis came through for us once again for the venue, working with Smarterer to let us use their office space and eat their snacks last Saturday afternoon. We also took advantage of the Python Software Foundation's sprint funding program to buy all the sprinters dinner.

Eight sprinters turned out to fix bugs, add features, and review changes.

Chris (radix) (triumphantly returned from the inhospitable southwest) worked on adding a history-tracking mode to Deferreds to aid in debugging. This feature will let a developer see everything that has happened to a Deferred - what callbacks it has run, what values have passed through it, and what other Deferreds it has been associated with.

Itamar picked up a ticket relating to HTTP client support for persistent connections. He also finished up the work on adding a new transport method, abortConnection, fixed a logging bug in the new TLS implementation, and did some ticket triage and a review.

Allister (amacleod) heroically tackled the problem of our insufficient Windows testing infrastructure, getting introduced to our build farm and setting up a new Windows 7 virtual machine on it.

Jessica (jesstess) did a ton of follow-up on old tickets (1247 2115 2447 2498 2507 2513 2861), trying to learn if they are still valid and elicit further information from the original reporters. She also managed to close one which was no longer valid (but she also filed a new one for a documentation bug, in case you were worried we might run out of tickets).

Glyph knuckled down on a review for our conversion to Sphinx. We're so close I can practically taste it.

Ying (cyli) took a look at improving the authentication options for some of Twisted's built-in servers. She started with the FTP server and made great headway.

As for myself, I reviewed that abortConnection feature as well as a ticket for HTML5 support in twisted.web.template and the TLS logging bug that Itamar fixed.

Thanks to all the sprinters and to Smarterer and the PSF for their support of the sprint! We're going to do another one of these real soon, watch out for an announcement!

Twisted's BuildBot master configuration

Twisted has a somewhat extensively customized BuildBot master configuration which has evolved over the last ten years. We've kept it private only because it contains some passwords which it would be better not to share, but over the years we have wanted to make it public - because it might serve as a good example to others, because it would let more people hack on it and relieve us of some work, etc. At last this has changed: I spent a couple hours today filtering out the sensitive parts. As a result, you can now take a look at lp:twisted-buildbot-configuration. Add some features! Do some refactoring! Write some unit tests!

Pre-trunk Testing

Twisted uses BuildBot for continuous integration - for testing every revision to ensure it passes the full test suite. BuildBot automatically runs the test suite on each version of Python and each operating system supported by Twisted. As of today, that's 22 different configurations, plus another 16 configurations where Twisted is known not to work completely but where we are workings towards full support.

This works very well for identifying regressions and for finding problems with new functionality on platforms a developer may not have considered. However, because Twisted prefers to keep trunk@HEAD completely working and the test suite fully passing on all platforms, if BuildBot identifies a problem with a trunk revision, we revert it completely. Then the problems with it can be addressed and another attempt made. This also works, but it has a couple shortcomings. Someone actually has to do the revert, which is a little bit of work. Also, someone might check out trunk@HEAD before the change is reverted, giving them a non-working version of Twisted. It's nice to be able to avoid both of these things.

The solution generally applied is to test the change before applying it to trunk. BuildBot can do this too, in a couple different ways. The way we use for Twisted development is to have BuildBot run the test suite on all those supported configurations on a branch of Twisted, rather than on trunk. Since the change is checked in to a branch, instead of to trunk, it avoids the needed to revert anything and it avoids offering broken code to people.

This approach involves a tiny custom tool to convince BuildBot to run the necessary builds. Normally BuildBot just watches for trunk commits and starts builds automatically. Since in this case we want the trunk commit to happen after we see that all the builds will succeed, we use force-builds.py to have BuildBot perform the builds before it sees the trunk commit.

For a long time, force-builds.py needed a hard-coded list of supported platforms on which to trigger builds. Recently I fixed this by having it retrieve that list from the BuildBot master, which already knows which platforms are supported. Publishing this information is done with a minor hack to master.cfg:

    supportedList = list(
        builder['name']
        for builder
        in builders
        if builder['category'] == 'supported')
    path = 'public_html/supported-builders.txt'
    supportedListFile = file(path, 'w')
    supportedListFile.write('\n'.join(supportedList) + '\n')
    supportedListFile.close()

The resulting file is publicly available from the master's web server, so force-builds.py can download it on-demand and then force a build on each supported builder. When the developer sees all of these builds have succeeded, he or she can be more confident that there aren't going to be any surprises when the change is actually applied to trunk.

New Buildbot!

And far to the East where strife sired shadows
A baelnorn.local did rise and approach the light.
Out came illidan.local over once fair meadows,
And in defeat left his people weeping for their plight.
                                                                 - kabuto.local's A Record

Jean-Paul and I had some hardware adventures a couple weekends ago.  As these things usually are, it was harrowing - but perhaps less so than it might have been.  By combining, in approximately equal parts, some unused components which have been ripening in the back of a closet in JP's apartment and some newly acquired materiels from the local vendor everyone (well, we, at least) loves to hate, and with only a few false starts involving incorrect memory channel configurations, Twisted now has the use of a somewhat hefty new computer, visible here in its new home, the undisclosed subterranean location that hosts all of Twisted's buildbot machines:

Don't ask my why it glows blue; it is my understanding that this is simply the fashion these days.

With enough cores and gigabytes of memory so that I have trouble counting them all, this machine should serve us well hosting many a build slave.  It has already taken over the duties of several older slaves, including those of a couple Windows slaves, care of kvm (the machine itself is running Ubuntu Lucid (10.04) of course).  The very nice network it is attached to means that remotely interacting with these is now only minimally unpleasant.

The machine is also translating PyPy (in a respectable 34 minutes) once a week so we can continue to test Twisted against the latest development versions of the PyPy runtime.

Volunteers needed for Twisted project jobs

The Twisted project is a lot more than just writing code. Volunteers help sustain Twisted by reviewing tickets, writing documentation, answering questions on IRC, the mailing lists, and Stack Overflow, donating build servers, managing releases, packaging Twisted for distributions, sponsoring development, and more.

To everyone who contributes to the Twisted project in some way, thank you!

Twisted project jobs

A small group of dedicated Twisted developers have been doing maintenance work over the years that we'd like to start distributing across more of the community. These tasks include:

• maintaining Buildbot buildslaves
• maintaining the Twisted Speed Center
• maintaining our Trac instance
• maintaining version control mirrors

We've broken these tasks into bite-sized chunks, and you can find descriptions for them on the Project Jobs wiki page. Take a look, and please claim a task and how long you're up for doing it on the mailing list. If you have questions about any of them, don't hesitate to ask on the mailing list or IRC. You'll have lots of help getting set up.

Twisted Matrix Labs high scores

Are you competitive? Need a little extra incentive to contribute? I challenge you to get to the top of the monthly Twisted Matrix Labs high scores list:

Ticket openers, reviewers, and closers all get varying amounts of points. Note that a review banks you 1000 points, the most for a single ticket action, and we have a long review queue, so brush up on our patch review guidelines and dig in.

May Sprint Report

Greetings audience.  Another month has gone by, and that means another excellent Twisted sprint is behind us.  A couple weekends ago Dave, Ying, Itamar, Glyph, Asheesh, and myself gathered at the now traditional sprinting location.  Quite a few things were worked on.  Amongst the highlights were the following tickets:

#4813 - provide permissions accessor for filepath
#5053 - Add Gzip support to web client
#5088 - cfreactor cannot be imported
#5063 - When TLS transport is in writeBlockedOnRead=True mode, data is buffered in memory but producer APIs are not respected
#5095 - twisted.protocols.tls should flush all of the bio data, rather than just 2**15 bytes

Evennia: Event-Driven Online Gaming with Twisted and Django

Do you have a hankering to run your own multi-user text-based game server? If you do, and you love Twisted, Evennia is the project for you! Through the power of Twisted, Evennia has built a MUD-server platform that not only provides traditional telnet access but also lets you play via the web! As Evennia maintainer Griatch Art put it:

Whereas traditional MUD servers only speak telnet, Evennia additionally comes with its own browser-based client and allows for expanding with custom game protocols in any combination. ... All of this is straightforward to implement and maintain thanks to Twisted!

Evennia also combines Twisted with Django, providing a comprehensive real-world case study of melding these two popular systems.  As Griatch puts it, Twisted and Django "work beautifully together".

To find out more about how Twisted was the perfect fit for this massively multi-player platform and why the Evennia team chose Twisted, check out the success story.

Have your own success story using Twisted? Let us know! Just send an e-mail to success@twistedmatrix.com

Twisted 11.0.0 released!

<lvh> PAS MAINTENANT CHEF! CHUIS EN TRAIN DE BRANCHER LE REACTEURRRRR

On behalf of Twisted Matrix Laboratories, I am honored to announce the release of Twisted 11.0.0.

Highlights include:

• a new templating system in Twisted Web, "twisted.web.template", derived from Divmod Nevow.
• improved behavior of subprocess spawning on FreeBSD.
• the 'twistd mail' plugin now uses the endpoints API, providing a more consistent command line and compatibility with endpoint plugins.
• twisted.plugin no longer emits a confusing traceback when it can't write a cache file.

and numerous other bugfixes and documentation improvements. For more information, see the NEWS file.

Download it now from:

<http://twistedmatrix.com/>

or install the 'Twisted' package from PyPI.

Many thanks to Glyph Lefkowitz and Jean-Paul Calderone for sanity-checking the pre-releases and release, and to the enthusiastic PyCon 2011 sprinters who annihilated dozens of tickets. Thanks to *everyone* who contributed tickets, patches, documentation, reviews, buildbots, feedback, and assistance to fellow users and developers leading up to this release. It is truly a group effort.

<http://twistedmatrix.com/highscores/?time=2011-03-01>

is a testament to how much work was done in March alone.

March (post PyCon) Twisted Sprint Report

We held a small but productive and successful Twisted sprint, kindly hosted once again by tenth, and attended by glyph, exarkun, jesstess and lewq.

The aim of the sprint was to do the first pre-release of Twisted 11.0 - and we managed to resolve almost all of the blocking tickets, so a partial success. :-)

One of the two main themes of the work was the twisted.web templating code, which got tidied up and properly documented - there's a new tutorial (temporarily available at http://buildbot.twistedmatrix.com/builds/sphinx-html/291-15849/projects/web/howto/twisted-templates.html), which demonstrates this neat little templating engine in action. The other theme was completing and merging the file descriptor closing branch which exarkun and lewq worked on, bringing FreeBSD just a few test-fixes away from being a fully supported platform - and importantly stopped the FreeBSD builder hanging on testing trunk, which will help catch regressions.

The following tickets got closed:

#4968 - Resource templating documentation has broken link to twisted.web.server.Resource which emphatically does not exist
#4881 - twisted.internet.process._listOpenFDs fails on FreeBSD because fdescfs isn't mounted by default
#4732 - Document @withRequest XML-RPC feature in the XML-RPC howto

And the following tickets got worked on:

#4854 - Replace the implementation of IReactorSSL with one based on twisted.protocols.tls
#3690 - SerialPort never calls connectionLost(reason)
#4969 - Add CONNECT support to HTTP client
#4951 - links in twisted.web docs are broken

That's it until next time, enjoy Twisted 11.0 when it comes out! We still have the PyCon sprint to report on, watch for that soon.

Twisted 11.0.0pre1 released -- please test

Intrepid Twisted developers and users: from Cambridge, Massachusetts I am pleased to announce the arrival of Twisted 11.0.0 pre-release 1.

Please download the pre-release tarballs and test away!

Highlights include:

• a new templating system in Twisted Web, "twisted.web.template", derived from Divmod Nevow.
• improved behavior of subprocess spawning on FreeBSD.
• the 'twistd mail' plugin now uses the endpoints API, providing a more consistent command line and compatibility with endpoint plugins.
• twisted.plugin no longer emits a confusing traceback when it can't write a cache file.

For more information, see the NEWS file.

Super Late February Twisted Sprint Report

Good news everyone,

Last month, Twisted held another very successful sprint.  A late write-up is better than no write-up, so here's what went on.

In attendance were Glyph, Tenth, Itamar, Mike Handverger, Luke, and myself (Jean-Paul).  Remotely, Jonathan J. and Stephen Thorne helped out.  Quite a few tickets were worked on, and various reviews were done as well.  Here's a list (bold indicates now closed):

#2036 - trial runs tests from .pyc files even if there is no .py file
#3834 - TCP client howto missing reactor.run() in one code sample#3844 - Parse IRC format codes
#3948 - wxdemo.py does not exit
#4008 - Most examples in documentation do not log messages and errors
#4520 - pb.CopiedFailure.throwExceptionIntoGenerator breaks in Python 2.6.
#4817 - IPv4Address and UNIXAddress not-equal comparison is broken
#4823 - Clock should re-sort pending calls when one of them is reset
#4836 - Make _getFunction method public in t.w.xmlrpc.XMLRPC
#4864 - Improve lore2sphinx buildbot results for `projects/core/howto/logging.xhtml`
 #4865 - Improve lore2sphinx buildbot results for `projects/core/howto/quotes.xhtml`

As I'm writing up this report at the March Twisted Sprint, look forward to another one of these real soon now. :)

Unclaimed PayPal donations

Hello Twisted supporters,

If you recently tried to donate money to the Twisted Software Foundation using PayPal, then you might find that your donation has gone "unclaimed".  This is due to an unfortunate mixup in our PayPal address.  If you have a donation in this state, you should cancel it; we have no way to claim it.

If you'd like to donate, use the PayPal form on http://twistedmatrix.com/. We've updated it so future donations will go to the right place.  If you'd rather use Google Checkout, the form for donating with that service is also still available in the same place.

January Twisted Sprint Report

Our highly productive Sunday Somerville sprint was attended by glyph, cyli, tpratt, tenth, itamar, mesozoic and myself (exarkun).  Also sprinting remotely were thijs, spiv, and magcius.  All told, we closed twelve tickets:

#3372 deprecate --extra option to trial

#3412 Include link to IMAP Server Tester in Twisted IMAP documentation

#4042 [Documentation] It is too hard to figure out how to do trivial common-case sending email with twisted.mail

#4054 Delete all of the out-of-date mumbo jumbo from the "im" howto

#4438 Declare dependencies explicitly when we add them

#4491 Release Twisted 10.0.1

#4547 Remove deprecated twisted.protocols.loopback.loopback

#4712 Missing bits of statinfo accessors in FilePath

#4738 ckeygen man page has wrong summary and synopsis

#4740 Use strcred for `twistd mail` authentication options

#4773 The core howto index should link to the endpoints howto

#4786 twcgi duplicate header

More tickets were also worked on and reviewed but not quite yet ready to close:

#4317 twisted.web.http.Request allows you to write to a finished request.
#3896 Passing a unicode object to request.write corrupts the entire response
#4796 twistd mail plugin esmtp problem
#4760 HTTPPageGetter shouldn't make two requests with an afterFoundGet
#4632 ability to cascade canceling inlineCallbacks's deferred
#3077 twistd --help plugin list isn't alphabetized

There was also a lively discussion about plans for the future support of Twisted on Windows.  If you're interested in Windows networking, keep an eye out for an upcoming announcement on that topic!

Once again we were lucky enough to have tenth host the sprint, so in addition to shelter from the frozen terrorscape scant meters away, sprinters also enjoyed delicious fresh baked bread and other tasty morsels.

If you couldn't make it to this sprint, watch for an announcement of the next one sometime soon.  Or join us at PyCon for as many as four days of Twisted hacking!