Monday, December 14, 2009

Sponsored Development, December 2009

Hello all,

It's been a while since my last report of sponsored Twisted development. After this hiatus, I'm happy to be able to bring you a new list of accomplishments I was able to achieve thanks to the support of the Twisted sponsors and the Software Freedom Conservancy.

The tickets I reviewed this time around (italics for tickets now resolved):

#3007 - Subsystem's connectionLost method executed twice in certain situation
#3226 - twisted.python.util.initgroups (and thus spawnProcess) potentially does tons of unnecessary IO
#3296 - Account for ISUPPORT information in irc_MODE when parsing mode strings
#3931 - t.p.amp.AmpBox.serialize will _not_ return a string if any value is unicode
#4009 - twisted.python.modules.PythonPath docstring typo
#4029 - Twisted Web's wsgi.input isn't sufficiently file-like in all cases
#4065 - Replace 'callable(foo)' with 'hasattr(foo_ "__call__")'

And the tickets I worked on (often with the help of others):

#733 - twisted's SIGCHLD handler breaks popen
#970 - twisted.names.root.Resolver eats filedescriptors
#1784 - disttrial --hosts=kunai_takkun_muon twisted
#2078 - Twisted shouldn't start processes until the reactor is running
#2884 - Tool to update / generate NEWS file
#3886 - Fixing the Socks4 proxy to be Socksv4a compliant
#3922 - twisted.protocols.amp.BinaryBoxProtocol.connectionLost chokes on reason = main.CONNECTION_DONE
#3932 - HTTPAuthSessionWrapper incorrectly returns a 500 if the wrapped resource doesn't implement the requested method
#3956 - Add arraysize option to runQuery in adbapi
#4019 - t.web.wsgi.WSGIResource - WSGI application error handling
#4029 - Twisted Web's wsgi.input isn't sufficiently file-like in all cases
#4031 - twisted.mail.smtp client doesn't really implement SASL LOGIN
#4049 - Error IMAP4Client.fetchSpecific an HTML body part
#4053 - Replace "x.has_key(y)" with "y in x"
#4063 - Ween trial's --dry-run away from the deprecated TestCase.visitor code
#4066 - `reduce` for Python 3.x
#4075 - Remove deprecated IReactorUDP.connectUDP
#4114 - Test all of the file-related twisted.web.wsgi code-paths with each of the possible file-like objects they might be used with
#4115 - lore -olint's `check_texturl_matches_href` and `check_a_py_listing` methods invoke getAttribute wrong
#4116 - Add a parameterized AMP argument type which can serialize and unserialize 0 or more of a specified type
#4131 - `twisted.web.client.Agent` does not add a Host header if `None` (the default) is passed for the `headers` argument to `request`
#4154 - Skip all release automation related tests on platforms which are not Linux

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit For example,

Additionally, you may have noticed that we managed to finally release Twisted 9.0. This was mostly thanks to the efforts of Chris Armstrong (as usual), but I also helped out reviewing some of that process.

This should signal the beginning of more regular sponsored development. Look forward to seeing another one of these reports soon. :)

Friday, November 20, 2009

Twisted 9 is in the works, first pre-release is available

Hi all! I've done an initial pre-release of Twisted 9. I've made available a set of tarballs (The one called Twisted-9.0.0pre1.tar.bz2 is the main release) and the release notes.

Please try out this prerelease and tell us about any regressions or other release-related problems you find. If all goes well, we want to hear about that too.

Friday, November 6, 2009

New OS X build slave

As some of you may recall, Apple lent Twisted an Xserve G5 a few years back. This was invaluable in maintaining Twisted support for OS X. A few weeks ago we unfortunately lost the ability to host this machine. Fortunately, David Reid came through and very generously donated a

which is now set up and running the Twisted test suite continuously. Thanks, David!

Monday, May 25, 2009

Canonical Men of Twistery

The title of this quick little post goes to the excellent name mwh gave the linked photo :-) Taken at a dinner during AllHands in a gorgeous area outside of Barcelona, this pic shows the collection of Twisted developers that Canonical has hired (so far!).

Top row, left to right: Andrew Bennetts (spiv), Thomas Herve (therve), Chris Armstrong (radix), Michael Hudson (mwh), Jamu Kakar (jkakar).

Bottom row, left to right: Nicola Larosa (tekNico), Duncan McGreggor (oubiwann), Jonathan Lange (jml).

Pretty freakin' cool :-)

Saturday, May 23, 2009

Sponsorship In 2009

It's that time of year again; the time when you all have the excellent opportunity to settle your karmic debt and sponsor the Twisted project.

Actually, that time is every day of the year.  Right now is just when we have arbitrarily decided to make you aware of this opportunity, so that we're not annoying you all year-round.  Sometime soon we'll be taking down the current list of sponsors for 2008 — hopefully, to make room for a new list of sponsors for 2009.  But without your help, that list will be empty!

Thanks to our founding sponsors, Twisted has had a great year. If you've been reading this news feed, you know that we've been able to review, respond to, and close tons of tickets this year.

As Jean-Paul Calderone previously wrote, this is the first year we've had a year-over-year decrease in the number of open tickets; and that's despite an increase in the number of filed tickets.  We believe that Twisted is setting a great example for how other open source projects can effectively use sponsorship to fund development.

Many open source projects are perpetually crunched for volunteer labor. Twisted is no different in this respect — we could always use more help from the community.  However, we are lucky to have something that most projects don't: the ability to convert dollars directly into maintenance that benefits the project.

When you contribute money to the Twisted project, you are directly contributing to get bugs fixed, features implemented, and tickets reviewed.   Last year we told the same story, but this year we have an excellent track record. I invite you to read the history of our sponsored development — just scroll down and have a look. I'll wait.

... Done?

If you prefer visual representations of our success instead, consider these graphs.  Here's a graph of all tickets reviewed this year, by the handle of the person who reviewed them:

Here's the same breakdown of all tickets that were closed this year:

In case you don't already know: "exarkun" is the handle for Jean-Paul Calderone.  By his own calculation, at least 75% of his work this year has been sponsored development.

This isn't to say that Twisted is being supported entirely by sponsorship: as you can see from the huge number of tickets that exarkun has reviewed, a substantial proportion of development has been coming from the community, but sponsorship has allowed us to be more responsive to the community and integrate contributions much faster.  (Developers may not review their own tickets, so there's no overlap between the code JP has been writing and the tickets he's been reviewing.)

This year of sponsorship has been great for the project.  But, we're coming to the end of our first year of funding, and we need your help to keep going.

Every little bit helps, from small, one-time individual donations to monthly subscriptions to corporate sponsorships. This year, we need even more help from individuals, as some of our corporate sponsors have been impacted by the current economic climate and may not be able to contribute at the same level this year.

To summarize, there are a few things you can do to help the Twisted project:
  1. Contribute code.  Any contribution is welcome, but the best would be to attack an existing ticket in the tracker, especially one that has stalled on its way through review.
  2. Donate money directly.  You'll find donation buttons on the front page of
  3. Donate money from your organization.  Doing so gets you a voice on our sponsors-only mailing list, and can help you to set priorities for the coming year of development.  You also get your logo on, our gratitude, and last but not least, more Twisted development in the coming year!
  4. Help us in our discussions with larger organizations.  If you're a software developer using Twisted in a big company, we need your help to convince your organization that they should be contributing.  Even if not, the board of the Twisted project is small and strapped for time - we could use some help just finding the right person to talk to, and making sure that all of their questions are answered promptly.  (If you'd like to help with this effort, please contact me directly.)

Thanks for your interest in Twisted.

Monday, March 2, 2009

February Sponsored Twisted Development

Another month, another round of Twisted development. February saw progress on a miscellany of bug fixes, as well as further work on the new SIP implementation. List of tickets I worked on (bold tickets are now closed):

#2036 - trial runs tests from .pyc files even if there is no .py file
#2194 - small bug in SIP Via header generation
#3634 - restat in render in static.File raise Exception if file doesn't exists
#3639 - suppress deprecation warnings emitted by twisted.test.test_sip
#3643 - MemCacheProtocol does not return an error if it is used after disconnecting
#3648 - twisted.cred.credentials.UsernameHashedPassword doesn't hash password strings when checkPassword
#3654 - unlink of unix socket in connectionLost may cause twistd daemons to ignore SIGTERM
#3584 - SIP transport and transaction layer
#3660 - IRCClient reports ":None:None" to CTCP version request

Work on #3654 unearthed a more general problem where twistd fails to exit when a service encounters an exception during shutdown. So hopefully there'll be less need to use kill -9 during development :)

RFC 3261 is the longest RFC for a network protocol the IETF has ever produced. The SIP transport/transaction branch only provides the bottom two layers of a full SIP stack, but introduces 1700 new lines of code. Special thanks to Chris Armstrong and Jean-Paul Calderone for their amazingly detailed review. I managed to get halfway through the issues they raised, and expect to finish it up next time.

Once again, thanks to the Software Freedom Conservancy, all of the Twisted Sponsors, and all the other developers who contribute to Twisted.

Sunday, February 1, 2009

January Sprint Report

Today, we had a last-minute unofficial Twisted sprint for January, at the Divmod office in Porter Square, Cambridge, Massachusetts.

Thanks to everyone who contributed.
Thanks also to other contributors today not at the sprint: 'atsuoi' who contributed a patch for an SMTP issue, and Paul Swartz, who (it looks like) spent some time working on testing user authentication in Conch.  (Commit those changes, Paul!)

We are tentatively planning to hold our next sprint on Sunday, February 22.  If you're in the Cambridge, Massachusetts area, please stop by!

Monday, January 26, 2009

More January Sponsored Twisted Development

First, a list of tickets I worked on:

  • #2036 - trial runs tests from .pyc files even if there is no .py file
  • #3575 - Common implementation of RFC 2617 digest authentication (closed)
  • #3582 - Improve SIP URI parsing/formatting
  • #3584 - SIP transport and transaction layer

It is the last of these on which I spent the bulk of the time, implementing the lower layers of SIP message handling as described in RFC 3261. As far as I know, this is the first implementation of this protocol in Python that has thorough unit test coverage! While reviewing the code from Sine, Divmod's SIP implementation, and comparing to the spec I discovered some significant misunderstandings that I had the chance to fix here. This code lays the foundation for both user agent code (like VoIP clients or SIMPLE instant messaging) and proxies.

Thanks to the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Monday, January 19, 2009

January Sponsored Twisted Development

On Friday I finished the first two weeks of sponsored Twisted development for 2009. This week, the number of open tickets in the Twisted issue tracker hit 1040 - going down. We hit 1040 on the upside in November of 2007. Thanks in very large part of the sponsors who made so much of my time available for Twisted development in 2008, we closed more tickets than we opened last year for the first time (Drew P quipped that Twisted will be bug-free and feature-complete in 2636 - better than never ;). On average, we closed one ticket every 14 hours last year - all year. This trend is clearly visible in the graph included in the weekly summary of issue tracker activity. The green line shows total tickets filed, the red line shows total open tickets.

During this round of development, there was the usual smattering of web-related work:

#446 - Twisted Web accepts any HTTP method as if it was GET
#886 - Add an HTTP 1.1 client protocol implementation to twisted.web
#1176 - A t.web glossary is needed (closed)
#1522 - Remove woven from next release of twisted.web (closed)
#2446 - Python 2.4 has poor round-trip behaviour for xmlrpclib.DateTime; Twisted should work around
#2030 - Deprecate twisted.web.trp
#3035 - deprecate Resource subclasses in twisted.web.error, add equivalents in twisted.web.resource
#3416 - rename `twisted.web2.iweb.IOldRequest` to `twisted.web.iweb.IRequest` (closed)
#3457 - Session expiry check frequency should be based on sessionTimeout
#3600 - test twisted.web.domhelpers against xml.dom.minidom (closed)
#3601 - crash in twisted.web.http triggered by a blackberry web browser (closed)
#3605 - Factor "identity" transfer encoding support out of HTTPChannel.rawDataReceived into an object like the chunked transfer encoding object (closed)
#3610 - Give twisted.web.http_headers.Headers a nice repr (closed)

Of course #886 shows up in the list yet again. At this point the protocol implementation is essentially done. The code has been reviewed and most of the feedback dealt with. The only reason it isn't closed yet is that we want to provide more than just an HTTP/1.1 client protocol implementation. We want to provide an easy-to-use, high-level web client. Before I call #886 finished, I want to implement a few of those high-level APIs. So #886 is going to remain open for a little while longer, but I expect it to be closed very soon now.

Another highlight on this list is #1522 - the removal of Woven from Twisted. That's right, it's gone, guys. I really hope that after almost three years of being deprecated and almost six years of there being better alternatives around no one still has code using this. If so, you're out of luck for the next Twisted release. Perhaps you should take a look at Nevow. :)

Aside from web tickets, there were some more FTP improvements:

#3593 - passivePortRange in not working (closed)
#3596 - Active mode makes crash (closed)

Some improvements which make writing unit tests with Twisted and for Twisted-based code easier:

#3598 - TestCase.flushWarnings shouldn't use source files to implement its "offendingFunctions" feature (closed)
#3609 - Add IConsumer and IProducer support to StringTransport to facilitate testing producers and consumers (closed)

And all kinds of other assorted fixes and enhancements:

#733 - twisted's SIGCHLD handler breaks popen
#1785 - should raise an error when called while the reactor is running (closed)
#3560 - remove usage of microdom from twisted.lore
#3576 - add high-level, cross-platform close-on-exit togglers to twisted.internet.fdesc (closed)
#3591 - ThreadedResolver--and twisted.internet.thread--assumes that the reactor is the installed reactor
#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented
#3602 - Improve factoring of various reactor implementations of "internal" FD tracking
#3606 - incorrect link in application howto (closed)

Changes to the internal factoring of reactors (#3602) and additions to fdesc (#3576) were both made in support of #733 - a very old ticket for a behavior which often trips people up. Glyph did the initial work on these improvements and Itamar and I came back and polished them a bit last week. With one of these closed and the other almost closed, Twisted is getting pretty close to not getting in the way of popen (and more generally, not causing syscalls to start to fail with EINTR). This is a generally nice thing, but it's taken a long time because it requires access to POSIX APIs Python doesn't expose.

Further work on #3560 has gotten Lore almost to the point where it's using minidom instead of microdom. Glyph raised a point in his review of the code which threw one further wrench into the workings, though. Currently, Lore input documents are implicitly XHTML. This means a Lore input document may use named character entities without declaring one of the XHTML DTDs. However, in order to continue to support this with minidom, it's necessary to reach way into expat and use a couple very confusing features. Itamar and I finally worked out how to do this, but if one is using xml.sax (which we are) then it involves using private attributes. Not nice. Tune in next month to learn how this turns out. ;)

Of course, I also review some tickets over the past two weeks:

#532 - Big jump from to in tutorial
#918 - If program has > 1024 fds, the select() used in t.i.process can fail (closed)
#1009 - Document defgen
#2110 - finger tutorial uses unclear python idioms (closed)
#2682 - twisted.conch.ssh.userauth is poorly tested
#3245 - __slots__ in Deferred class
#3498 - conch connection-sharing should be disabled by default, at least until it works a little
#3575 - Common implementation of RFC 2617 digest authentication (closed)
#3584 - SIP transaction layer
#3613 - a streaming producer doesn't have to be resumed when a buffer is empty (closed)

Several documentation tickets there, as well as some good SIP stuff (more of Allen's work). The digest authentication ticket (#3575) was a nice one. Twisted has something like three different implementations of digest authentication in it, each for a different protocol with slightly different requirements. #3575 is the beginning of the consolidation of that code, a pretty important task.

Meanwhile, Allen has been working on the SIP code in Twisted in parallel with my development. A report of his work will show up here shortly.

Thanks to the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.