More January Sponsored Twisted Development

First, a list of tickets I worked on:

• #2036 - trial runs tests from .pyc files even if there is no .py file
  
• #3575 - Common implementation of RFC 2617 digest authentication (closed)
  
• #3582 - Improve SIP URI parsing/formatting
  
• #3584 - SIP transport and transaction layer
  

It is the last of these on which I spent the bulk of the time, implementing the lower layers of SIP message handling as described in RFC 3261. As far as I know, this is the first implementation of this protocol in Python that has thorough unit test coverage! While reviewing the code from Sine, Divmod's SIP implementation, and comparing to the spec I discovered some significant misunderstandings that I had the chance to fix here. This code lays the foundation for both user agent code (like VoIP clients or SIMPLE instant messaging) and proxies.

Thanks to the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

January Sponsored Twisted Development

On Friday I finished the first two weeks of sponsored Twisted development for 2009. This week, the number of open tickets in the Twisted issue tracker hit 1040 - going down. We hit 1040 on the upside in November of 2007. Thanks in very large part of the sponsors who made so much of my time available for Twisted development in 2008, we closed more tickets than we opened last year for the first time (Drew P quipped that Twisted will be bug-free and feature-complete in 2636 - better than never ;). On average, we closed one ticket every 14 hours last year - all year. This trend is clearly visible in the graph included in the weekly summary of issue tracker activity. The green line shows total tickets filed, the red line shows total open tickets.


During this round of development, there was the usual smattering of web-related work:

#446 - Twisted Web accepts any HTTP method as if it was GET
#886 - Add an HTTP 1.1 client protocol implementation to twisted.web
#1176 - A t.web glossary is needed (closed)
#1522 - Remove woven from next release of twisted.web (closed)
#2446 - Python 2.4 has poor round-trip behaviour for xmlrpclib.DateTime; Twisted should work around
#2030 - Deprecate twisted.web.trp
#3035 - deprecate Resource subclasses in twisted.web.error, add equivalents in twisted.web.resource
#3416 - rename `twisted.web2.iweb.IOldRequest` to `twisted.web.iweb.IRequest` (closed)
#3457 - Session expiry check frequency should be based on sessionTimeout
#3600 - test twisted.web.domhelpers against xml.dom.minidom (closed)
#3601 - crash in twisted.web.http triggered by a blackberry web browser (closed)
#3605 - Factor "identity" transfer encoding support out of HTTPChannel.rawDataReceived into an object like the chunked transfer encoding object (closed)
#3610 - Give twisted.web.http_headers.Headers a nice repr (closed)

Of course #886 shows up in the list yet again. At this point the protocol implementation is essentially done. The code has been reviewed and most of the feedback dealt with. The only reason it isn't closed yet is that we want to provide more than just an HTTP/1.1 client protocol implementation. We want to provide an easy-to-use, high-level web client. Before I call #886 finished, I want to implement a few of those high-level APIs. So #886 is going to remain open for a little while longer, but I expect it to be closed very soon now.

Another highlight on this list is #1522 - the removal of Woven from Twisted. That's right, it's gone, guys. I really hope that after almost three years of being deprecated and almost six years of there being better alternatives around no one still has code using this. If so, you're out of luck for the next Twisted release. Perhaps you should take a look at Nevow. :)

Aside from web tickets, there were some more FTP improvements:

#3593 - passivePortRange in ftp.py not working (closed)
#3596 - Active mode makes ftp.py crash (closed)

Some improvements which make writing unit tests with Twisted and for Twisted-based code easier:

#3598 - TestCase.flushWarnings shouldn't use source files to implement its "offendingFunctions" feature (closed)
#3609 - Add IConsumer and IProducer support to StringTransport to facilitate testing producers and consumers (closed)

And all kinds of other assorted fixes and enhancements:

#733 - twisted's SIGCHLD handler breaks popen
#1785 - reactor.run should raise an error when called while the reactor is running (closed)
#3560 - remove usage of microdom from twisted.lore
#3576 - add high-level, cross-platform close-on-exit togglers to twisted.internet.fdesc (closed)
#3591 - ThreadedResolver--and twisted.internet.thread--assumes that the reactor is the installed reactor
#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented
#3602 - Improve factoring of various reactor implementations of "internal" FD tracking
#3606 - incorrect link in application howto (closed)

Changes to the internal factoring of reactors (#3602) and additions to fdesc (#3576) were both made in support of #733 - a very old ticket for a behavior which often trips people up. Glyph did the initial work on these improvements and Itamar and I came back and polished them a bit last week. With one of these closed and the other almost closed, Twisted is getting pretty close to not getting in the way of popen (and more generally, not causing syscalls to start to fail with EINTR). This is a generally nice thing, but it's taken a long time because it requires access to POSIX APIs Python doesn't expose.

Further work on #3560 has gotten Lore almost to the point where it's using minidom instead of microdom. Glyph raised a point in his review of the code which threw one further wrench into the workings, though. Currently, Lore input documents are implicitly XHTML. This means a Lore input document may use named character entities without declaring one of the XHTML DTDs. However, in order to continue to support this with minidom, it's necessary to reach way into expat and use a couple very confusing features. Itamar and I finally worked out how to do this, but if one is using xml.sax (which we are) then it involves using private attributes. Not nice. Tune in next month to learn how this turns out. ;)

Of course, I also review some tickets over the past two weeks:

#532 - Big jump from finger18.py to finger19.py in tutorial
#918 - If program has > 1024 fds, the select() used in t.i.process can fail (closed)
#1009 - Document defgen
#2110 - finger tutorial uses unclear python idioms (closed)
#2682 - twisted.conch.ssh.userauth is poorly tested
#3245 - __slots__ in Deferred class
#3498 - conch connection-sharing should be disabled by default, at least until it works a little
#3575 - Common implementation of RFC 2617 digest authentication (closed)
#3584 - SIP transaction layer
#3613 - a streaming producer doesn't have to be resumed when a buffer is empty (closed)

Several documentation tickets there, as well as some good SIP stuff (more of Allen's work). The digest authentication ticket (#3575) was a nice one. Twisted has something like three different implementations of digest authentication in it, each for a different protocol with slightly different requirements. #3575 is the beginning of the consolidation of that code, a pretty important task.

Meanwhile, Allen has been working on the SIP code in Twisted in parallel with my development. A report of his work will show up here shortly.

Thanks to the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.