Tuesday, December 30, 2008

Summary of December Sponsored Twisted Development

With a lot of help from Allen Short, two more weeks of sponsored Twisted development have just been completed. During this round, I continued to work on the new HTTP client and spent some time trying to resolve some long-standing issues around Twisted's support for starting and controlling child processes. Meanwhile, Allen developed a plan for integrating the SIP code, maintained by Divmod, Inc. outside of Twisted for several years, into Twisted and the existing SIP support in Twisted.

The SIP-related work that Allen did encompassed these tickets:

#2194 - small bug in SIP Via header generation
#3575 - Common implementation of RFC 2617 digest authentication
#3582 - Improve SIP URI parsing/formatting
#3583 - Include SIP message-parsing changes from Sine
#3584 - SIP transport layer and transaction layer

These are the process tickets that I worked on:

#733 - twisted's SIGCHLD handler breaks popen
#1997 - perhaps wakeUp could be slightly simpler (closed)
#2967 - Reaping child processes has superlinear complexity on POSIX
#3571 - intermittent spawnProcess failure in test_process on Linux (closed)
#3576 - add high-level, cross-platform close-on-exit togglers to t.i.fdesc

A lot of this work consisted of getting some old code which Glyph had previously worked on but never completed updated and brought closer to being ready for a real code review. The summary of #1997 is a bit misleading - while the change did simplify the reactor's "wake up" mechanism, it also removed a race condition, fixing a bug which could cause certain events to fall into a limbo where they could only ever be processed after another event arrived.

Aside from process related tickets and the HTTP client, I worked on a few other tickets as well:

#2808 - AMP should raise MissingArgument (or other) if a callRemote is called with wrong arguments (closed)
#3246 - remove all mentions of plugins.tml from the documentation (closed)
#3562 - Setup Python 3.0 buildslave(s) (closed)
#3568 - ERROR from conch test when pycrypto is not installed
#3569 - Twisted Web WSGI container sometimes emits too many (or duplicate) headers (closed)

The Python 3.0 buildslave will probably garner the most interest of the bunch. The resolution of this ticket does not mean that Twisted supports Python 3.0 now. It just means that we've added a column to our buildbot (continuous integration system). We can now tell at any given time that Twisted does not support Python 3.0. ;) But seriously, with this slave set up, we can accept contributions which move Twisted towards Python 3.0 compatibility, but I don't plan to spend any time doing such development myself in the near future. There's a ton of other more pressing issues, so I'll leave Python 3.0 work to people who think they'll benefit from it.

As for the new HTTP client, this round of development moves it inexorably towards completion. Itamar Shtull-Trauring and I spent several days improving its error handling, simplifying some of the more hideous parts of the implementation, and dealing with various corner cases (and HTTP 1.1 sure has a lot of them). The development branch also includes a sketch (only a sketch) of the higher-level API we're planning to provide on top of the low-level protocol implementation (the sketch is currently undocumented and a bit obtuse, so it may not make sense without me or Itamar looking over your shoulder) and an example which uses the APIs provided by the low-level protocol to implement a simple web client (something along the lines of wget or curl, but obviously much, much, much more rudamentary).

That's it for now. It'll be 2009 when I post the next one of these. 2008 has been a great year for Twisted development, and I know that things are just going to get better. :) Thanks again to the Software Freedom Conservancy, all of the Twisted Sponsors, and all the other developers who contribute to Twisted.

Wednesday, December 3, 2008

Summary of November Sponsored Twisted Development

I've just completed another round of sponsored Twisted development (the sixth so far). This period sees a lot of documentation improvements, as well as various bug fixes and continued work on a new HTTP client for Twisted Web (#886).

These are the documentation issues which were resolved:

#2281 - Annotations for Twisted Finger Tutorial
#3548 - twisted.conch.client.knownhosts.PlainEntry misdocuments its "_hostnames" attribute.
#3455 - CONNECTION_LOST not an Integer...
#3537 - Conch's public key authentication process is confusing.
#3490 - FTPClient errors should provide ftp errorcode

There were some improvements to Twisted Web:

#1878 - twisted.web.monitor traceback_ AttributeError: class IChangeNotified has no attribute '__class__'
#2402 - client.py crashes on URL's that would be no problem for most browsers
#3192 - HTTPClientFactory sets followRedirect on the HTTPPageGetter class
#3469 - Exception is rendered when NotFound is more appropriate.

A problem with Conch's SFTP server's reporting of modification times was fixed (and then a problem with the unit tests for the fix was fixed):

#3503 - Wrong date format delivered by twisted.conch.ls.lsLine
#3551 - TZ=Pacific/Auckland python2.4 ./bin/trial twisted.conch.test.test_cftp.ListingTests fails

With #3551, I was once again reminded that working with timezones is extremely challenging. This time, I found that the range of local renderings of any particular UTC timestamp is greater than 24 hours, so you cannot rely on any particular timestamp falling on a particular day in an arbitrary timezone. So like other unit tests in Twisted, the unit tests for lsLine explicitly set the timezone when they want to make assertions which depend on it, then reset it to its original value. Unfortunately, this means they can't run on Windows, since Windows apparently lacks APIs for doing this. I also rediscovered that time.tzset() is a no-op, at least on Linux (but the unit tests call it anyway, in case Solaris or HP-UX or some other POSIX implementation requires it).

An important issue with Twisted's Jabber support was fixed as well (#3463). This one prevented Twisted's Jabber client from successfully negotiating TLS when connected to Google Talk (and possibly other Java-based Jabber servers). This problem was ultimately caused by a bug in the TLS support on Google Talk which caused TLS negotiation to fail if the client included a session ticket (RFC 5077) section in the handshake. This is allowed and servers which do not support session tickets should ignore the section, but for some reason it causes problems with Google Talk. OpenSSL (on which Twisted's SSL support is based) 0.9.9 enables session tickets and the 0.9.8 package distributed with some platforms (eg Ubuntu 8.10) includes a backport of this feature. So Twisted's Jabber client cannot communicate with Google Talk if one of those versions of OpenSSL is installed.

A couple issues related to Python 2.6 support were fixed:

#2763 - md5 and sha module will be deprecated in python 2.6
#3545 - Support Python 2.6 in the Windows build system.

A bug in Twisted Mail's IMAP4 client which prevented the unseen part of a server's response to a select or examine command from being made available to applications was fixed (#3550).

And there were several other assorted fixes:

#3521 - Documentation for `processExited()` conflicts with the implementation
#3315 - t.p.reflect.safe_repr includes the wrong traceback and misformats the return value
#3541 - twisted.internet.abstract.FileDescriptor.loseConnection drops reason (will be reported as clean shutdown)
#3544 - bin/admin/change-versions should update the main README file

I also spent some time working on converting Twisted Lore from using microdom, Twisted's XML parser and DOM implementation (circa 2002), to using minidom, the XML parser and DOM implementation in the Python standard library. For a long time, microdom was better than any of the alternatives, but it's seen very little maintenance in the past several years and there are some problems with Lore (eg #414) which are caused by behavior of microdom that it would be difficult to change.

Unfortunately, switching to minidom brings a new set of problems. It's still probably worthwhile, but it seems like it's harder to use than it should be. Some of the issues I've run into so far (and I'm not done yet):

  • minidom's constructors are less convenient than microdom's constructors. For example, the Text constructor doesn't accept a string to use as the text node's value. Instead, you have to instantiate Text and then set an attribute. This expands code which should have been one line into three lines. And to make things worse, a Text instance with no data set raises an exception from the __repr__ method.
  • The parse error exceptions raised by expat are less informative than the parse error exceptions raised by microdom. For example, if a document contains mismatched tags, microdom reports the name and location of both the opening and closing tags. minidom will report only the location of the closing tag. It's easy enough to find out the name of the closing tag by finding that location in the input document, but finding the offending start tag means parsing the document in your head. For any non-trivial document this is ridiculous. Fortunately, by switching to sax and providing custom error and content handlers, most of the information can be recovered. This information is always useful though, and it would be better if minidom provided it by default.
  • Once you switch to xml.sax, you have to remember to disable its validation features or it will try to retrieve DTDs from the internet every time you parse something. This is bad, bad default behavior.
Some of these issues may turn into Python bug reports once I've made more progress on converting Lore. A much bigger difficulty with the conversion than the problems minidom has is the fact that Lore is largely pre-UQDS code. Some of it has tests, but they're mostly whole-system tests which compare gigantic xhtml strings and are subject to extremely obscure failures. And most of it doesn't have any tests at all.

Switching away from microdom should be worth the effort though. minidom is a bit faster and Lore will generate better looking output once the switch is done.

Itamar points out I didn't separate tickets into groups for those I reviewed and those I did development on myself this time. So let me point out that of the above tickets, many were developed by others and reviewed by me. The Twisted development process is highly collaborative. I couldn't accomplish anything without the help of all the other great Twisted developers who volunteer to contribute to Twisted in their free time. If you want to find out which are which, head over to the Twisted issue tracker where you can look up the development(/authorship/etc) history of any ticket.

That's all for now. Thanks again to the Software Freedom Conservancy, all of the Twisted Sponsors, and all the other developers who contribute to Twisted (hi Michael!).

Monday, November 3, 2008

Summary of October Sponsored Twisted Development

Hello again,

I've just spent another two weeks on sponsored Twisted development. This round sees 25 tickets resolved. Considering the fact that two tickets in particular took up a lot of development time, I'm pretty happy with this progress. I did a lot of reviews for other people this time around. Here are some of the more interesting things developed by other people to which I gave the thumbs up:

#2026 - Edit the twisted.internet docstring to make contents clearer
#2902 - bad flag value in twisted.conch.ssh.filetransfer
#3335 - FTPClient should support renaming files
#3491 - FTPClient should support deleting files
#3500 - Add directory creation method to FTPClient

(The other tickets I reviewed which weren't quite as as interesting were: #532 #745 #1853 #2281 #2375 #2682 #2748 #3197 #3216 #3315 #3381 #3393 #3439 #3464 #3466 #3471 #3482 #3486 #3504 #3505 #3507)

And here are some of the more interesting ones I worked on myself (bold tickets are now closed):

#886 - Rewrite twisted.web.client.getPage to support streaming of result data and returning headers etc
#2346 - Header generation not conforming to RFC822 and RFC2046
#2605 - stdlib unittest change breaks a trial test
#3298 - twisted.internet.defer.FirstError masks errors when logged
#3329 - HTTP's #(...) syntax allows null contents
#3477 - BinaryBoxProtocol.makeConnection calls startReceivingBoxes too early
#3478 - AMP should enforce MAX_KEY_LENGTH in the protocol parser
#3487 - Add TestCase.flushWarnings

(The other slightly less exciting tickets I worked were: #3169 #3197 #3216 #3515 #3519)

#3487 and #886 took up a lot of my time this round. With #3487 resolved, the way is clear for the next Twisted release to run well on Python 2.6 (ie, there are no more known issues). Once I got that out of the way, I moved back to web issues. #886 is going to represent a big step towards having a good HTTP client API in Twisted. Anyone who has tried to use twisted.web.client for anything non-trivial knows how much work is needed in this area. :)

As usual, this work is made possible by the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code!

That's all until next time.

Friday, September 26, 2008

Summary of September Sponsored Twisted Development

Another round (the fourth) of sponsored Twisted development has just wrapped up.

I had a lot of excellent help again from Thomas Hervé and Thijs Triemstra. Thomas is a great reviewer and Thijs has been making steady progress on resolving lots of documentation tickets.

Here are the tickets I spent time developing:

#1068 - UNIX Port creation has race condition in permission setting code
#1382 - twisted.web.client.HTTPClientFactory sends Host header (and others) more than once
#1903 - Twisted doesn't reset timeout when getting POST data
#2542 - twisted.web.microdom.Node.isEqualToNode does no actual comparison
#2683 - twisted/web/http.py has self._header instead of self.__header
#2878 - Intermittent unclean errors from twisted.test.test_ftp.FTPServerPasvDataConnectionTestCase.testTwoDirLIST on OS X
#2947 - intermittent twisted.test.test_pb.NSPTestCase.test_NSP failure on OS X
#2974 - twisted.vfs.test.test_vfs.OSVFSTest.setUp does not close the files it opens
#2976 - twisted.mail.smtp.xtext_encode and xtext_decode take the wrong number of arguments
#3133 - Whois function for the ircclient class
#3211 - twisted.python.log.showwarning doesn't take the new line argument added in Python 2.6
#3222 - Intermittent failure of twisted.test.test_twistd.AppProfilingTestCase.test_profileSaveStats on Windows
#3223 - twisted.trial.unittest.TestCase.assertWarns always fails when the C warnings module_ new in Python trunk_ is in use
#3239 - Intermittent failure of twisted.test.test_iutils.UtilsTestCase.testOutputWithErrorIgnored on Windows
#3266 - Provide tools for managing new deprecation policy
#3402 - Intermittent failure of twisted.test.test_tcp.LocalRemoteAddressTestCase.test_hostAddress on OS X
#3404 - twisted.test.test_process.ProcessTestCase.testManyProcesses fails on WinXP
#3416 - rename `twisted.web2.iweb.IOldRequest` to `twisted.web.iweb.IRequest`
#3424 - Intermittent failure of twisted.names.test.test_names.ServerDNSTestCase.testZoneTransfer on OS X
#3426 - twisted/internet/test/test_process.py tests which might fail while waiting for an event should use runReactor instead of reactor.run

As you can see, work improving Twisted Web continues here. There's also been some long-needed fixes to tests in the suite which fail intermittently on various platforms. With more of these problems resolved, it gets easier and easier to determine if new proposed changes introduce any regressions.

Of course, I reviewed a lot of tickets over the last few weeks as well. Here they are:

#532 - Big jump from finger18.py to finger19.py in tutorial.
#1124 - getHost and getPeer not needed for IProcessTransport
#1157 - web.http.HTTPClient does not support chunked transfer-encoding
#1262 - Document Twisted copyright policy
#1328 - People keep asking about _trial_temp
#1852 - Replace uses of @ivar in interfaces with z.i.Attribute
#1853 - Fix/eliminate @cvar in Interfaces.
#2026 - Edit the twisted.internet docstring to make contents clearer
#2500 - Add @since to the coding standard
#2514 - Documentation bug in the list of possible classes
#2555 - Documentation update: deferred results in PB
#2909 - Document XXX Comment Policy
#3045 - Coding standard should forbid the use of "foo %s" % notATuple
#3197 - IProcessTransport.pid is incompletely documented
#3236 - Remove the FAQ from doc/core/howto_ since it is a duplicate of the FAQ wiki page
#3254 - twisted.python.deprecate does the wrong thing for methods
#3315 - t.p.reflect.safe_repr includes the wrong traceback and misformats the return value
#3382 - coding-standard.xhtml mislinks to twisted.python.compat
#3414 - Coding standard refers to old tools
#3415 - Get rid of X{} references in docstrings
#3422 - Remove acceptance tests in coding standard
#3423 - Get rid of references to old admin tools
#3435 - XmlStream should be convenient to use in a server context where there should be one authenticator per connection
#3446 - Infinite loop/memory-usage in irc.split()

These tickets represent a lot of good work by other Twisted developers (and you can see there are a lot of documentation tickets here, almost all the work of Thijs!).

I also spent a bit of time cleaning up our issue tracker. When you've been keeping track of bugs and feature requests for more than five years, it's hard to avoid having a certain amount of invalid tickets pile up. It's a bit of a drag going through the issue tracker to try to find these, but it's well worth while (and I can't expect everything to be fun ;).

That's it for this time. I'll be back with another report in about a month.

Tuesday, August 5, 2008

Summary of July/August TSF Sponsored Twisted Development

I've just completed the third two-week period of TSF sponsored Twisted development.

Tickets I worked on during this period:

#686 - [TEST] startTLS is broken if there's already data in the outgoing buffer.
#966 - [PATCH] Add --umask option to twistd
#1200 - twisted.test.test_internet calls reactor.iterate()
#1493 - static File web module doesn't support byte ranges
#2276 - Changes to TwistedNames to make it support NAPTR records
#2338 - trial should handle concurrent usage in the same directory gracefully
#2753 - twisted.web WSGI support
#2790 - UDP Transport write() raises socket.error EWOULDBLOCK
#2845 - twisted.internet.thread._putResultInDeferred should be public
#2931 - FilePath.setContent writes and renames but does not sync
#3342 - twisted.names dns-spoofing vulnerability
#3347 - twisted.names dns-spoofing vulnerability (birthday paradox)
#3367 - twisted.python.lockfile.FilesystemLock.lock fails with EEXIST

Tickets I reviewed during this period:

#637 - Odd filenaming in tutorial/intro.xhtml
#638 - Allow overriding twistd's logging options
#689 - twistd man page needs a section on signals
#1246 - reactor.callWhenRunning is not in the Using Processes document
#1253 - Create index.xhtml files for non-core doc trees
#1490 - Allow twistd to "run" packages_ e.g. 'twistd run mypackage --port 8080'
#1821 - Turn deferredgenerator wiki page into howto
#1888 - Review man pages
#1971 - Links in SEE ALSO section of doc/lore/man/lore.1 are bogus
#2208 - Standardize on the Python shebang line
#2375 - these objects' docstrings are not proper epytext:
#2438 - Get rid of references to maintainer email addresses from code
#2607 - conch.checks.SSHPublicKeyDatabase calls os.seteuid/os.setegid even if it's not necessary
#3244 - runInteraction exceptions are swallowed if rollback fails
#3254 - twisted.python.deprecate does the wrong thing for methods
#3285 - Add ISUPPORT implementation for irc.py
#3332 - SSHAgent implementation_ unit tests
#3355 - t.application.app.AppProfiler handles options oddly
#3365 - a few more IRCClient docstrings
#3366 - add IRCClient.back()
#3377 - words.protocol - irc.py assumes nickchange to be successful before server ack

Allen Short also helped out with a few reviews this time. Thijs Triemstra, a new Twisted contributor, has also been very active lately working though documentation tickets and making great headway.

Twisted's open ticket count also fell below 1100 this week. For the last three months, for the first time ever, we've been able to consistently resolve more tickets than have been filed.

Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Thursday, July 10, 2008

June/July TSF Sponsored Development

The second (of what should end up being many) round of TSF sponsored development has just wrapped up. Like last time, this was two weeks of work by yours truly (with an afternoon of Glyph's time spent reviewing code to keep things moving). This time around there were fewer tickets waiting to be reviewed when I got started, however there was still plenty of code to look at. Over the full period, I reviewed:

#1144 Documentation: twisted.internet.reactor does not appear in the API docs
#1253 Create index.xhtml files for non-core doc trees
#1255 Update copyrights in the man pages
#1878 twisted.web.monitor traceback_ AttributeError: class IChangeNotified has no attribute '__class__'
#1890 move examples from core into correct sub-packages
#1900 Error in documentation online
#2169 twisted.plugin documentation errors
#2208 Standardize on the Python shebang line
#2438 Get rid of references to maintainer email addresses from code
#2552 broken links in intro.xhtml
#2607 conch.checks.SSHPublicKeyDatabase calls os.seteuid/os.setegid even if it's not necessary
#2716 Eliminate relative imports from twisted.conch
#2815 Update VFS backends to an async interface
#2821 create twistd plugin for vfs
#2845 twisted.internet.thread._putResultInDeferred should be public
#3182 Trivial typo in twisted.internet.protocol
#3257 Rewrite twisted.web.static.File.directoryListing to not use woven
#3269 curses.setupterm must only be called once per process
#3300 twistd should support setting the syslog facility
#3326 Typo in gtkmanhole.py

Other tickets I worked on included:

#1069 log observers that throw exceptions should not be removed
#1152 xmlrpc.html doesn't describe how to return errors to the client
#1291 Expose "process exited" hook on ProcessProtocol_ turning current processEnded into user-overriddable behavior
#1493 static File web module doesn't support byte ranges
#2303 Deprecate setUpClass and tearDownClass and_ if possible_ fix the subclassing behaviour.
#2327 Intermittent failure in PB tests
#2631 Update coding standard to indicate preference for TestCase methods which being with "assert" and which do not have an underscore in their name
#2874 _sslverify.problemsFromTransport should be deprecated
#3029 documentation for twisted.python.deprecate.deprecated is incomplete (and other sundries)
#3059 twisted.internet.tcp.Client.getPeer incorrectly returns hostnames
#3116 Errors in processEnded can cause processes to be eternally reaped
#3159 t.i.utils process functions should have a default cwd of None_ not '.'
#3218 SSL disconnection sometimes hangs indefinitely with pyOpenSSL 0.7
#3255 Trial fails to display the line where the error occured in case of SyntaxError
#3300 twistd should support setting the syslog facility
#3301 superfluous local in AMP.__init__
#3305 CR IAC ignored in TelnetClient
#3306 twisted.test.test_ssl.StolenTCPTestCase has a number of defects
#3339 mailmail raises an exception instead of giving an error message

In particular, I spent a lot of time on #1291 which will make it possible to control child processes more precisely with Twisted by separating out the notification that a child process has actually exited from the notification that all of its file descriptors have been closed.

Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Wednesday, July 2, 2008

Twisted in the News

Everyone's Hiring Twisted Talent

The jobs category made it to the top of the list this time around, with folks like Rackspace hiring Knowers in the Ways of Twist. This automation engineer position puts Twisted on par with .Net and C#, giving it title space. This one is looking for talent working with custom protocols and mobile device products. Fluidinfo is also looking for some Twisted talent; if you're interested, email me and I'll put you in touch with them.

The Rackspace job posting is interesting, because a reliable source said that they are doing some very sexy stuff with Twisted and cloud computing. This would be quite the catch, job hunters...

The Fredericksburg Twist

Zope Corp recently released zc.twist 1.3, "Talking to the ZODB in Twisted Reactor Calls." Their page on PyPI has extensive documentation and some great examples. With their Zope 3 work, Zope Corp has consistently provided some of the best technical documentation I've seen in the Python world (possibly fueled by their excellent use of doctests). This release makes me want to jump back into the ZODB :-)

Related to the ZODB, be sure to read Martijn Faassen's recent post.


Terry Jones referred to the Twisted crew nicely in his blog post, saying that the framework is "an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers." Well, we're starting to get grey hairs, bald spots, and love handles... so perhaps we're not quite as young as we used to be ;-) But we'll take the compliment anyway, Terry ;-)

Holy Fervor

Colin Alston had some interesting things to say about "religious" technology debates and managed to nicely support Twisted and Divmod's Nevow at the same time. I'm saddened when people try to debate with us about the "right way" to do network programming (right now, everyone's asking how we compare to Erlang). This isn't a jihad, folks. Colin said it very well:
Why is it though that people become so fanatical about a particular framework or language? At the end of the day our choices of language and framework should be based on merit and not on emotion.
Colin also highlighted a few of the many reasons I like to use Twisted and Nevow:
Once you can break this mould, the endless possibilities of web applications become far more apparent. Twisted and Nevow gives you a clean web server tailored for your application, and you can keep your own state information and event control without needing the client to hit on special pages or abusing AJAX to poll resources.

Next up is Glyph's absolutely fantastic essay on scaling with Twisted and Mantissa. JP and I have actually been talking to him about this stuff in great detail lately, due to the messaging work that Glyph is doing. It's a delight that Glyph took the time to put his thoughts and analysis on "paper." There's too much good stuff in there to quote it; you'll have to read the whole thing.

Intermediate to Advanced

Glyph gets the spotlight again with his response to a recent critique of Twisted by a programmer who prefers to write threaded applications. This is a must-read for intermediate to advanced Twisters.

In a similar vein, folks wanting to push their Twisted skills further and do some integration would do well to read this post, which got upmodded at reddit.com last week. It covers some basic advice on creating Twisted-ready async code.

More Blog Bits

JP sketched out a quick prototype of TCP in the browser using Nevow/Athena. The PyPy team has Nevow running on PyPy now! Prasanna Gautam has a quick example of sending binary data over XML-RPC using Twisted. Jack Moffitt of Chesspark fame blogged about their new chat product (based on Twisted) called speeqe. Jack gives us another node in his earlier Firefox GTD blog post. There was a post by Duncan covering some examples of "batching" deferreds in Twisted. Glyph posted an interesting comment on Steve Holden's blog. And finally, Jonathan Lange gives a shout out to the crew on his post about code reviews.

Thursday, June 12, 2008

Twisted in the News

Love our Monkey

Two great posts from the Chesspark (and WuChess!) guys:

Oh, and did I mention Twisted-based WuChess was released?

Contracts and Jobs

Looks like someone wants some Twisted development done on LDAP and CIFS.

France's Olfeo and the U.K.'s Isotoma also looking for Twisted talent (among other skill sets).


Travis Branham is using Twisted in a CS/CE university internship:
The project revolves around mapping the uncertainty in position of moving objects. If an object is moving, and providing updates for its position at discrete intervals, it can sometimes be useful to know where that object could be between those updates.
He's written a custom SMTP and HTTP server for this.


Holden Web and Divmod are providing an introductory Twisted seminar for executives, managers, and technical directors in Washington, D.C. this Summer.

Monday, June 9, 2008

TSF Sponsored Development

Hello all,

On Friday I finished the first two weeks of funded Twisted development, I'm happy to say the two weeks went great and I got a lot accomplished.

The very, very high-level summary is that I started off by emptying the Twisted review queue and then moved on to finishing some twisted.web tasks which had been started previously by other developers and then abandoned for lack of time or interest or whatever other reason. All told, I did 35 reviews on 29 tickets and did development on 8 others. I also spent some time cleaning duplicate or invalid tickets out of the Twisted tracker. This collectively resulted in 32 tickets being closed over the past two weeks. Here are the tickets closed as a direct result of this work:

#1789 - twisted.conch.test.test_filetransfer.TestOurServerOurClient.testOpenFileAttributes (and perhaps others) leaks file descriptors
#3213 - twisted.conch.ssh.filetransfer.FileTransferServer leaks file descriptors
#3203 - PortableGtkReactor is broken
#3098 - filepath.FilePath.walk loops forever if there is a symlink pointing back up the file tree
#3033 - mkdir(a); mkdir(a) logs an unhandled error in SFTP
#3054 - Small error in the documentation for pb
#2881 - Add a variable in twisted.python.runtime to check for vista
#2374 - testOpenSSLBuffering fails intermittently on a dapper machine (running in vmware) because it uses reactor.iterate
#2918 - intermittent twisted.test.test_ssl.SpammyTLSTestCase.testUnTLS failure on OS X
#3210 - test_pb.NewCredTestCase is insanely long and causes some intermittent failures
#3119 - trial leaks memory/objects/something
#3052 - twistd logging options needs to be tested
#3266 - Provide tools for managing new deprecation policy
#3191 - in manhole in gnome-terminal_ lines longer than the terminal is wide result in excessive scrolling and duplication
#3105 - symlink support and docstrings in filepath.FilePath.copyTo/moveTo
#3267 - Typo in the Asynchronous Programming howto
#3184 - Trial minimal reporter prints a negative run time in summary
#1410 - spawnProcess on unix doesn't handle case where os.fork fails cleanly
#2305 - twisted/internet/process.py leaks file descriptors when os.fork() raises
#3272 - sorting the --profile output in trial
#3259 - Improved some error messagge in reflect.py
#2713 - trial -u can fail doctests on second pass
#2698 - Build Bug on Cygwin
#914 - xmlrpc.QueryFactory failed to catch some exceptions
#1057 - Incorrect and misleading arg name msSinceEpoch
#2017 - t.w.client.getPage returning ConnectionDone: Connection was closed cleanly error
#1412 - ExpatError on xmlrpc client connection close
#393 - [PATCH] HTTP Basic Auth based guard
#1108 - web2 needs SOAP support
#909 - microdom getElementsByTagName
#687 - web.client needs documentation and/or data-checking for method argument
#165 - [PATCH]Multiple header with same name support in http

Of particular note, I think, are the following:

#3213: Prior to this fix, the Twisted SFTP server would leak any file descriptors not closed before the end of an SFTP session.

#3203: This fix restores Gtk2 support on Windows. We have also set up a Gtk2 Windows builder which is now passing all but one test.

#1410: Prior to this fix, an error when calling os.fork() would cause reactor.spawnProcess to leak any pipes it opened and potentially to continue running in the main process with the wrong UID/GID.

#165: This adds support for multiple values for each header to the twisted.web server. Previously only the last value received would be made available to applications.

Six more tickets are waiting to be reviewed and will hopefully be closed soon. I want to take this opportunity to thank everyone who worked on tickets I reviewed over the past couple weeks. Twisted development is a collaborative process and I wouldn't have made nearly as big a dent without all of their help. I also want to thank Thomas Hervé in particular, who authored a lot of the code I reviewed and also (if I kept track properly) did every review of code I submitted over the the past two weeks.

In addition to doing work on tickets from the tracker, I also addressed two other issues with Twisted's buildbot. First, I took one of the compact build result views and split it into a supported-only view and an everything view to make it easier to find out the status of recent builds on supported platforms. These two views are linked from the main Twisted buildbot page, http://buildbot.twistedmatrix.com/ - see the "Latest builds" link. Second, I improved the error reporting for the setuptools-enabled builder, making it easier to track down the cause of some setuptools-related problems.

Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible.


Wednesday, June 4, 2008

A Place for Everyone

After an intense volley on the Twisted mail list, phone conversations, IRC discussions, and lots of thinking by everyone, we are pleased to announce the latest addition to Twisted: Twisted Community Code.

This is what Launchpad calls a "super project" or "project group" but what it means for the Twisted community is that anyone with Twisted code can play. Some details are up on a dedicated wiki page, with more info, help, and documentation to come.

Over the years, many have complained that the Twisted core developers are elitist perfectionists who make it impossible for anyone to contribute code. Most of us would object to such a sweeping statement, but we can't deny that there is at least a little truth in that complaint. The ultimate motivation for our standards and adherence to them is quality. We want Twisted to be as stable, useful, and maintainable as possible. However, quality always comes with a cost, and we're tired of paying it.

What about those who are just learning Twisted? Who want to jump in and contribute some code without having to spend months learning not only Twisted itself, but also its coding standard and our famous review process? Do we have a place in the community for them? Now, the answer to that is "yes" :-)

And this isn't just a dumping ground for newbs; within a day of creating this "super project" on Launchpad -- and with no public announacement -- we had three projects join (today there are five), all by serious long-time Twisted hackers. We expect that folks will contribute everything from excellent to horrible code: we encourage all to do so. Through such things as participation in the community, conversations, and voluntary peer review, good code will get better (and may be merged into Twisted proper) and bad code will learn to mend its evil ways. We have wanted to provide just such a space where everyone in the Twisted community can participate with what they love. Thanks to Canonical's and Launchpad platform, we're starting to do this.

You might notice that the "Twisted Community Code" lives at the "tx" URL. This came up in the email discussions mentioned above. Also, some of the projects are prefixed with "tx." The use of this term is likened to the "py" that is used for general Python projects and we encourange those creating Twisted projects to use the analogous "tx." However, Glyph said it well:
The whole point here is that it's a suggestion and a convention, not a mandate from the core team like so many things in the Twistedverse.
Phil Christensen has graciously agreed to take a leadership role in the management of this online community, and we give him many hearty pirate cheers for this. If you have any questions, feel free to send an email to the Twisted mail list... you'll likely get a quick response from Phil :-)

We're hoping that both new Twisted projects and those that have felt alienated in the past can find a home on Launchpad with Twisted Community Code Team Alpha Super Awesome Cool Dynamite Wolf Squadron.

Tuesday, May 20, 2008

ITA, The Twisted Foundation Rockstar

This is amazing: the Software Freedom Conservancy has just received a Twisted sponsorship check from ITA... and it's a big one: $20,000!

ITA is our first Diamond-level sponsor!

We are deeply grateful and excited that our project means so much to people and businesses.

We expect to be able to share some good news about Google, Canonical, and Microsoft in the next week or two, so stay tuned :-)

Saturday, May 17, 2008

Twisted in the News

More Thoughts on Concurrency

The folks at isotoma took some time to thoughtfully respond to my inquiry for more information on the areas where they felt that Twisted was lacking in support of concurrency. This was a fabulous read. Really well done with some hilarious Steve Yegge quotes. I encourage everyone to check it out (but I won't spoil the ending :-) For related reading, you might want to check out these PDFs (thanks Allen Short!):

Software Releases

The open source world has seen some Twisted-related software releases:


We got a mention in Jon Resig's blog post about Mozilla Build and Test Integration as one of the projects that uses Buildbot. Sadly, he failed to mention that Buidlbot itself is built with Twisted ;-)


Zenoss, a Founding Sponsor of the Twisted Software Foundation, has issues a press release about their support of Twisted. Additionally, we're now sponsored by three subsidiaries (Contentinople, Internet Evolution, and Light Reading) of the publishing megacorp UBM (formerly CMP).


Divmod has a new web site up, where they announce that they're officially providing specialized Twisted services in the form of consulting (full disclosure: I work for Divmod).

Friday, May 9, 2008

Zenoss' Twisted Press Release

Get it while it's hot: Zenoss Announces Membership in the TSF.

It's exciting to see companies supporting Twisted like this; increasing exposure to the community's efforts is one of the best things that can be done by organizations using Twisted.

Thanks Zenoss!

Technorati Tags: , ,

Tuesday, May 6, 2008

Twisted in the News

An Inflection Point

In a recent blog post, the London-based Isotoma consulting firm gave Twisted a mention. Their post was a commentary on Tim Bray's Multi-Inflection-Point Alert, a concise run-down on what's going on with internet tech, as he puts, "right now."

Isotoma had this to say:
"We’re a lot further down this particular inflection curve than most, I think. We make heavy use of Twisted, a single-threaded cooperatively multitasking network programming system that specifically addresses the threading problem."
They went on to mention that it doesn't seem like a complete solution. I'd love to hear their comments on where they find it lacking, what they'd like to see supported -- either specific features or general thoughts on the future of computing in the consulting industry.

Imitation and Flattery

Ruby has something called EventMachine and was described as implementing the "Twisted-introduced" deferred pattern in a nutrun post.

Mashed and Twisted?

Twisted and Nevow made a mashup list on InnovationStartups' blog of 120+ Web Development Resources. Also on the list were launchpad, rBuilder, Ohloh, Ruby on Rails, MochiKit, Django and Zope.

Scratch that Itch!

Jesse Noller blogs about Bruce Eckle's latest Twisted mention, saying "Dangit Bruce gave me the twisted itch again."

Bruce Eckle talks about Concurrency with Python, Twisted, and Flex in his Artima Developer post, where he also mentions PyAMF (which I helped update the Twisted docs for :-)). Bruce takes a quick look at asynchronous programming, object brokering with Twisted, and XML-RPC. I wish he'd used twisted.application.service instead of running the reactor directly, but hey -- it's a great Twisted post about stuff that lots of people are using right now :-)

New Users

Paul Stevens blogged about setting up Twisted on his mac. He's now off to the races :-) Good luck, Paul! Be sure to stop by IRC and/or post messages on the mail list with questions...

Technorati Tags: , , , , , , ,

Wednesday, April 23, 2008

The Twisted Show: Episode 3

In our third installment, we have interviewed Synthesis Studios of Boston, MA. Co-founder Raffi Krikorian gave a fabulous interview with some phenomenal Twisted sound bites. We'll definitely be quoting him!

One of my favorite bits was the value Raffi sees in Twisted as a tool for improving the actual mind of the software developer; in particular, finding "outside the box" solutions to challenging problems.

Check out the interview from either the wiki page or the Twisted Show blog...

Technorati Tags: , , , ,

April 20th Sprint Report

Hello all,

The first TSF sponsored Twisted sprint is done. Here's a quick rundown of some of the things that happened (this is the same summary as I sent to the mailing list so you can skip it if you already read it there).
  • Glyph Lefkowitz made a huge dent in the current review queue, dealing with more than half of the outstanding tickets.
  • After some discussion about the value of a bugfix-only release vs a normal bugfix and feature enhancement release, Christopher Armstrong got the wheels rolling on 8.1.0 (which will be of the latter type). You probably already saw the pre1 announcement. Give it a spin!
  • Paul Swartz continued to strive towards the goal of Twisted Conch being as soft and cuddly as a little bunny rabbit.
  • Ying Li worked on permission and ownership features for FilePath.
  • Jeff Mancuso came by for a while and talked about Twisted Conch performance (and did some benchmarks) and SFTP access control issues.
  • Roland Hedberg got his hands dirty in pyOpenSSL working on adding an API to get arbitrary X509v3 extensions.
  • Itamar Shtull-Trauring helped out all around.
  • I finished fixing some reactor startup and shutdown issues introduced in 8.0.
We also discussed Twisted's backwards compatibility policy, guided mostly by Jonathan Lange's email of the 18th to this list, with the goal of creating an official document.

The next sprint is tentatively scheduled for May 18th at the Divmod office.

Tuesday, April 15, 2008

Founding Sponsors Window Closes May 15th

Current Founding Sponsors: You guys rock! Thanks!

Potential Founding sponsors: You too can rock, but you only have a limited window of opportunity in which to do so!

We were going to set the deadline to April 30th, but we've been so busy working with those of you that have been sponsoring that we didn't get the chance (just look at those awesome logos on the front page!) to send out an email. We wanted to provide people and companies with a 30 day notice, so May 15th it is.

Thanks again, everyone -- you are helping make Twisted a thriving project that gives you more and more of what you need.

Originally posted on the Twisted Python mail list.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

Technorati Tags: , , ,

Thursday, March 27, 2008

Twisted 8.0 released!

MASSACHUSETTS (DP) -- Version 8.0 of the Twisted networking framework has been released, Twisted Matrix Laboratories announced Wednesday.

Enslaved by his new robotic overlords, Master of the Release Christopher Armstrong presented the new package to the Internet on March 26th. Armstrong was unable to comment, because of a device worn around his neck preventing him from doing so, scientists say.

Secretary of Defense Jean-Paul Calderone was asked about concerns that French interference may have played a role in the delay of this release. "I find such speculation preposterous. Thomas Hervé is an upstanding member of the Labs and his loyalties lie with us. He is a fine addition to our team." Rumors in the community allege that Secretary Calderone is holding Hervé's cat ransom until the release is successfully distributed. Hervé was unavailable for comment.

This release comes shortly after the announcement by Chief of Public Affairs Duncan McGreggor that Twisted had joined the Software Freedom Conservancy. "We're happy to join the SFC, and we are now accepting sponsorship. The fact that we are now ruled by a cabal of robots should not put off potential donors. Our robotic overlords are running us at peak efficiency, so we can most effectively distribute The Love."

Asked about the version number jump in this release, Commander-in-Chief Glyph Lefkowitz had the following to say: "Our benefactors have found our previous dice-rolling version number scheme to be inadequate, and have deigned to propose to us a more... logical system of versioning."

Twisted 8.0 is a major feature release, with several new features and a great number of bug fixes. Some of the highlights follow.
  • The IOCP reactor is now much improved and many bugs have been resolved.
  • Twisted is now easy_installable.
  • Many improvements were made to Trial, Twisted's unit testing system.
  • A new memcache client protocol implementation was added.
  • So much more!
To see the full list of changes in its fifteen kilobytes of glory, see the release notes. We welcome you to download and enjoy, and please file any bugs you find and send comments to the mailing list.

Why the large version number bump? We've decided to switch to a time-based versioning scheme. "8.0" means the first release in 2008.


Sunday, March 16, 2008

Twisted Announces Membership in the Software Freedom Conservancy

At PyCon 2008 we were delighted to share some good news with the Python community: the announcement of the Twisted Software Foundation. Glyph had previously emailed the Twisted mail list, letting those close to the project know of this major event shortly after the paperwork was signed. Before we brought this before a more public audience, we wanted to give companies a chance to become "founding sponsors" -- and this is what has happened at PyCon. We're looking forward to issuing a formal press release about this (including info on the founding sponsors) and will have more details in the near future.

We would like to send a special thank-you out to those sponsors who made contributions immediately. In order of contribution, they are:

  • Chris Armstrong
  • Michel Pelletier
  • AdytumSolutions, Inc.
  • tummy.com
  • Drew Perttula
  • Glenn Tarbox
  • Ilya Novoselov

Read the initial announcement here. If the video for the announcement made at PyCon gets published somewhere, we will post an update.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

Tuesday, February 26, 2008

The Twisted Show: Episode 2

Yesterday I had the chance to talk with Jack Moffitt and Christopher Zorn of Chesspark about their company, their vision, and how Twisted played an integral role in their application and helped them to meet their goals.

The interview is on the site and the feed is available here.

We're still experimenting with podcasting and feed enclosures, so please be patient with us! If you have any issues, the media files are available at archive.org.

Monday, February 25, 2008

Twisted Talks at PyCon 2008

There will be two Twisted-related talks at PyCon this year:

"Tahoe: A Robust Distributed Secure Filesystem" -- Brian Warner, of Twisted Foolscap fame, discusses his company's distributed file system, built on Twisted. Tahoe is free, secure, decentralized, fault-tolerant filesystem. This filesystem is encrypted and spread over multiple peers in such a way that it remains available even when some of the peers are unavailable, malfunctioning, or malicious.

"Adventures in Stackless Python/Twisted Integration" -- Andrew Francis will be discussing the Web Service Business Process Execution Language, particularly he integrated Stackless Python and Twisted.

Technorati Tags: , ,

Wednesday, February 20, 2008

RSS Flood Apology

Hey all,

A quick note to apologize for the RSS updates you'll be getting that are full of old news. I just finished sync'ing Twisted news to the labs blog (and tagging them with "news"). There weren't many items, so the pain of the RSS flood should be minimal. I offer apologies nonetheless :-)

The upshot of all this is that we now have a couple different RSS feeds for your viewing pleasure:
An RSS feed for our news items has been a long-standing request of the community, and we're glad to be able to finally offer this.

You can also view the news items in either of these three places:
You can expect more news, howtos, theses and other fascinating textual content on labs.twistedmatrix.com. Be sure to visit us often for your Twisted fix :-)

Sunday, February 17, 2008

Twisted's filepath Module

Glyph recently blogged about some of the buried treasure in Twisted, the filepath module in particular. Since working at Divmod, I've made use of this module quite a bit, and thought I'd share some of the features that I've found most useful (and intuitively nice to use).

Assuming you've got Twisted installed, let's fire up a python interpreter and import the FilePath class and instantiate it with /tmp:

>>> from twisted.python.filepath import FilePath
>>> tmp = FilePath('/tmp')

Now let's test some basic operations:

>>> tmp.exists()
>>> bogus = tmp.child('bogus')
>>> bogus.exists()

Isn't that nice? I love not having to import and use the os.path.exists function; it's a method on the object representing a file or a path, as it should be. I also enjoy the convenience FilePath offers when it comes to creating paths:

>>> mydir = tmp.child('notbogus').child('anotherdir').child('mydir')
>>> mydir.exists()
>>> mydir.makedirs()
>>> mydir.restat()
>>> mydir.exists()

We had to call restat() so that the object would check the file system again (since we just made some changes). Now, for some files:

>>> for filename in ['test1.txt', 'test2.txt', 'test3.tst']:
... mydir.child(filename).touch()
>>> mydir.listdir()
['test1.txt', 'test2.txt', 'test3.tst']

And if you don't believe that, we can switch to shell:

lorien:oubiwann 20:51:58 $ ls -al /tmp/notbogus/anotherdir/mydir/
total 0
drwxr-xr-x 2 oubiwann wheel 170 Feb 17 20:51 .
drwxr-xr-x 3 oubiwann wheel 102 Feb 17 20:46 ..
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test1.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test2.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test3.tst

Reading and writing operations are the same as usual:

>>> myfile = mydir.child(filename)
>>> myfile.isdir()
>>> myfile.islink()
>>> myfile.isfile()
>>> fh = myfile.open('w+')
>>> fh.write('do the usual thing')
>>> fh.close()
>>> myfile.getsize()
>>> myfile.open().read()
'do the usual thing'

But you can use shortcuts like this, too:

>>> myfile.getContent()
'do the usual thing'
>>> myfile.setContent('and now for somethibg completely different...')
>>> myfile.getContent()
'and now for somethibg completely different...'

Let's get the path and recheck the file size, just to make sure:

>>> myfile.path
>>> myfile.restat()
>>> myfile.getsize()

Or, again from shell:

lorien:oubiwann 20:52:06 $ ls -al /tmp/notbogus/anotherdir/mydir/
total 8
drwxr-xr-x 2 oubiwann wheel 170 Feb 17 20:58 .
drwxr-xr-x 3 oubiwann wheel 102 Feb 17 20:46 ..
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test1.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test2.txt
-rw-r--r-- 1 oubiwann wheel 45 Feb 17 20:58 test3.tst

Hmm... for some reason, I really like this file and want to keep it. What do I do?

>>> docs = FilePath('/Users/oubiwann/Documents')
>>> myfile.moveTo(docs.child('special_file.txt'))

That's it! Nice, eh? Of course, we're careful, so we check to make sure it happened:

>>> myfile.exists()
>>> newfile = docs.child('special_file.txt')
>>> newfile.exists()
>>> newfile.path
>>> newfile.getsize()
>>> newfile.getContent()
'and now for somethibg completely different...'

To see more, check out the source, or at least do dir(FilePath) to see what other goodies this class has to offer, and enjoy :-)

Technorati Tags: , ,

Wednesday, February 13, 2008

Simple Python Web Server

Corey Goldberg blogged about making simple web interfaces. I was going to comment with another example of a simple web server, but his blog kept spitting back errors, so I'll post it here instead.

Corey's code was based on BaseHTTPServer, here's another way to do this (as a Twisted tac file):

from twisted.web import server, resource
from twisted.application import internet, service

class Foo(resource.Resource):
def render(self, request):
return "hello world"

root = resource.Resource()
root.putChild("foo", Foo())

application = service.Application("Tiny Web")
internet.TCPServer(8080, server.Site(root)).setServiceParent(application)

Of course, it requires a bit more than just the standard library. ;) This has a couple extra nice features though - it does logging, for example, and it can be trivially daemonized, or run under the profiler or the debugger, or chrooted or as a different uid/gid. Plus the Foo resource can easily be relocated into a "real" program, since it is entirely distinct from the HTTP server part of the code.

There's an even simpler way to express this though, as an rpy instead of a tac:

from twisted.web.resource import Resource

class Foo(Resource):
def render(self, request):
return "hello world"

resource = Foo()

Just drop this into a directory and run a Twisted web server pointed at it (eg, twistd web --processor rpy=twisted.web.script.ResourceScript --path bar/). You can even change the code without restarting the server if you do it this way. rpys can be convenient for one-off hacks, but clearly it's not suitable for anything larger scale than that.