Fluidinfo sponsors Twisted!

Fluidinfo is Twisted's latest sponsor!  We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

    – Esteve Fernandez

       CTO, Fluidinfo Inc. 

Sponsored Development, July 2010

Hello readers,

It's that time once again. I've just spent a very productive two weeks dedicated to Twisted developments, thanks to the support of the Twisted sponsors and the Software Freedom Conservancy.

There were quite a few tickets to review when I got started this time. Here's the list (italicised tickets are now resolved):

#411 - Returning a Deferred from the callback of another Deferred too many times results a RecursionError

#1384 - Document the meaning of trial's terminal output stanza

#1702 - Kill Defer.setTimeout

#2485 - 2to3 cross-compilation tools

#2661 - Deprecate IFinishableConsumer so it can be deleted

#2680 - abdapi ConnectionPool doesn't allow for synchronous close

#3546 - Add custom time zone support to twisted.log.FileLogObserver

#3586 - I want to install twisted without a c compiler

#3802 - win32 SerialPort requires a reactor that provides "addEvent" method, but doesn't have a good error message if it gets a different one

#4038 - DeprecationWarning: the MimeWriter module is deprecated; use the email package instead

#4045 - Twisted's Failure fake '_Frame' object lacks f_locals

#4138 - A fresh Twisted checkout should support "setup.py sdist"

#4299 - Remove deprecated twisted.web.trp

#4328 - Deprecate twisted.python.text.docstringLStrip

#4378 - Error handling in NetstringReceiver broken

#4419 - IRCClient.noticed() default is prone to causing loops

#4477 - UNIX ports fail to log their path when stopped

#4511 - trial(1) man page says "I don't know why this is in trial." for --disablegc

#4519 - Delay parsing of request body until Request.args is accessed

#4520 - pb.CopiedFailure.throwExceptionIntoGenerator breaks in Python 2.6.

#4522 - Speed up `reactor.spawnProcess()` by only closing open file descriptors

#4527 - Change either documentation or code of t.i.test.reactormixins.ReactorBuilder.requiredInterface

#4531 - Static analysis of module attributes in twisted.python.modules

#4536 - Credentials materials are compared unsafely throughout Twisted

#4540 - t.n.hosts.searchFileFor does not close the file

#4555 - Replace loop-like uses of map() with a loop

#4566 - Improve lore2sphinx buildbot results for `projects/conch`

#4567 - Improve lore2sphinx buildbot results for `projects/core/development`

Additionally, I had lots of time to spend worked on other tickets:

#526 - [PATCH] make t.w.microdom respect case sensitivity

#989 - Default log observer chokes on unicode messages

#1784 - disttrial --hosts=kunai,takkun,muon twisted

#2179 - Parameterize the root directory of the FTP server for non-anonymous logins

#3169 - Add FilePath.descendant(segmentsToChild) to replace fp.child(a).child(b).child(c)

#3242 - use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available

#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented

#3690 - SerialPort never calls connectionLost(reason)

#4300 - Write some unit tests for Deferreds

#4473 - strports.endpoint

#4487 - report multiple tests with the same results at the same time

#4492 - deprecatedModuleAttribute emits warning twice when deprecated module is imported

#4494 - FTPRealm home directory code hardcodes '/home'

#4500 - Use Sphinx for Twisted Documentation

#4501 - trial's error reporting (especially --tbformat=emacs) does not like unicode exceptions

#4503 - domish gets confused by spaces in xmlns names

#4504 - Document reactor.listenMulticast's listenMultiple keyword argument

#4513 - twisted.names server sends incorrect authority section

#4517 - documentation for blockingCallFromThread doesn't exactly say the given callable can return a Deferred

#4528 - t.news.database should use t.m.smtp.sendmail and the email package

#4534 - Update twisted.application.service.IProcess to use Attributes

#4535 - zippath.child('..') does not raise InsecurePath

#4536 - Credentials materials are compared unsafely throughout Twisted

#4539 - fileno() gets called on a ReadDescriptor after removing it from the reactor

#4542 - `LineReceiver` API documentation should indicate when delimiter is stripped etc

#4556 - KQueue reactor misspelled in reactorbuilder

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit http://twistedmatrix.com/trac/ticket/ticketnumber. For example, http://twistedmatrix.com/trac/ticket/4487

Many of these which are still open are now up for review and should be closed over the next few weeks as other developers have a chance to take a look and provide feedback.

A careful reading of this list reveals a few tickets related to our attempts to convert Twisted's documentation from Lore to Sphinx, an undertaking you already know about if you follow the Twisted mailing list. Kevin Horn has spearheaded this project and made great progress towards the goal. Even though the conversion isn't complete yet, there's already some really great looking results.

Twisted 10.1.0 released

Twisted 10.1.0 is finally out. It's a month late, but it's finally out there and free from the shackles of British rule.

Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

Note also that this is the last release in which we will support Windows for Python 2.4.

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download it the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Update: We now have executable Windows installers for Python 2.5 and Python 2.6 to complement the MSIs.

Many thanks to Glyph Lefkowitz, who helped prepare the release, and the PyCon 2010 sprinters, who did so much of the work for this release.

Twisted 10.1.0pre1 released – please test

From the dank and grey depths of a London summer, a thin, reedy voice wanders across the waves of the Internet, saying that it's jolly well pleased to announce the first 10.1.0 pre-release.

We would very much appreciate it if you could download the tarballs, test them on your systems, and file tickets as appropriate.


Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

For more information, see the NEWS file.

If there is only positive feedback from the pre-release, we aim to do the final release some time later this month.

Powering the Silver Screen

Think Twisted is only good for Internet-facing servers? Even then, it's not like Python itself that touches every part of your day in some way, right? Well my friend, you'd be delightfully wrong.

When you watch a movie crediting Lucasfilm for special effects, Twisted could be behind the scenes. Since 2004, Lucasfilm has used Twisted as part of its rendering operations to proxy database connectivity in the render farm (and much more), 24/7 x 365. That's right my friends, Twisted is not only the engine of your Internet...it might have a hand in your entertainment too.

To read more check out the great Success Story from Dave Peticolas from Lucasfilm: Twisted at Lucasfilm

Have your own success story using Twisted? Let us know! Just send an e-mail to success@twistedmatrix.com

Twisted Git (and Bzr) Mirror

With the multitude of distributed version control systems now available, it's getting pretty hard for any open source project to get by on a centralized system. Distributed systems like bzr, mercurial, or git give equal tools to all contributors, not just a select few trusted to vette changes for inclusion in an official branch. Forcing new contributors to work with diff and patch is just plain mean!

So, while Twisted's source is still hosted in subversion, and likely will be for some time to come (due to various toolchain requirements), I'm happy to announce that there is now a git mirror to complement the existing bzr mirror.

With these mirrors available, developers who don't have commit access to the subversion server can still work with Twisted using modern revision control tools - branching, merging, history, and all the rest.

The workflow for merging changes based on the bzr or git mirror back into Twisted trunk isn't yet completely worked out. Likely it won't be until people start using these and trying to push changes back to us. So if you're a fan of one of these two systems, please, start using these mirrors and help us figure out the best way to integrate things back into subversion.

Twisted test suite code coverage

As anyone who's contributed to Twisted knows, we currently have a very high standard for code changes and additions. For a long time, we've relied mainly on the standard library trace module. The trace module is quite serviceable, but recently some other options have come along which are much better. In particular, coverage.py seems to provide enough additional features that it's actually worth integrating into Twisted's development process. For example, it has some basic branch coverage features (trace will show you what lines were executed; coverage.py will show you this, plus tell you whether both possible outcomes of your "if" statement were taken), and can generate fairly nice HTML reports.

So, as a first step, I've set up a coverage.py builder on Twisted's BuildBot. This means that code coverage reports are part of Twisted's continuous integration system now: for every trunk revision, new coverage reports will be generated. At the moment, these reports end up jumbled up in the builds directory along with a lot of other stuff. For example, here's the latest report as of this post. Something I still want to do is clean up how these reports are organized so they're easier to find and compare.

As you can see from this report, Twisted has 86% line coverage and 67% branch coverage (which I had to compute myself from the totals at the bottom of the page - one of a few coverage.py nits that I hope will be fixed soon :). coverage.py decides this makes overall coverage 83%, though I'm not sure how exactly it arrives at that number.

That's pretty good, but there's plenty of room for improvement. I hope these reports will become an easy way for people to find areas of Twisted which need more coverage and motivate them to contribute new tests.

Oh, and since coverage.py only reports on Python coverage, I've also set up a gcov/lcov builder which uploads its coverage reports for execution of the C code in Twisted to a similar location.