Making chat hipper with Twisted!

HipChat has been changing what it means to communicate in teams of any size and taking the world by storm over the past 12 months. What you may not know is that Twisted is at the core of HipChat's highly successful service:

HipChat is a hosted group chat and IM service for businesses which uses the XMPP protocol behind the scenes. We use Twisted to power our fully clustered chat server and run many important job queue workers.

Bottom line: we built and launched a successful business on Twisted & Python in 6 months with no prior knowledge of either. It's a very productive environment with great performance and maintainability. What more could you want?

    – Garret Heaton, Co-founder, HipChat

To learn more about how HipChat uses Twisted check out the success story!

Twisted 10.2.0 Released!

Twisted 10.2.0, the third Twisted release of 2010, has emerged from the mysterious depths of Twisted Matrix Labs, as so many releases before it. Survivors of the release process - what few there were of them - have been heard to claim that this version is "awesome", "even more robust", "fun-sized" and "oven fresh".

Crossing several things that shouldn't ought to be, including the streams and the rubicon, I have assumed the triple responsibilities of feature author, project leader, and release manager for 10.2: with this dark and terrible power - a power which no man ought to wield alone - I have wrought a release which contains many exciting new features, including:

• A plug-in API for adding new types of endpoint descriptions.
• A new, simpler, substantially more robust CoreFoundation reactor.
• Improvements to the implementation of Deferred which should both improve performance and fix certain runtime errors with long callback chains.
• Deferred.setTimeout is (finally) gone.  To quote the author of this change: "A new era of peace has started."
• NetstringReceiver is substantially faster.

And, of course, nearly one hundred smaller bug fixes, documentation updates, and general improvements. See the NEWS file included in the release for more details.

Look upon our Twisted, ye mighty, and make your network applications event-driven: get it now, from:
   <http://twistedmatrix.com/>
... or simply install the 'Twisted' package from PyPI.

Many thanks to Christopher Armstrong, for his work on release-automation tools that made this possible; to Jonathan Lange, for thoroughly documenting the process and thereby making my ascent to the throne of release manager possible, and to Jean-Paul Calderone for his tireless maintenance of our build and test infrastructure as well as his help with the release.

Most of all, thanks to everyone who contributed a patch, reported a bug or reviewed a ticket for 10.2.  Not including those already thanked, there are 41 of you, so it would be a bit tedious to go through everyone, but you know who you are and we absolutely couldn't do it without you!  Thanks a ton!

Fluidinfo sponsors Twisted!

Fluidinfo is Twisted's latest sponsor!  We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

    – Esteve Fernandez

       CTO, Fluidinfo Inc. 

Sponsored Development, July 2010

Hello readers,

It's that time once again. I've just spent a very productive two weeks dedicated to Twisted developments, thanks to the support of the Twisted sponsors and the Software Freedom Conservancy.

There were quite a few tickets to review when I got started this time. Here's the list (italicised tickets are now resolved):

#411 - Returning a Deferred from the callback of another Deferred too many times results a RecursionError

#1384 - Document the meaning of trial's terminal output stanza

#1702 - Kill Defer.setTimeout

#2485 - 2to3 cross-compilation tools

#2661 - Deprecate IFinishableConsumer so it can be deleted

#2680 - abdapi ConnectionPool doesn't allow for synchronous close

#3546 - Add custom time zone support to twisted.log.FileLogObserver

#3586 - I want to install twisted without a c compiler

#3802 - win32 SerialPort requires a reactor that provides "addEvent" method, but doesn't have a good error message if it gets a different one

#4038 - DeprecationWarning: the MimeWriter module is deprecated; use the email package instead

#4045 - Twisted's Failure fake '_Frame' object lacks f_locals

#4138 - A fresh Twisted checkout should support "setup.py sdist"

#4299 - Remove deprecated twisted.web.trp

#4328 - Deprecate twisted.python.text.docstringLStrip

#4378 - Error handling in NetstringReceiver broken

#4419 - IRCClient.noticed() default is prone to causing loops

#4477 - UNIX ports fail to log their path when stopped

#4511 - trial(1) man page says "I don't know why this is in trial." for --disablegc

#4519 - Delay parsing of request body until Request.args is accessed

#4520 - pb.CopiedFailure.throwExceptionIntoGenerator breaks in Python 2.6.

#4522 - Speed up `reactor.spawnProcess()` by only closing open file descriptors

#4527 - Change either documentation or code of t.i.test.reactormixins.ReactorBuilder.requiredInterface

#4531 - Static analysis of module attributes in twisted.python.modules

#4536 - Credentials materials are compared unsafely throughout Twisted

#4540 - t.n.hosts.searchFileFor does not close the file

#4555 - Replace loop-like uses of map() with a loop

#4566 - Improve lore2sphinx buildbot results for `projects/conch`

#4567 - Improve lore2sphinx buildbot results for `projects/core/development`

Additionally, I had lots of time to spend worked on other tickets:

#526 - [PATCH] make t.w.microdom respect case sensitivity

#989 - Default log observer chokes on unicode messages

#1784 - disttrial --hosts=kunai,takkun,muon twisted

#2179 - Parameterize the root directory of the FTP server for non-anonymous logins

#3169 - Add FilePath.descendant(segmentsToChild) to replace fp.child(a).child(b).child(c)

#3242 - use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available

#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented

#3690 - SerialPort never calls connectionLost(reason)

#4300 - Write some unit tests for Deferreds

#4473 - strports.endpoint

#4487 - report multiple tests with the same results at the same time

#4492 - deprecatedModuleAttribute emits warning twice when deprecated module is imported

#4494 - FTPRealm home directory code hardcodes '/home'

#4500 - Use Sphinx for Twisted Documentation

#4501 - trial's error reporting (especially --tbformat=emacs) does not like unicode exceptions

#4503 - domish gets confused by spaces in xmlns names

#4504 - Document reactor.listenMulticast's listenMultiple keyword argument

#4513 - twisted.names server sends incorrect authority section

#4517 - documentation for blockingCallFromThread doesn't exactly say the given callable can return a Deferred

#4528 - t.news.database should use t.m.smtp.sendmail and the email package

#4534 - Update twisted.application.service.IProcess to use Attributes

#4535 - zippath.child('..') does not raise InsecurePath

#4536 - Credentials materials are compared unsafely throughout Twisted

#4539 - fileno() gets called on a ReadDescriptor after removing it from the reactor

#4542 - `LineReceiver` API documentation should indicate when delimiter is stripped etc

#4556 - KQueue reactor misspelled in reactorbuilder

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit http://twistedmatrix.com/trac/ticket/ticketnumber. For example, http://twistedmatrix.com/trac/ticket/4487

Many of these which are still open are now up for review and should be closed over the next few weeks as other developers have a chance to take a look and provide feedback.

A careful reading of this list reveals a few tickets related to our attempts to convert Twisted's documentation from Lore to Sphinx, an undertaking you already know about if you follow the Twisted mailing list. Kevin Horn has spearheaded this project and made great progress towards the goal. Even though the conversion isn't complete yet, there's already some really great looking results.

Twisted 10.1.0 released

Twisted 10.1.0 is finally out. It's a month late, but it's finally out there and free from the shackles of British rule.

Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

Note also that this is the last release in which we will support Windows for Python 2.4.

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download it the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Update: We now have executable Windows installers for Python 2.5 and Python 2.6 to complement the MSIs.

Many thanks to Glyph Lefkowitz, who helped prepare the release, and the PyCon 2010 sprinters, who did so much of the work for this release.

Twisted 10.1.0pre1 released – please test

From the dank and grey depths of a London summer, a thin, reedy voice wanders across the waves of the Internet, saying that it's jolly well pleased to announce the first 10.1.0 pre-release.

We would very much appreciate it if you could download the tarballs, test them on your systems, and file tickets as appropriate.


Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

For more information, see the NEWS file.

If there is only positive feedback from the pre-release, we aim to do the final release some time later this month.

Powering the Silver Screen

Think Twisted is only good for Internet-facing servers? Even then, it's not like Python itself that touches every part of your day in some way, right? Well my friend, you'd be delightfully wrong.

When you watch a movie crediting Lucasfilm for special effects, Twisted could be behind the scenes. Since 2004, Lucasfilm has used Twisted as part of its rendering operations to proxy database connectivity in the render farm (and much more), 24/7 x 365. That's right my friends, Twisted is not only the engine of your Internet...it might have a hand in your entertainment too.

To read more check out the great Success Story from Dave Peticolas from Lucasfilm: Twisted at Lucasfilm

Have your own success story using Twisted? Let us know! Just send an e-mail to success@twistedmatrix.com

Twisted Git (and Bzr) Mirror

With the multitude of distributed version control systems now available, it's getting pretty hard for any open source project to get by on a centralized system. Distributed systems like bzr, mercurial, or git give equal tools to all contributors, not just a select few trusted to vette changes for inclusion in an official branch. Forcing new contributors to work with diff and patch is just plain mean!

So, while Twisted's source is still hosted in subversion, and likely will be for some time to come (due to various toolchain requirements), I'm happy to announce that there is now a git mirror to complement the existing bzr mirror.

With these mirrors available, developers who don't have commit access to the subversion server can still work with Twisted using modern revision control tools - branching, merging, history, and all the rest.

The workflow for merging changes based on the bzr or git mirror back into Twisted trunk isn't yet completely worked out. Likely it won't be until people start using these and trying to push changes back to us. So if you're a fan of one of these two systems, please, start using these mirrors and help us figure out the best way to integrate things back into subversion.

Twisted test suite code coverage

As anyone who's contributed to Twisted knows, we currently have a very high standard for code changes and additions. For a long time, we've relied mainly on the standard library trace module. The trace module is quite serviceable, but recently some other options have come along which are much better. In particular, coverage.py seems to provide enough additional features that it's actually worth integrating into Twisted's development process. For example, it has some basic branch coverage features (trace will show you what lines were executed; coverage.py will show you this, plus tell you whether both possible outcomes of your "if" statement were taken), and can generate fairly nice HTML reports.

So, as a first step, I've set up a coverage.py builder on Twisted's BuildBot. This means that code coverage reports are part of Twisted's continuous integration system now: for every trunk revision, new coverage reports will be generated. At the moment, these reports end up jumbled up in the builds directory along with a lot of other stuff. For example, here's the latest report as of this post. Something I still want to do is clean up how these reports are organized so they're easier to find and compare.

As you can see from this report, Twisted has 86% line coverage and 67% branch coverage (which I had to compute myself from the totals at the bottom of the page - one of a few coverage.py nits that I hope will be fixed soon :). coverage.py decides this makes overall coverage 83%, though I'm not sure how exactly it arrives at that number.

That's pretty good, but there's plenty of room for improvement. I hope these reports will become an easy way for people to find areas of Twisted which need more coverage and motivate them to contribute new tests.

Oh, and since coverage.py only reports on Python coverage, I've also set up a gcov/lcov builder which uploads its coverage reports for execution of the C code in Twisted to a similar location.

New, New, New!

New Success Story

We've added a great success story for PowerCard, a loyalty rewards program for local restaurants. One of the terrific things about the PowerCard story is the use of Twisted for both back-end and client-facing systems. It's a great reminder that Twisted is as useful solving problems on Windows platforms as it is on the Linux/UNIX-based systems many of us code for.

Read the full success story here.

New Library Listing

Looking to use Twisted with new technologies like RabbitMQ (txAMQP) and Redis (txRedis)? Browse the updated libraries listing. We bet you'll find what you're looking for.

New Twisted Ambassador

And why do we have these new and updated listings for 3rd-party code? It's because we have a new guy who has volunteered to maintain and update the listings!

Jason J. W. Williams is Twisted's new ambassador to 3rd-party projects of all kinds.Primarily this responsibility involves keeping the "Success Stories" and "Projects Using Twisted" pages up to date, and occasionally writing small updates (like this one) for the news feed.

Please contact success at twisted matrix dot com if you have a success story about how you used Twisted, a pointer to some hitherto unknown code that uses Twisted, or for more information. And watch this space for more updates!

Barcelona Twisted Sprint

Yesterday's Twisted sprint at the Flumotion office was quite a success. There were six of us there, and many first time Twisted contributors.

Jan Urbański worked on fixing a couple IMAP4 client search bugs. He also spent some time using psycopg's asynchronous interface to create a very Twisted-friendly postgres API, with methods that return Deferreds, and all without using any threads.

Thomas Vander Stichele and Zaheer Merali worked on addressing ambiguities on the AMP API documentation and adding support for dates and decimals to AMP.

Later Zaheer switched to pairing with Sebastien Merle who was working on adding string streaming to AMP. Meanwhile Thomas worked with Andoni Morales (who had been pairing with Sebastien) on improving the idle behavior of Twisted's glib2 integration.

Esteve Fernandez worked on adding streaming upload support to Twisted Web's resource model.

Jessica McKellar helped out remotely with reviews of fixes for Windows child process issues, Conch example bugs, Windows+Python 2.7 compatibility, and release-related documentation errors.

Kevin Horn also worked remotely on some problems with the structure of our documentation.

And I walked around and answered questions about Twisted development and made suggestions about implementation approaches to try out. I also reviewed and merged the AMP documentation fix, and also worked on another IMAP4 ticket, improving test coverage for the client's SELECT support.

Twisted 10.0.0 released

On behalf of Twisted Matrix Laboratories, I am honored to announce the release of Twisted 10.0.

Highlights include:

• Improved documentation, including "Twisted Web in 60 seconds"
• Faster Perspective Broker applications
• A new Windows installer that ships without zope.interface
• Twisted no longer supports Python 2.3
• Over one hundred closed tickets

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Many thanks to Jean-Paul Calderone and Chris Armstrong, whose work on release automation tools and answers to numerous questions made this possible. Thanks also to the supporters of the Twisted Software Foundation and to the many contributors for this release.

PyCon 2010 sprint report

Residents of the once-great city of Atlanta are only now beginning to piece their lives back together after the event that Glyph Lefkowitz described as "the best Twisted sprint ever".

Quite apart from the unprecedented levels of wanton destruction, Twisted's PyCon 2010 sprint was also better attended than any previous sprints, which have laid waste to the outskirts of Boston, inner-city Dallas and regions of Australia.

At its peak, there were eighteen people seated around the two tables where the sprint took place. People even joined in remotely, working with the Atlanta team through #twisted on freenode.

Although buried deep within a secret bunker in Atlanta, the sprint still managed to attract a fair number of new contributors and Twisted users who wanted to help out.

Attendees who made themselves known to your correspondent include:

• Jonathan Lange
• Jean-Paul Calderone
• David Reid
• Thomas Hervé
• Tim Couper
• Pavel Pergamenshchik
• Maciej Fijalkowski
• Glyph Lefkowitz
• Itamar Turner-Trauring
• Terry Jones
• Wilfredo Sánchez Vega
• Kevin Horn

Jessica McKellar, Ralph Meijer and Andrew Bennetts also participated remotely.

Attendees all had immense fun and expressed a desire to have another sprint soon. One observed that it had been far too long since giant mecha robot warriors descended on Paris.

Twisted prevails,
jml


More simply, we had a great time at the sprint and got a ridiculous amount of stuff done.

We fixed many, many bugs, including all of the known regressions since the 9.0 release. Once we fixed the regressions, we started work on getting the first 10.0 pre-release out the door.

The pre-release probably would not have been possible without the sprint since it allowed the new release manager ready access to Jean-Paul, who has quite a lot of knowledge stuck in his head. One result of this is a new release process document, which is being developed alongside the actual 10.0 release.

We also reached an in-principle agreement to aim for a release every three months. This means that although hardly any of the good work from the sprint is in the 10.0 release, you won't have to wait long for it to appear in 10.1. Hopefully. I plan write up how this will work as soon as the 10.0 release is done.

The sprint proved to be a perfect opportunity to talk about some of the more gutly bits of Twisted, including Deferred cancellation (see #990) and endpoints (see #1442). The former is done, and the latter has made good progress. Stay tuned.

Thomas Hervé did a lot of work on upgrading our Trac instance, which was getting kind of old and currently crashes all of the time.

We talked a bit about how we could make it easier for people to contribute to Twisted without compromising our high standards of quality. We put as many of the ideas that we could remember on to http://twistedmatrix.com/trac/wiki/ContributionIdeas.

Maciej Fijalkowski was at the sprint working on getting PyPy benchmarks for Twisted and on making PyPy work faster with Twisted. Personally, it's really exciting to see PyPy live up to its promise of a faster Python, particularly with a big codebase like Twisted.

As I said, we fixed lots of bugs. Here's the complete list:

• #4179 - HTTP proxy hanging in Twisted 9.0
• #3056 - twisted.spread.pb.IUsernameMD5Password's docstring should say it accepts an MD5 digest of a password, not plaintext
• #3289 - startLoggingWithObserver raises IndexError
• #4178 - Function singleSearchStep was renamed to _singleSearchStep not everywhere
• #4192 - Move the content from exarkun's Twisted Web in 60 Seconds series into the Twisted Web howto
• #4284 - t.e.adbapi leaks threads
• #4282 - _release.filePathDelta is broken
• #3718 - twisted.internet.posixbase._Waker not defined on Jython
• #4143 - twisted.web.wsgi incorrectly unquotes QUERY_STRING
• #4255 - BaseLogFile should provide a method to reopen logs
• #972 - Add linux inotify support to twisted core
• #2367 - twisted.test.test_failure.FailureTestCase.testBrokenStr break #trial in debug mode
• #1083 - gtk2reactor thread initialization confuses python-2.3
• #3462 - FTP server: upload should support async close
• #990 - Deferred cancellation
• #733 - twisted's SIGCHLD handler breaks popen.
• #2376 - release-twisted should generate guides with links to API docs
• #4272 - flavors.py still talks about getObjectAt
• #4172 - UDP crashes python.exe when using IOCPReactor
• #3925 - test_addresses fails with Glib2 and Gtk2 reactors
• #4255 - BaseLogFile should provide a method to reopen logs
• #4293 - Update links in testing documentation
• #4311 - XMPP Stanza Error 'remote-server-timeout' yield no type and code
• #3811 - Allow aborting HTTP client connections in the _newclient API

We also worked on:

• Getting rid of 3k warnings and generally doing 2to3
• Event support for iocpreactor
• Making Deferreds more iterative (see #411)
• Twisted Web features and bug fixes
• Fixing obscure Windows/gtk2 compatibility issues

Thank you very much to PyCon US, the Python Software Foundation and the many sponsors of the Twisted Software Foundation for their hard-work and support in making this all possible.

Thanks also to the attendees for making it such a blast.

If you want to encourage more of this sort of behaviour, then please donate to the Twisted Software Foundation.

jml