Making chat hipper with Twisted!

HipChat has been changing what it means to communicate in teams of any size and taking the world by storm over the past 12 months. What you may not know is that Twisted is at the core of HipChat's highly successful service:

HipChat is a hosted group chat and IM service for businesses which uses the XMPP protocol behind the scenes. We use Twisted to power our fully clustered chat server and run many important job queue workers.

Bottom line: we built and launched a successful business on Twisted & Python in 6 months with no prior knowledge of either. It's a very productive environment with great performance and maintainability. What more could you want?

    – Garret Heaton, Co-founder, HipChat

To learn more about how HipChat uses Twisted check out the success story!

Twisted 10.2.0 Released!

Twisted 10.2.0, the third Twisted release of 2010, has emerged from the mysterious depths of Twisted Matrix Labs, as so many releases before it. Survivors of the release process - what few there were of them - have been heard to claim that this version is "awesome", "even more robust", "fun-sized" and "oven fresh".

Crossing several things that shouldn't ought to be, including the streams and the rubicon, I have assumed the triple responsibilities of feature author, project leader, and release manager for 10.2: with this dark and terrible power - a power which no man ought to wield alone - I have wrought a release which contains many exciting new features, including:

• A plug-in API for adding new types of endpoint descriptions.
• A new, simpler, substantially more robust CoreFoundation reactor.
• Improvements to the implementation of Deferred which should both improve performance and fix certain runtime errors with long callback chains.
• Deferred.setTimeout is (finally) gone.  To quote the author of this change: "A new era of peace has started."
• NetstringReceiver is substantially faster.

And, of course, nearly one hundred smaller bug fixes, documentation updates, and general improvements. See the NEWS file included in the release for more details.

Look upon our Twisted, ye mighty, and make your network applications event-driven: get it now, from:
   <http://twistedmatrix.com/>
... or simply install the 'Twisted' package from PyPI.

Many thanks to Christopher Armstrong, for his work on release-automation tools that made this possible; to Jonathan Lange, for thoroughly documenting the process and thereby making my ascent to the throne of release manager possible, and to Jean-Paul Calderone for his tireless maintenance of our build and test infrastructure as well as his help with the release.

Most of all, thanks to everyone who contributed a patch, reported a bug or reviewed a ticket for 10.2.  Not including those already thanked, there are 41 of you, so it would be a bit tedious to go through everyone, but you know who you are and we absolutely couldn't do it without you!  Thanks a ton!

Fluidinfo sponsors Twisted!

Fluidinfo is Twisted's latest sponsor!  We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

    – Esteve Fernandez

       CTO, Fluidinfo Inc. 

Sponsored Development, July 2010

Hello readers,

It's that time once again. I've just spent a very productive two weeks dedicated to Twisted developments, thanks to the support of the Twisted sponsors and the Software Freedom Conservancy.

There were quite a few tickets to review when I got started this time. Here's the list (italicised tickets are now resolved):

#411 - Returning a Deferred from the callback of another Deferred too many times results a RecursionError

#1384 - Document the meaning of trial's terminal output stanza

#1702 - Kill Defer.setTimeout

#2485 - 2to3 cross-compilation tools

#2661 - Deprecate IFinishableConsumer so it can be deleted

#2680 - abdapi ConnectionPool doesn't allow for synchronous close

#3546 - Add custom time zone support to twisted.log.FileLogObserver

#3586 - I want to install twisted without a c compiler

#3802 - win32 SerialPort requires a reactor that provides "addEvent" method, but doesn't have a good error message if it gets a different one

#4038 - DeprecationWarning: the MimeWriter module is deprecated; use the email package instead

#4045 - Twisted's Failure fake '_Frame' object lacks f_locals

#4138 - A fresh Twisted checkout should support "setup.py sdist"

#4299 - Remove deprecated twisted.web.trp

#4328 - Deprecate twisted.python.text.docstringLStrip

#4378 - Error handling in NetstringReceiver broken

#4419 - IRCClient.noticed() default is prone to causing loops

#4477 - UNIX ports fail to log their path when stopped

#4511 - trial(1) man page says "I don't know why this is in trial." for --disablegc

#4519 - Delay parsing of request body until Request.args is accessed

#4520 - pb.CopiedFailure.throwExceptionIntoGenerator breaks in Python 2.6.

#4522 - Speed up `reactor.spawnProcess()` by only closing open file descriptors

#4527 - Change either documentation or code of t.i.test.reactormixins.ReactorBuilder.requiredInterface

#4531 - Static analysis of module attributes in twisted.python.modules

#4536 - Credentials materials are compared unsafely throughout Twisted

#4540 - t.n.hosts.searchFileFor does not close the file

#4555 - Replace loop-like uses of map() with a loop

#4566 - Improve lore2sphinx buildbot results for `projects/conch`

#4567 - Improve lore2sphinx buildbot results for `projects/core/development`

Additionally, I had lots of time to spend worked on other tickets:

#526 - [PATCH] make t.w.microdom respect case sensitivity

#989 - Default log observer chokes on unicode messages

#1784 - disttrial --hosts=kunai,takkun,muon twisted

#2179 - Parameterize the root directory of the FTP server for non-anonymous logins

#3169 - Add FilePath.descendant(segmentsToChild) to replace fp.child(a).child(b).child(c)

#3242 - use python 2.5 'spwd' module instead of z3p secret 'shadow' module when available

#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented

#3690 - SerialPort never calls connectionLost(reason)

#4300 - Write some unit tests for Deferreds

#4473 - strports.endpoint

#4487 - report multiple tests with the same results at the same time

#4492 - deprecatedModuleAttribute emits warning twice when deprecated module is imported

#4494 - FTPRealm home directory code hardcodes '/home'

#4500 - Use Sphinx for Twisted Documentation

#4501 - trial's error reporting (especially --tbformat=emacs) does not like unicode exceptions

#4503 - domish gets confused by spaces in xmlns names

#4504 - Document reactor.listenMulticast's listenMultiple keyword argument

#4513 - twisted.names server sends incorrect authority section

#4517 - documentation for blockingCallFromThread doesn't exactly say the given callable can return a Deferred

#4528 - t.news.database should use t.m.smtp.sendmail and the email package

#4534 - Update twisted.application.service.IProcess to use Attributes

#4535 - zippath.child('..') does not raise InsecurePath

#4536 - Credentials materials are compared unsafely throughout Twisted

#4539 - fileno() gets called on a ReadDescriptor after removing it from the reactor

#4542 - `LineReceiver` API documentation should indicate when delimiter is stripped etc

#4556 - KQueue reactor misspelled in reactorbuilder

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit http://twistedmatrix.com/trac/ticket/ticketnumber. For example, http://twistedmatrix.com/trac/ticket/4487

Many of these which are still open are now up for review and should be closed over the next few weeks as other developers have a chance to take a look and provide feedback.

A careful reading of this list reveals a few tickets related to our attempts to convert Twisted's documentation from Lore to Sphinx, an undertaking you already know about if you follow the Twisted mailing list. Kevin Horn has spearheaded this project and made great progress towards the goal. Even though the conversion isn't complete yet, there's already some really great looking results.

Twisted 10.1.0 released

Twisted 10.1.0 is finally out. It's a month late, but it's finally out there and free from the shackles of British rule.

Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

Note also that this is the last release in which we will support Windows for Python 2.4.

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download it the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Update: We now have executable Windows installers for Python 2.5 and Python 2.6 to complement the MSIs.

Many thanks to Glyph Lefkowitz, who helped prepare the release, and the PyCon 2010 sprinters, who did so much of the work for this release.

Twisted 10.1.0pre1 released – please test

From the dank and grey depths of a London summer, a thin, reedy voice wanders across the waves of the Internet, saying that it's jolly well pleased to announce the first 10.1.0 pre-release.

We would very much appreciate it if you could download the tarballs, test them on your systems, and file tickets as appropriate.


Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

For more information, see the NEWS file.

If there is only positive feedback from the pre-release, we aim to do the final release some time later this month.

Powering the Silver Screen

Think Twisted is only good for Internet-facing servers? Even then, it's not like Python itself that touches every part of your day in some way, right? Well my friend, you'd be delightfully wrong.

When you watch a movie crediting Lucasfilm for special effects, Twisted could be behind the scenes. Since 2004, Lucasfilm has used Twisted as part of its rendering operations to proxy database connectivity in the render farm (and much more), 24/7 x 365. That's right my friends, Twisted is not only the engine of your Internet...it might have a hand in your entertainment too.

To read more check out the great Success Story from Dave Peticolas from Lucasfilm: Twisted at Lucasfilm

Have your own success story using Twisted? Let us know! Just send an e-mail to success@twistedmatrix.com