Wednesday, March 3, 2010

Twisted 10.0.0 released

On behalf of Twisted Matrix Laboratories, I am honored to announce the release of Twisted 10.0.

Highlights include:
  • Improved documentation, including "Twisted Web in 60 seconds"
  • Faster Perspective Broker applications
  • A new Windows installer that ships without zope.interface
  • Twisted no longer supports Python 2.3
  • Over one hundred closed tickets
For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Many thanks to Jean-Paul Calderone and Chris Armstrong, whose work on release automation tools and answers to numerous questions made this possible. Thanks also to the supporters of the Twisted Software Foundation and to the many contributors for this release.

Tuesday, March 2, 2010

PyCon 2010 sprint report

Residents of the once-great city of Atlanta are only now beginning to piece their lives back together after the event that Glyph Lefkowitz described as "the best Twisted sprint ever".

Quite apart from the unprecedented levels of wanton destruction, Twisted's PyCon 2010 sprint was also better attended than any previous sprints, which have laid waste to the outskirts of Boston, inner-city Dallas and regions of Australia.

At its peak, there were eighteen people seated around the two tables where the sprint took place. People even joined in remotely, working with the Atlanta team through #twisted on freenode.

Although buried deep within a secret bunker in Atlanta, the sprint still managed to attract a fair number of new contributors and Twisted users who wanted to help out.

Attendees who made themselves known to your correspondent include:

  • Jonathan Lange
  • Jean-Paul Calderone
  • David Reid
  • Thomas Hervé
  • Tim Couper
  • Pavel Pergamenshchik
  • Maciej Fijalkowski
  • Glyph Lefkowitz
  • Itamar Turner-Trauring
  • Terry Jones
  • Wilfredo Sánchez Vega
  • Kevin Horn
Jessica McKellar, Ralph Meijer and Andrew Bennetts also participated remotely.

Attendees all had immense fun and expressed a desire to have another sprint soon. One observed that it had been far too long since giant mecha robot warriors descended on Paris.

Twisted prevails,
jml


More simply, we had a great time at the sprint and got a ridiculous amount of stuff done.

We fixed many, many bugs, including all of the known regressions since the 9.0 release. Once we fixed the regressions, we started work on getting the first 10.0 pre-release out the door.

The pre-release probably would not have been possible without the sprint since it allowed the new release manager ready access to Jean-Paul, who has quite a lot of knowledge stuck in his head. One result of this is a new release process document, which is being developed alongside the actual 10.0 release.

We also reached an in-principle agreement to aim for a release every three months. This means that although hardly any of the good work from the sprint is in the 10.0 release, you won't have to wait long for it to appear in 10.1. Hopefully. I plan write up how this will work as soon as the 10.0 release is done.

The sprint proved to be a perfect opportunity to talk about some of the more gutly bits of Twisted, including Deferred cancellation (see #990) and endpoints (see #1442). The former is done, and the latter has made good progress. Stay tuned.

Thomas Hervé did a lot of work on upgrading our Trac instance, which was getting kind of old and currently crashes all of the time.

We talked a bit about how we could make it easier for people to contribute to Twisted without compromising our high standards of quality. We put as many of the ideas that we could remember on to http://twistedmatrix.com/trac/wiki/ContributionIdeas.

Maciej Fijalkowski was at the sprint working on getting PyPy benchmarks for Twisted and on making PyPy work faster with Twisted. Personally, it's really exciting to see PyPy live up to its promise of a faster Python, particularly with a big codebase like Twisted.

As I said, we fixed lots of bugs. Here's the complete list:

  • #4179 - HTTP proxy hanging in Twisted 9.0
  • #3056 - twisted.spread.pb.IUsernameMD5Password's docstring should say it accepts an MD5 digest of a password, not plaintext
  • #3289 - startLoggingWithObserver raises IndexError
  • #4178 - Function singleSearchStep was renamed to _singleSearchStep not everywhere
  • #4192 - Move the content from exarkun's Twisted Web in 60 Seconds series into the Twisted Web howto
  • #4284 - t.e.adbapi leaks threads
  • #4282 - _release.filePathDelta is broken
  • #3718 - twisted.internet.posixbase._Waker not defined on Jython
  • #4143 - twisted.web.wsgi incorrectly unquotes QUERY_STRING
  • #4255 - BaseLogFile should provide a method to reopen logs
  • #972 - Add linux inotify support to twisted core
  • #2367 - twisted.test.test_failure.FailureTestCase.testBrokenStr break #trial in debug mode
  • #1083 - gtk2reactor thread initialization confuses python-2.3
  • #3462 - FTP server: upload should support async close
  • #990 - Deferred cancellation
  • #733 - twisted's SIGCHLD handler breaks popen.
  • #2376 - release-twisted should generate guides with links to API docs
  • #4272 - flavors.py still talks about getObjectAt
  • #4172 - UDP crashes python.exe when using IOCPReactor
  • #3925 - test_addresses fails with Glib2 and Gtk2 reactors
  • #4255 - BaseLogFile should provide a method to reopen logs
  • #4293 - Update links in testing documentation
  • #4311 - XMPP Stanza Error 'remote-server-timeout' yield no type and code
  • #3811 - Allow aborting HTTP client connections in the _newclient API


We also worked on:
  • Getting rid of 3k warnings and generally doing 2to3
  • Event support for iocpreactor
  • Making Deferreds more iterative (see #411)
  • Twisted Web features and bug fixes
  • Fixing obscure Windows/gtk2 compatibility issues
Thank you very much to PyCon US, the Python Software Foundation and the many sponsors of the Twisted Software Foundation for their hard-work and support in making this all possible.

Thanks also to the attendees for making it such a blast.

If you want to encourage more of this sort of behaviour, then please donate to the Twisted Software Foundation.

jml

Monday, December 14, 2009

Sponsored Development, December 2009

Hello all,

It's been a while since my last report of sponsored Twisted development. After this hiatus, I'm happy to be able to bring you a new list of accomplishments I was able to achieve thanks to the support of the Twisted sponsors and the Software Freedom Conservancy.

The tickets I reviewed this time around (italics for tickets now resolved):

#3007 - Subsystem's connectionLost method executed twice in certain situation
#3226 - twisted.python.util.initgroups (and thus spawnProcess) potentially does tons of unnecessary IO
#3296 - Account for ISUPPORT information in irc_MODE when parsing mode strings
#3931 - t.p.amp.AmpBox.serialize will _not_ return a string if any value is unicode
#4009 - twisted.python.modules.PythonPath docstring typo
#4029 - Twisted Web's wsgi.input isn't sufficiently file-like in all cases
#4065 - Replace 'callable(foo)' with 'hasattr(foo_ "__call__")'


And the tickets I worked on (often with the help of others):

#733 - twisted's SIGCHLD handler breaks popen
#970 - twisted.names.root.Resolver eats filedescriptors
#1784 - disttrial --hosts=kunai_takkun_muon twisted
#2078 - Twisted shouldn't start processes until the reactor is running
#2884 - Tool to update / generate NEWS file
#3886 - Fixing the Socks4 proxy to be Socksv4a compliant
#3922 - twisted.protocols.amp.BinaryBoxProtocol.connectionLost chokes on reason = main.CONNECTION_DONE
#3932 - HTTPAuthSessionWrapper incorrectly returns a 500 if the wrapped resource doesn't implement the requested method
#3956 - Add arraysize option to runQuery in adbapi
#4019 - t.web.wsgi.WSGIResource - WSGI application error handling
#4029 - Twisted Web's wsgi.input isn't sufficiently file-like in all cases
#4031 - twisted.mail.smtp client doesn't really implement SASL LOGIN
#4049 - Error IMAP4Client.fetchSpecific an HTML body part
#4053 - Replace "x.has_key(y)" with "y in x"
#4063 - Ween trial's --dry-run away from the deprecated TestCase.visitor code
#4066 - `reduce` for Python 3.x
#4075 - Remove deprecated IReactorUDP.connectUDP
#4114 - Test all of the file-related twisted.web.wsgi code-paths with each of the possible file-like objects they might be used with
#4115 - lore -olint's `check_texturl_matches_href` and `check_a_py_listing` methods invoke getAttribute wrong
#4116 - Add a parameterized AMP argument type which can serialize and unserialize 0 or more of a specified type
#4131 - `twisted.web.client.Agent` does not add a Host header if `None` (the default) is passed for the `headers` argument to `request`
#4154 - Skip all release automation related tests on platforms which are not Linux

You can find more details about these tickets in the Twisted issue tracker. To look up a ticket, visit http://twistedmatrix.com/trac/ticket/ticketnumber. For example, http://twistedmatrix.com/trac/ticket/4063

Additionally, you may have noticed that we managed to finally release Twisted 9.0. This was mostly thanks to the efforts of Chris Armstrong (as usual), but I also helped out reviewing some of that process.

This should signal the beginning of more regular sponsored development. Look forward to seeing another one of these reports soon. :)

Friday, November 20, 2009

Twisted 9 is in the works, first pre-release is available

Hi all! I've done an initial pre-release of Twisted 9. I've made available a set of tarballs (The one called Twisted-9.0.0pre1.tar.bz2 is the main release) and the release notes.

Please try out this prerelease and tell us about any regressions or other release-related problems you find. If all goes well, we want to hear about that too.

Friday, November 6, 2009

New OS X build slave

As some of you may recall, Apple lent Twisted an Xserve G5 a few years back. This was invaluable in maintaining Twisted support for OS X. A few weeks ago we unfortunately lost the ability to host this machine. Fortunately, David Reid came through and very generously donated a

which is now set up and running the Twisted test suite continuously. Thanks, David!

Monday, May 25, 2009

Canonical Men of Twistery



The title of this quick little post goes to the excellent name mwh gave the linked photo :-) Taken at a dinner during AllHands in a gorgeous area outside of Barcelona, this pic shows the collection of Twisted developers that Canonical has hired (so far!).

Top row, left to right: Andrew Bennetts (spiv), Thomas Herve (therve), Chris Armstrong (radix), Michael Hudson (mwh), Jamu Kakar (jkakar).

Bottom row, left to right: Nicola Larosa (tekNico), Duncan McGreggor (oubiwann), Jonathan Lange (jml).

Pretty freakin' cool :-)


Saturday, May 23, 2009

Sponsorship In 2009

It's that time of year again; the time when you all have the excellent opportunity to settle your karmic debt and sponsor the Twisted project.

Actually, that time is every day of the year.  Right now is just when we have arbitrarily decided to make you aware of this opportunity, so that we're not annoying you all year-round.  Sometime soon we'll be taking down the current list of sponsors for 2008 — hopefully, to make room for a new list of sponsors for 2009.  But without your help, that list will be empty!

Thanks to our founding sponsors, Twisted has had a great year. If you've been reading this news feed, you know that we've been able to review, respond to, and close tons of tickets this year.

As Jean-Paul Calderone previously wrote, this is the first year we've had a year-over-year decrease in the number of open tickets; and that's despite an increase in the number of filed tickets.  We believe that Twisted is setting a great example for how other open source projects can effectively use sponsorship to fund development.

Many open source projects are perpetually crunched for volunteer labor. Twisted is no different in this respect — we could always use more help from the community.  However, we are lucky to have something that most projects don't: the ability to convert dollars directly into maintenance that benefits the project.

When you contribute money to the Twisted project, you are directly contributing to get bugs fixed, features implemented, and tickets reviewed.   Last year we told the same story, but this year we have an excellent track record. I invite you to read the history of our sponsored development — just scroll down and have a look. I'll wait.

... Done?

If you prefer visual representations of our success instead, consider these graphs.  Here's a graph of all tickets reviewed this year, by the handle of the person who reviewed them:


Here's the same breakdown of all tickets that were closed this year:

In case you don't already know: "exarkun" is the handle for Jean-Paul Calderone.  By his own calculation, at least 75% of his work this year has been sponsored development.

This isn't to say that Twisted is being supported entirely by sponsorship: as you can see from the huge number of tickets that exarkun has reviewed, a substantial proportion of development has been coming from the community, but sponsorship has allowed us to be more responsive to the community and integrate contributions much faster.  (Developers may not review their own tickets, so there's no overlap between the code JP has been writing and the tickets he's been reviewing.)

This year of sponsorship has been great for the project.  But, we're coming to the end of our first year of funding, and we need your help to keep going.

Every little bit helps, from small, one-time individual donations to monthly subscriptions to corporate sponsorships. This year, we need even more help from individuals, as some of our corporate sponsors have been impacted by the current economic climate and may not be able to contribute at the same level this year.

To summarize, there are a few things you can do to help the Twisted project:
  1. Contribute code.  Any contribution is welcome, but the best would be to attack an existing ticket in the tracker, especially one that has stalled on its way through review.
  2. Donate money directly.  You'll find donation buttons on the front page of twistedmatrix.com.
  3. Donate money from your organization.  Doing so gets you a voice on our sponsors-only mailing list, and can help you to set priorities for the coming year of development.  You also get your logo on twistedmatrix.com, our gratitude, and last but not least, more Twisted development in the coming year!
  4. Help us in our discussions with larger organizations.  If you're a software developer using Twisted in a big company, we need your help to convince your organization that they should be contributing.  Even if not, the board of the Twisted project is small and strapped for time - we could use some help just finding the right person to talk to, and making sure that all of their questions are answered promptly.  (If you'd like to help with this effort, please contact me directly.)

Thanks for your interest in Twisted.