Sponsored Development, October 2011

I'm happy to report that I've just completed another two weeks of sponsored Twisted development.  As usual, the issue tracker directed most of my work.  I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development.  The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on.  Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software.  Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure.  I:

• cleaned up our BuildBot master configuration and published it
• implemented improved error reporting for our API documentation build automation
• converted the subversion repository for the Twisted website to bzr and made it public
• moved some Windows build infrastructure to a new host with better network connectivity to try to reduce spurious build failures

For more details about a ticket, see <http://tm.tl/NNNN>.

The tickets I reviewed, with tickets now closed in bold:

#78 - abortConnection() method for transports

#2674 - t.i.defer.inlineCallbacks documentation issues

#2675 - Test timeout policy should be decided by runner

#3077 - twistd --help plugin list isn't alphabetized

#3078 - Dynamic ZSH tab completion for any commands using t.p.usage

#3350 - make each peer register as a producer in portforward

#3453 - twisted.web._auth.digest mostly ignores the value of the uri field

#3896 - Passing a unicode object to request.write corrupts the entire response

#3926 - twisted.positioning -- a better positioning framework

#4159 - Add pre-exec hook to spawnProcess

#4519 - Delay parsing of request body until Request.args is accessed

#4751 - t.c.telnet.ITelnetProtocol.unhandledSubnegotiation doesn't reflect reality

#4849 - Add stream compression to twisted.words.jabber

#5024 - MSI buildbot patches version to avoid breaking distutils MSI builder, but does it in the wrong place

#5040 - twisted.web.template is lacking patterns

#5044 - Client documentation should use explicit buildProtocol()s rather than magic "protocol = MyProtocol" on factories

#5085 - Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.connectTCP

#5139 - Replace usage of os.path.walk in favor of twisted.python.filepath.FilePath.walk

#5156 - no ambiguous antecedents (plugins documentation edition)

#5192 - 100-continue support for twisted.web.client.Agent

#5194 - log.msg can fail if someone removes an observer at the wrong time

#5250 - finger15.tac patch

#5252 - Manhole should support CTRL-A and CTRL-E for home/end

#5253 - Add support for input history persistence in twisted.conch.manhole

#5263 - add multipart/form-data support to twisted.web.client.Agent for easy form posts

#5267 - allow twisted.internet.ssl.[Private]Certificate (and friends) to be backed with TLSLite rather than pyOpenSSL

#5271 - trial test method timeout support is fragile

#5275 - t.w.template does not always properly escape comments

#5279 - pass mode to FilePath.createDirectory()

#5282 - ILogObserver, log.msg, log.err, and logging.html are insufficiently stern about log observer thread safety

#5283 - opt_user unused in manhole twistd plugin

#5285 - win32eventreactor misses disconnect events that happen too soon

#5286 - twisted.web.template documentation doesn't cover very basic use-case of rendering more than one of something

#5288 - twisted.web.template.Tag.fillSlots documentation doesn't mention it returns self

#5301 - test_tcp_internals.PlatformAssumptionsTestCase fails when we leak fds

#5312 - Create some release automation for building Sphinx documentation

And tickets I developed:

#581 - CopiedFailure.getTraceback should include the result of CopiedFailure.getErrorMessage even if unsafeTracebacks is not set

#745 - writeSequence not well-tested

#1946 - ErrorHolder is basically untested

#2838 - _dumbwin32proc.Process does not implement IProcessTransport completely

#4603 - names should handle unknown records

#4666 - t.i.task.Cooperator scheduler objects must return an IDelayedCall

#4671 - sometimes I want to have some constant values that are part of a set

#5040 - twisted.web.template is lacking patterns

#5062 - ProtocolWrapper masks the name of the wrapped protocol in the logs

#5075 - IntNStringReceiver copies too much data in dataReceived, causing AMP to be slow

#5140 - Deprecate twisted.scripts.tkunzip

#5158 - Installing on Pypy fails because of CPython-specific extension modules

#5233 - win32eventreactor doesn't notice TCP connection is lost in certain (hopefully rare) cases

#5237 - Add SFTP example

#5273 - Improve the API documentation for MultiService.addService to clarify its use in comparison to Service.setServiceParent

#5278 - When connecting with endpoints, the application factory used is neither started nor stopped, and irrelevant start/stop messages are logged instead

#5285 - win32eventreactor misses disconnect events that happen too soon

#5291 - twisted.trial TestCase should support assertDictEqual from Python 2.7 unittest

#5292 - Listening ports used with wrapping factories could log better start messages

#5293 - Remove mktap and corresponding twistd options (mktap is broken in trunk)

#5299 - udp.Port schedules connectionLost call in the global reactor

#5303 - Default transport implementation doesn't write out bytes immediately

#5304 - Win32Reactor.spawnProcess is redundant

#5308 - Make UDP ports identify themselves when stopping

#5316 - Incorrect API link in twisted.web.proxy.Proxy docstring

October Sprint Report

After a brief rest, the Boston Twisted crew is back on the sprinting treadmill. Dave Sturgis came through for us once again for the venue, working with Smarterer to let us use their office space and eat their snacks last Saturday afternoon. We also took advantage of the Python Software Foundation's sprint funding program to buy all the sprinters dinner.

Eight sprinters turned out to fix bugs, add features, and review changes.

Chris (radix) (triumphantly returned from the inhospitable southwest) worked on adding a history-tracking mode to Deferreds to aid in debugging. This feature will let a developer see everything that has happened to a Deferred - what callbacks it has run, what values have passed through it, and what other Deferreds it has been associated with.

Itamar picked up a ticket relating to HTTP client support for persistent connections. He also finished up the work on adding a new transport method, abortConnection, fixed a logging bug in the new TLS implementation, and did some ticket triage and a review.

Allister (amacleod) heroically tackled the problem of our insufficient Windows testing infrastructure, getting introduced to our build farm and setting up a new Windows 7 virtual machine on it.

Jessica (jesstess) did a ton of follow-up on old tickets (1247 2115 2447 2498 2507 2513 2861), trying to learn if they are still valid and elicit further information from the original reporters. She also managed to close one which was no longer valid (but she also filed a new one for a documentation bug, in case you were worried we might run out of tickets).

Glyph knuckled down on a review for our conversion to Sphinx. We're so close I can practically taste it.

Ying (cyli) took a look at improving the authentication options for some of Twisted's built-in servers. She started with the FTP server and made great headway.

As for myself, I reviewed that abortConnection feature as well as a ticket for HTML5 support in twisted.web.template and the TLS logging bug that Itamar fixed.

Thanks to all the sprinters and to Smarterer and the PSF for their support of the sprint! We're going to do another one of these real soon, watch out for an announcement!

New, New, New!

New Success Story

We've added a great success story for PowerCard, a loyalty rewards program for local restaurants. One of the terrific things about the PowerCard story is the use of Twisted for both back-end and client-facing systems. It's a great reminder that Twisted is as useful solving problems on Windows platforms as it is on the Linux/UNIX-based systems many of us code for.

Read the full success story here.

New Library Listing

Looking to use Twisted with new technologies like RabbitMQ (txAMQP) and Redis (txRedis)? Browse the updated libraries listing. We bet you'll find what you're looking for.

New Twisted Ambassador

And why do we have these new and updated listings for 3rd-party code? It's because we have a new guy who has volunteered to maintain and update the listings!

Jason J. W. Williams is Twisted's new ambassador to 3rd-party projects of all kinds.Primarily this responsibility involves keeping the "Success Stories" and "Projects Using Twisted" pages up to date, and occasionally writing small updates (like this one) for the news feed.

Please contact success at twisted matrix dot com if you have a success story about how you used Twisted, a pointer to some hitherto unknown code that uses Twisted, or for more information. And watch this space for more updates!

Twisted in the News

Everyone's Hiring Twisted Talent

The jobs category made it to the top of the list this time around, with folks like Rackspace hiring Knowers in the Ways of Twist. This automation engineer position puts Twisted on par with .Net and C#, giving it title space. This one is looking for talent working with custom protocols and mobile device products. Fluidinfo is also looking for some Twisted talent; if you're interested, email me and I'll put you in touch with them.

The Rackspace job posting is interesting, because a reliable source said that they are doing some very sexy stuff with Twisted and cloud computing. This would be quite the catch, job hunters...

The Fredericksburg Twist

Zope Corp recently released zc.twist 1.3, "Talking to the ZODB in Twisted Reactor Calls." Their page on PyPI has extensive documentation and some great examples. With their Zope 3 work, Zope Corp has consistently provided some of the best technical documentation I've seen in the Python world (possibly fueled by their excellent use of doctests). This release makes me want to jump back into the ZODB :-)

Related to the ZODB, be sure to read Martijn Faassen's recent post.

Compliments

Terry Jones referred to the Twisted crew nicely in his blog post, saying that the framework is "an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers." Well, we're starting to get grey hairs, bald spots, and love handles... so perhaps we're not quite as young as we used to be ;-) But we'll take the compliment anyway, Terry ;-)

Holy Fervor

Colin Alston had some interesting things to say about "religious" technology debates and managed to nicely support Twisted and Divmod's Nevow at the same time. I'm saddened when people try to debate with us about the "right way" to do network programming (right now, everyone's asking how we compare to Erlang). This isn't a jihad, folks. Colin said it very well:

Why is it though that people become so fanatical about a particular framework or language? At the end of the day our choices of language and framework should be based on merit and not on emotion.

Colin also highlighted a few of the many reasons I like to use Twisted and Nevow:

Once you can break this mould, the endless possibilities of web applications become far more apparent. Twisted and Nevow gives you a clean web server tailored for your application, and you can keep your own state information and event control without needing the client to hit on special pages or abusing AJAX to poll resources.

Scaling

Next up is Glyph's absolutely fantastic essay on scaling with Twisted and Mantissa. JP and I have actually been talking to him about this stuff in great detail lately, due to the messaging work that Glyph is doing. It's a delight that Glyph took the time to put his thoughts and analysis on "paper." There's too much good stuff in there to quote it; you'll have to read the whole thing.

Intermediate to Advanced

Glyph gets the spotlight again with his response to a recent critique of Twisted by a programmer who prefers to write threaded applications. This is a must-read for intermediate to advanced Twisters.

In a similar vein, folks wanting to push their Twisted skills further and do some integration would do well to read this post, which got upmodded at reddit.com last week. It covers some basic advice on creating Twisted-ready async code.

More Blog Bits

JP sketched out a quick prototype of TCP in the browser using Nevow/Athena. The PyPy team has Nevow running on PyPy now! Prasanna Gautam has a quick example of sending binary data over XML-RPC using Twisted. Jack Moffitt of Chesspark fame blogged about their new chat product (based on Twisted) called speeqe. Jack gives us another node in his earlier Firefox GTD blog post. There was a post by Duncan covering some examples of "batching" deferreds in Twisted. Glyph posted an interesting comment on Steve Holden's blog. And finally, Jonathan Lange gives a shout out to the crew on his post about code reviews.

Episode 1 of the "Twisted Show"

The Twisted Show is a new experiment we are trying at TM Labs in an effort to let people and business who are solving unique and/or interesting problems with Twisted share their solutions with a wide audience.

This first installment is an interview with a few folks at Renkoo who are using Twisted to innovate meetings as part of the social media landscape.

Twisted 2.5.0

Major Twisted release! Read the release announcement and click on things on the Downloads page.

New Twisted Site Goes Live!

Thanks to the efforts of the Twisted Team and the visual design and CSS skills of Huw Wilkins, we have rolled out our latest version of the Twisted Matrix Labs web site.

Enjoy!

Apple Releases iCal Server Code

Apple's iCal Server is written in Twisted. Visit the development site to read more about the software and to get started with it now.

Twisted 2.4.0

Twisted 2.4.0 is released, including updates to Words, Web, Names, Mail, Conch, and more.

Twisted Sprint in Sydney

Twisted's Australian hackers are converging in Sydney on the June 3-4 weekend. If you are Australian and think you might be made of internet, see the wiki to find out more about the SydneySprint.

Twisted 2.2.0

Twisted 2.2.0 is out. See the Twisted wiki.

Twisted (core) 2.1.0 + First Virtual Sprint

In Twisted's first virtual sprint the weekend of 2005-10-08, many people contributed to Twisted in general and helped get the Twisted 2.1.0 release out. See the Twisted project page and read about what's new.

Australian Twisted Sprint

A Twisted Sprint will be held in Sydney Australia on the 19-21 August 2005. Topics will probably include documentation, a VFS layer (including WebDAV support), the new Axiom database, and whatever else anyone can think of. For more information, please see the Australian Twisted Sprint Wiki.

Twisted 2.0.1 NOW released

Twisted 2.0.1 is now out. This is a minor release, only including bugfixes since 2.0.0. One of the most important fixes was a bug causing many gtk GUI apps to crash. Twisted News is now properly included in the Sumo release. Enjoy.