The second (of what should end up being many) round of TSF sponsored development has just wrapped up. Like last time, this was two weeks of work by yours truly (with an afternoon of Glyph's time spent reviewing code to keep things moving). This time around there were fewer tickets waiting to be reviewed when I got started, however there was still plenty of code to look at. Over the full period, I reviewed:
#1144 Documentation: twisted.internet.reactor does not appear in the API docs
#1253 Create index.xhtml files for non-core doc trees
#1255 Update copyrights in the man pages
#1878 twisted.web.monitor traceback_ AttributeError: class IChangeNotified has no attribute '__class__'
#1890 move examples from core into correct sub-packages
#1900 Error in documentation online
#2169 twisted.plugin documentation errors
#2208 Standardize on the Python shebang line
#2438 Get rid of references to maintainer email addresses from code
#2552 broken links in intro.xhtml
#2607 conch.checks.SSHPublicKeyDatabase calls os.seteuid/os.setegid even if it's not necessary
#2716 Eliminate relative imports from twisted.conch
#2815 Update VFS backends to an async interface
#2821 create twistd plugin for vfs
#2845 twisted.internet.thread._putResultInDeferred should be public
#3182 Trivial typo in twisted.internet.protocol
#3257 Rewrite twisted.web.static.File.directoryListing to not use woven
#3269 curses.setupterm must only be called once per process
#3300 twistd should support setting the syslog facility
#3326 Typo in gtkmanhole.py
Other tickets I worked on included:
#1069 log observers that throw exceptions should not be removed
#1152 xmlrpc.html doesn't describe how to return errors to the client
#1291 Expose "process exited" hook on ProcessProtocol_ turning current processEnded into user-overriddable behavior
#1493 static File web module doesn't support byte ranges
#2303 Deprecate setUpClass and tearDownClass and_ if possible_ fix the subclassing behaviour.
#2327 Intermittent failure in PB tests
#2631 Update coding standard to indicate preference for TestCase methods which being with "assert" and which do not have an underscore in their name
#2874 _sslverify.problemsFromTransport should be deprecated
#3029 documentation for twisted.python.deprecate.deprecated is incomplete (and other sundries)
#3059 twisted.internet.tcp.Client.getPeer incorrectly returns hostnames
#3116 Errors in processEnded can cause processes to be eternally reaped
#3159 t.i.utils process functions should have a default cwd of None_ not '.'
#3218 SSL disconnection sometimes hangs indefinitely with pyOpenSSL 0.7
#3255 Trial fails to display the line where the error occured in case of SyntaxError
#3300 twistd should support setting the syslog facility
#3301 superfluous local in AMP.__init__
#3305 CR IAC ignored in TelnetClient
#3306 twisted.test.test_ssl.StolenTCPTestCase has a number of defects
#3339 mailmail raises an exception instead of giving an error message
In particular, I spent a lot of time on #1291 which will make it possible to control child processes more precisely with Twisted by separating out the notification that a child process has actually exited from the notification that all of its file descriptors have been closed.
Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.
Thursday, July 10, 2008
Wednesday, July 2, 2008
Twisted in the News
Everyone's Hiring Twisted Talent
The jobs category made it to the top of the list this time around, with folks like Rackspace hiring Knowers in the Ways of Twist. This automation engineer position puts Twisted on par with .Net and C#, giving it title space. This one is looking for talent working with custom protocols and mobile device products. Fluidinfo is also looking for some Twisted talent; if you're interested, email me and I'll put you in touch with them.
The Rackspace job posting is interesting, because a reliable source said that they are doing some very sexy stuff with Twisted and cloud computing. This would be quite the catch, job hunters...
The Fredericksburg Twist
Zope Corp recently released zc.twist 1.3, "Talking to the ZODB in Twisted Reactor Calls." Their page on PyPI has extensive documentation and some great examples. With their Zope 3 work, Zope Corp has consistently provided some of the best technical documentation I've seen in the Python world (possibly fueled by their excellent use of doctests). This release makes me want to jump back into the ZODB :-)
Related to the ZODB, be sure to read Martijn Faassen's recent post.
Compliments
Terry Jones referred to the Twisted crew nicely in his blog post, saying that the framework is "an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers." Well, we're starting to get grey hairs, bald spots, and love handles... so perhaps we're not quite as young as we used to be ;-) But we'll take the compliment anyway, Terry ;-)
Holy Fervor
Colin Alston had some interesting things to say about "religious" technology debates and managed to nicely support Twisted and Divmod's Nevow at the same time. I'm saddened when people try to debate with us about the "right way" to do network programming (right now, everyone's asking how we compare to Erlang). This isn't a jihad, folks. Colin said it very well:
Next up is Glyph's absolutely fantastic essay on scaling with Twisted and Mantissa. JP and I have actually been talking to him about this stuff in great detail lately, due to the messaging work that Glyph is doing. It's a delight that Glyph took the time to put his thoughts and analysis on "paper." There's too much good stuff in there to quote it; you'll have to read the whole thing.
Intermediate to Advanced
Glyph gets the spotlight again with his response to a recent critique of Twisted by a programmer who prefers to write threaded applications. This is a must-read for intermediate to advanced Twisters.
In a similar vein, folks wanting to push their Twisted skills further and do some integration would do well to read this post, which got upmodded at reddit.com last week. It covers some basic advice on creating Twisted-ready async code.
More Blog Bits
JP sketched out a quick prototype of TCP in the browser using Nevow/Athena. The PyPy team has Nevow running on PyPy now! Prasanna Gautam has a quick example of sending binary data over XML-RPC using Twisted. Jack Moffitt of Chesspark fame blogged about their new chat product (based on Twisted) called speeqe. Jack gives us another node in his earlier Firefox GTD blog post. There was a post by Duncan covering some examples of "batching" deferreds in Twisted. Glyph posted an interesting comment on Steve Holden's blog. And finally, Jonathan Lange gives a shout out to the crew on his post about code reviews.
The jobs category made it to the top of the list this time around, with folks like Rackspace hiring Knowers in the Ways of Twist. This automation engineer position puts Twisted on par with .Net and C#, giving it title space. This one is looking for talent working with custom protocols and mobile device products. Fluidinfo is also looking for some Twisted talent; if you're interested, email me and I'll put you in touch with them.
The Rackspace job posting is interesting, because a reliable source said that they are doing some very sexy stuff with Twisted and cloud computing. This would be quite the catch, job hunters...
The Fredericksburg Twist
Zope Corp recently released zc.twist 1.3, "Talking to the ZODB in Twisted Reactor Calls." Their page on PyPI has extensive documentation and some great examples. With their Zope 3 work, Zope Corp has consistently provided some of the best technical documentation I've seen in the Python world (possibly fueled by their excellent use of doctests). This release makes me want to jump back into the ZODB :-)
Related to the ZODB, be sure to read Martijn Faassen's recent post.
Compliments
Terry Jones referred to the Twisted crew nicely in his blog post, saying that the framework is "an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers." Well, we're starting to get grey hairs, bald spots, and love handles... so perhaps we're not quite as young as we used to be ;-) But we'll take the compliment anyway, Terry ;-)
Holy Fervor
Colin Alston had some interesting things to say about "religious" technology debates and managed to nicely support Twisted and Divmod's Nevow at the same time. I'm saddened when people try to debate with us about the "right way" to do network programming (right now, everyone's asking how we compare to Erlang). This isn't a jihad, folks. Colin said it very well:
Why is it though that people become so fanatical about a particular framework or language? At the end of the day our choices of language and framework should be based on merit and not on emotion.Colin also highlighted a few of the many reasons I like to use Twisted and Nevow:
Once you can break this mould, the endless possibilities of web applications become far more apparent. Twisted and Nevow gives you a clean web server tailored for your application, and you can keep your own state information and event control without needing the client to hit on special pages or abusing AJAX to poll resources.Scaling
Next up is Glyph's absolutely fantastic essay on scaling with Twisted and Mantissa. JP and I have actually been talking to him about this stuff in great detail lately, due to the messaging work that Glyph is doing. It's a delight that Glyph took the time to put his thoughts and analysis on "paper." There's too much good stuff in there to quote it; you'll have to read the whole thing.
Intermediate to Advanced
Glyph gets the spotlight again with his response to a recent critique of Twisted by a programmer who prefers to write threaded applications. This is a must-read for intermediate to advanced Twisters.
In a similar vein, folks wanting to push their Twisted skills further and do some integration would do well to read this post, which got upmodded at reddit.com last week. It covers some basic advice on creating Twisted-ready async code.
More Blog Bits
JP sketched out a quick prototype of TCP in the browser using Nevow/Athena. The PyPy team has Nevow running on PyPy now! Prasanna Gautam has a quick example of sending binary data over XML-RPC using Twisted. Jack Moffitt of Chesspark fame blogged about their new chat product (based on Twisted) called speeqe. Jack gives us another node in his earlier Firefox GTD blog post. There was a post by Duncan covering some examples of "batching" deferreds in Twisted. Glyph posted an interesting comment on Steve Holden's blog. And finally, Jonathan Lange gives a shout out to the crew on his post about code reviews.
Thursday, June 12, 2008
Twisted in the News
Love our Monkey
Two great posts from the Chesspark (and WuChess!) guys:
- 5 Reasons You Should Love Twisted Python
- The Twisted-based BOSH Cloud
Oh, and did I mention Twisted-based WuChess was released?
Contracts and Jobs
Looks like someone wants some Twisted development done on LDAP and CIFS.
France's Olfeo and the U.K.'s Isotoma also looking for Twisted talent (among other skill sets).
Research
Travis Branham is using Twisted in a CS/CE university internship:
The project revolves around mapping the uncertainty in position of moving objects. If an object is moving, and providing updates for its position at discrete intervals, it can sometimes be useful to know where that object could be between those updates.He's written a custom SMTP and HTTP server for this.
Seminar
Holden Web and Divmod are providing an introductory Twisted seminar for executives, managers, and technical directors in Washington, D.C. this Summer.
Monday, June 9, 2008
TSF Sponsored Development
Hello all,
On Friday I finished the first two weeks of funded Twisted development, I'm happy to say the two weeks went great and I got a lot accomplished.
The very, very high-level summary is that I started off by emptying the Twisted review queue and then moved on to finishing some twisted.web tasks which had been started previously by other developers and then abandoned for lack of time or interest or whatever other reason. All told, I did 35 reviews on 29 tickets and did development on 8 others. I also spent some time cleaning duplicate or invalid tickets out of the Twisted tracker. This collectively resulted in 32 tickets being closed over the past two weeks. Here are the tickets closed as a direct result of this work:
#1789 - twisted.conch.test.test_filetransfer.TestOurServerOurClient.testOpenFileAttributes (and perhaps others) leaks file descriptors
#3213 - twisted.conch.ssh.filetransfer.FileTransferServer leaks file descriptors
#3203 - PortableGtkReactor is broken
#3098 - filepath.FilePath.walk loops forever if there is a symlink pointing back up the file tree
#3033 - mkdir(a); mkdir(a) logs an unhandled error in SFTP
#3054 - Small error in the documentation for pb
#2881 - Add a variable in twisted.python.runtime to check for vista
#2374 - testOpenSSLBuffering fails intermittently on a dapper machine (running in vmware) because it uses reactor.iterate
#2918 - intermittent twisted.test.test_ssl.SpammyTLSTestCase.testUnTLS failure on OS X
#3210 - test_pb.NewCredTestCase is insanely long and causes some intermittent failures
#3119 - trial leaks memory/objects/something
#3052 - twistd logging options needs to be tested
#3266 - Provide tools for managing new deprecation policy
#3191 - in manhole in gnome-terminal_ lines longer than the terminal is wide result in excessive scrolling and duplication
#3105 - symlink support and docstrings in filepath.FilePath.copyTo/moveTo
#3267 - Typo in the Asynchronous Programming howto
#3184 - Trial minimal reporter prints a negative run time in summary
#1410 - spawnProcess on unix doesn't handle case where os.fork fails cleanly
#2305 - twisted/internet/process.py leaks file descriptors when os.fork() raises
#3272 - sorting the --profile output in trial
#3259 - Improved some error messagge in reflect.py
#2713 - trial -u can fail doctests on second pass
#2698 - Build Bug on Cygwin
#914 - xmlrpc.QueryFactory failed to catch some exceptions
#1057 - Incorrect and misleading arg name msSinceEpoch
#2017 - t.w.client.getPage returning ConnectionDone: Connection was closed cleanly error
#1412 - ExpatError on xmlrpc client connection close
#393 - [PATCH] HTTP Basic Auth based guard
#1108 - web2 needs SOAP support
#909 - microdom getElementsByTagName
#687 - web.client needs documentation and/or data-checking for method argument
#165 - [PATCH]Multiple header with same name support in http
Of particular note, I think, are the following:
#3213: Prior to this fix, the Twisted SFTP server would leak any file descriptors not closed before the end of an SFTP session.
#3203: This fix restores Gtk2 support on Windows. We have also set up a Gtk2 Windows builder which is now passing all but one test.
#1410: Prior to this fix, an error when calling os.fork() would cause reactor.spawnProcess to leak any pipes it opened and potentially to continue running in the main process with the wrong UID/GID.
#165: This adds support for multiple values for each header to the twisted.web server. Previously only the last value received would be made available to applications.
Six more tickets are waiting to be reviewed and will hopefully be closed soon. I want to take this opportunity to thank everyone who worked on tickets I reviewed over the past couple weeks. Twisted development is a collaborative process and I wouldn't have made nearly as big a dent without all of their help. I also want to thank Thomas Hervé in particular, who authored a lot of the code I reviewed and also (if I kept track properly) did every review of code I submitted over the the past two weeks.
In addition to doing work on tickets from the tracker, I also addressed two other issues with Twisted's buildbot. First, I took one of the compact build result views and split it into a supported-only view and an everything view to make it easier to find out the status of recent builds on supported platforms. These two views are linked from the main Twisted buildbot page, http://buildbot.twistedmatrix.com/ - see the "Latest builds" link. Second, I improved the error reporting for the setuptools-enabled builder, making it easier to track down the cause of some setuptools-related problems.
Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible.
Jean-Paul
On Friday I finished the first two weeks of funded Twisted development, I'm happy to say the two weeks went great and I got a lot accomplished.
The very, very high-level summary is that I started off by emptying the Twisted review queue and then moved on to finishing some twisted.web tasks which had been started previously by other developers and then abandoned for lack of time or interest or whatever other reason. All told, I did 35 reviews on 29 tickets and did development on 8 others. I also spent some time cleaning duplicate or invalid tickets out of the Twisted tracker. This collectively resulted in 32 tickets being closed over the past two weeks. Here are the tickets closed as a direct result of this work:
#1789 - twisted.conch.test.test_filetransfer.TestOurServerOurClient.testOpenFileAttributes (and perhaps others) leaks file descriptors
#3213 - twisted.conch.ssh.filetransfer.FileTransferServer leaks file descriptors
#3203 - PortableGtkReactor is broken
#3098 - filepath.FilePath.walk loops forever if there is a symlink pointing back up the file tree
#3033 - mkdir(a); mkdir(a) logs an unhandled error in SFTP
#3054 - Small error in the documentation for pb
#2881 - Add a variable in twisted.python.runtime to check for vista
#2374 - testOpenSSLBuffering fails intermittently on a dapper machine (running in vmware) because it uses reactor.iterate
#2918 - intermittent twisted.test.test_ssl.SpammyTLSTestCase.testUnTLS failure on OS X
#3210 - test_pb.NewCredTestCase is insanely long and causes some intermittent failures
#3119 - trial leaks memory/objects/something
#3052 - twistd logging options needs to be tested
#3266 - Provide tools for managing new deprecation policy
#3191 - in manhole in gnome-terminal_ lines longer than the terminal is wide result in excessive scrolling and duplication
#3105 - symlink support and docstrings in filepath.FilePath.copyTo/moveTo
#3267 - Typo in the Asynchronous Programming howto
#3184 - Trial minimal reporter prints a negative run time in summary
#1410 - spawnProcess on unix doesn't handle case where os.fork fails cleanly
#2305 - twisted/internet/process.py leaks file descriptors when os.fork() raises
#3272 - sorting the --profile output in trial
#3259 - Improved some error messagge in reflect.py
#2713 - trial -u can fail doctests on second pass
#2698 - Build Bug on Cygwin
#914 - xmlrpc.QueryFactory failed to catch some exceptions
#1057 - Incorrect and misleading arg name msSinceEpoch
#2017 - t.w.client.getPage returning ConnectionDone: Connection was closed cleanly error
#1412 - ExpatError on xmlrpc client connection close
#393 - [PATCH] HTTP Basic Auth based guard
#1108 - web2 needs SOAP support
#909 - microdom getElementsByTagName
#687 - web.client needs documentation and/or data-checking for method argument
#165 - [PATCH]Multiple header with same name support in http
Of particular note, I think, are the following:
#3213: Prior to this fix, the Twisted SFTP server would leak any file descriptors not closed before the end of an SFTP session.
#3203: This fix restores Gtk2 support on Windows. We have also set up a Gtk2 Windows builder which is now passing all but one test.
#1410: Prior to this fix, an error when calling os.fork() would cause reactor.spawnProcess to leak any pipes it opened and potentially to continue running in the main process with the wrong UID/GID.
#165: This adds support for multiple values for each header to the twisted.web server. Previously only the last value received would be made available to applications.
Six more tickets are waiting to be reviewed and will hopefully be closed soon. I want to take this opportunity to thank everyone who worked on tickets I reviewed over the past couple weeks. Twisted development is a collaborative process and I wouldn't have made nearly as big a dent without all of their help. I also want to thank Thomas Hervé in particular, who authored a lot of the code I reviewed and also (if I kept track properly) did every review of code I submitted over the the past two weeks.
In addition to doing work on tickets from the tracker, I also addressed two other issues with Twisted's buildbot. First, I took one of the compact build result views and split it into a supported-only view and an everything view to make it easier to find out the status of recent builds on supported platforms. These two views are linked from the main Twisted buildbot page, http://buildbot.twistedmatrix.com/ - see the "Latest builds" link. Second, I improved the error reporting for the setuptools-enabled builder, making it easier to track down the cause of some setuptools-related problems.
Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible.
Jean-Paul
Wednesday, June 4, 2008
A Place for Everyone
After an intense volley on the Twisted mail list, phone conversations, IRC discussions, and lots of thinking by everyone, we are pleased to announce the latest addition to Twisted: Twisted Community Code.
This is what Launchpad calls a "super project" or "project group" but what it means for the Twisted community is that anyone with Twisted code can play. Some details are up on a dedicated wiki page, with more info, help, and documentation to come.
Over the years, many have complained that the Twisted core developers are elitist perfectionists who make it impossible for anyone to contribute code. Most of us would object to such a sweeping statement, but we can't deny that there is at least a little truth in that complaint. The ultimate motivation for our standards and adherence to them is quality. We want Twisted to be as stable, useful, and maintainable as possible. However, quality always comes with a cost, and we're tired of paying it.
What about those who are just learning Twisted? Who want to jump in and contribute some code without having to spend months learning not only Twisted itself, but also its coding standard and our famous review process? Do we have a place in the community for them? Now, the answer to that is "yes" :-)
And this isn't just a dumping ground for newbs; within a day of creating this "super project" on Launchpad -- and with no public announacement -- we had three projects join (today there are five), all by serious long-time Twisted hackers. We expect that folks will contribute everything from excellent to horrible code: we encourage all to do so. Through such things as participation in the community, conversations, and voluntary peer review, good code will get better (and may be merged into Twisted proper) and bad code will learn to mend its evil ways. We have wanted to provide just such a space where everyone in the Twisted community can participate with what they love. Thanks to Canonical's and Launchpad platform, we're starting to do this.
You might notice that the "Twisted Community Code" lives at the "tx" URL. This came up in the email discussions mentioned above. Also, some of the projects are prefixed with "tx." The use of this term is likened to the "py" that is used for general Python projects and we encourange those creating Twisted projects to use the analogous "tx." However, Glyph said it well:
We're hoping that both new Twisted projects and those that have felt alienated in the past can find a home on Launchpad with Twisted Community Code Team Alpha Super Awesome Cool Dynamite Wolf Squadron.
This is what Launchpad calls a "super project" or "project group" but what it means for the Twisted community is that anyone with Twisted code can play. Some details are up on a dedicated wiki page, with more info, help, and documentation to come.
Over the years, many have complained that the Twisted core developers are elitist perfectionists who make it impossible for anyone to contribute code. Most of us would object to such a sweeping statement, but we can't deny that there is at least a little truth in that complaint. The ultimate motivation for our standards and adherence to them is quality. We want Twisted to be as stable, useful, and maintainable as possible. However, quality always comes with a cost, and we're tired of paying it.
What about those who are just learning Twisted? Who want to jump in and contribute some code without having to spend months learning not only Twisted itself, but also its coding standard and our famous review process? Do we have a place in the community for them? Now, the answer to that is "yes" :-)
And this isn't just a dumping ground for newbs; within a day of creating this "super project" on Launchpad -- and with no public announacement -- we had three projects join (today there are five), all by serious long-time Twisted hackers. We expect that folks will contribute everything from excellent to horrible code: we encourage all to do so. Through such things as participation in the community, conversations, and voluntary peer review, good code will get better (and may be merged into Twisted proper) and bad code will learn to mend its evil ways. We have wanted to provide just such a space where everyone in the Twisted community can participate with what they love. Thanks to Canonical's and Launchpad platform, we're starting to do this.
You might notice that the "Twisted Community Code" lives at the "tx" URL. This came up in the email discussions mentioned above. Also, some of the projects are prefixed with "tx." The use of this term is likened to the "py" that is used for general Python projects and we encourange those creating Twisted projects to use the analogous "tx." However, Glyph said it well:
The whole point here is that it's a suggestion and a convention, not a mandate from the core team like so many things in the Twistedverse.Phil Christensen has graciously agreed to take a leadership role in the management of this online community, and we give him many hearty pirate cheers for this. If you have any questions, feel free to send an email to the Twisted mail list... you'll likely get a quick response from Phil :-)
We're hoping that both new Twisted projects and those that have felt alienated in the past can find a home on Launchpad with Twisted Community Code Team Alpha Super Awesome Cool Dynamite Wolf Squadron.
Tuesday, May 20, 2008
ITA, The Twisted Foundation Rockstar
This is amazing: the Software Freedom Conservancy has just received a Twisted sponsorship check from ITA... and it's a big one: $20,000!
ITA is our first Diamond-level sponsor!
We are deeply grateful and excited that our project means so much to people and businesses.
We expect to be able to share some good news about Google, Canonical, and Microsoft in the next week or two, so stay tuned :-)
Saturday, May 17, 2008
Twisted in the News
More Thoughts on Concurrency
The folks at isotoma took some time to thoughtfully respond to my inquiry for more information on the areas where they felt that Twisted was lacking in support of concurrency. This was a fabulous read. Really well done with some hilarious Steve Yegge quotes. I encourage everyone to check it out (but I won't spoil the ending :-) For related reading, you might want to check out these PDFs (thanks Allen Short!):
Software Releases
The open source world has seen some Twisted-related software releases:
Buildbot
We got a mention in Jon Resig's blog post about Mozilla Build and Test Integration as one of the projects that uses Buildbot. Sadly, he failed to mention that Buidlbot itself is built with Twisted ;-)
Press
Zenoss, a Founding Sponsor of the Twisted Software Foundation, has issues a press release about their support of Twisted. Additionally, we're now sponsored by three subsidiaries (Contentinople, Internet Evolution, and Light Reading) of the publishing megacorp UBM (formerly CMP).
Business
Divmod has a new web site up, where they announce that they're officially providing specialized Twisted services in the form of consulting (full disclosure: I work for Divmod).
Subscribe to:
Posts (Atom)