June End-month report

I spent the majority of my time in the last half of the month, clearing the review queue, from 45 tickets when I started to a low of 19 tickets. All told, 34 tickets got some attention that they would not have received without the sponsored development. The result was 9 closed tickets and 22 other tickets unblocked for other developers to resume work on.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

The tickets I reviewed, that have now been closed:

• #5329: Replace deprecated string functions with str methods in t.w.p.irc
• #5894: ckeygen –changepass truncates private keys to zero bytes
• #5992: twistd dns without any options crashes when it is queried
• #5780: words: Unbound variable nickname in IRCUser for invalid user name
• #6217: iocpreactor events do not occur at the time they should
• #6220: Replace Deprecated Assertion Methods in twisted.conch
• #6328: twisted.names.client lookup functions duplicate or reference the documentation of `IResolver`
• #6499: Teach chain certificates to SSL server endpoints string syntax
• #6544: Replace usage of twisted.python.text in twisted.words

Other tickets I have reviewed:

• #2835: _pollingfile calls pauseProducing on registered Pull producers
• #2839: _pollingfile fails to call stopProducing when connection is lost
• #4353: Automate uploading pre-release tarballs
• #5001: replace call to functions from the string module in twisted.scripts.tap2deb
• #5004: replace call to functions from the string module in web
• #5387: Get rid of references and code specific to Python 2.4
• #5562: iocp reactor fails to support PB & AMP
• #5574: Add support for SOCK_DGRAM to IReactorSocket
• #5685: SMTP Client Tutorial should be self-contained
• #5795: Failure's startDebugMode should allow using a debugger other than pdb
• #6164: Returning an already-chained deferred breaks the callback chain
• #6357: LineReceiver uses O(n**2) complexity
• #6475: FileAuthority._lookup incorrectly returns AuthoritativeDomainError for non-subdomain names which happen to share the same suffix as the zone (eg the-example.com and example.com)
• #6491: test_pollingfile: producer data wrongly dropped when loseConnection() requested
• #6492: test_pollingfile: bufferEmpty sets producerPaused attribute on the producer instead of on itself
• #6493: test_pollingfile: writeSequence() doesn't pause the producer when outgoing buffer is full
• #6523: Constants HOWTO doesn’t document that constants are ordered.
• #6542: Use of map where the result is unused.
• #6543: Replace usage of twisted.python.text in twisted.conch
• #6545: Replace usage of twisted.python.text in twisted.mail
• #6546: Replace usage of twisted.python.text in twisted.lore
• #6547: Class mail.pb.MaildirBroker apparently broken and obsolete

Tickets I have worked on:

• #6301: There should be some clear documentation on how to document tests.
• #6595: twisted should provide a paste entry point, so that it can be used as a wsgi container by `paster serve`.
• #6596: twisted.application.reactors.installReactor should return the installed reactor.
• #6587: Remove out-of-place refernce to lore and howtos from coding-standard doc.

June Mid-month report

During the first half of the month, 24 tickets got some attention that they would not have received without the sponsored development. The result was 2 closed tickets and 22 other tickets unblocked for other developers to resume work on.

Also, as mentioned before, I've released a collection of tools for helping with twisted development on github and pypi.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

The tickets I reviewed, that have now been closed:

• #1518: trial option to exit on failure
• #5435: twisted.web.client.RedirectAgent doesn't inform caller of final URI

Other tickets I have reviewed:

• #1601: t.py.text.greedyWrap makes t.py.usage.Options.longdesc ugly
• #2835: _pollingfile calls pauseProducing on registered Pull producers
• #2839: _pollingfile fails to call stopProducing when connection is lost
• #3926: twisted.positioning – a better positioning framework
• #4929: Add LIST to IRCClient
• #5365: _pollingfile makes wrong assumption about WriteFile
• #5574: Add support for SOCK_DGRAM to IReactorSocket
• #5668: Support the OPT pseudo-record
• #5894: ckeygen –changepass truncates private keys to zero bytes
• #6064: Write tests for `twisted.internet.protocol.FileWrapper`
• #6116: twisted.web.http.stringToDatetime has incomplete test coverage
• #6135: twisted.web.http.HTTPFactory is incompletely tested
• #6137: Add unit tests for twisted.python.dist.getAllScripts
• #6217: iocpreactor events do not occur at the time they should
• #6393: Utility function to pretty-format list of things
• #6401: twisted.web.test.requesthelper.DummyChannel.TCP does not have the unregisterProducer method.
• #6475: twistd dns returns NXDomain for valid records when multiple zone names share a common suffix or parent domain
• #6484: Add unit tests for twisted.python.dist.get_setup_args
• #6491: test_pollingfile: producer data wrongly dropped when loseConnection() requested
• #6492: test_pollingfile: bufferEmpty sets producerPaused attribute on the producer instead of on itself
• #6493: test_pollingfile: writeSequence() doesn't pause the producer when outgoing buffer is full
• #6499: Teach chain certificates to SSL server endpoints string syntax

May end-of-month report

The migration to the new machine has been successfully completed, and reports are, that trac is much faster now. Special thanks to Jonathan Stoppani, whose help was invaluable in preparing for the migration.

Since the migration, I've returned to doing ticket review, 11 tickets got some attention that they would not have received without the sponsored development. The result was 2 closed tickets and 9 other tickets unblocked for other developers to resume work on.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

The tickets I reviewed, that have now been closed:

• #6443: setuid should be called in the child after forking the process in spawnProcess
• #6525: Constant docs have a mistake in an example

Other tickets I have reviewed:

• #3926: twisted.positioning – a better positioning framework
• #5387: Get rid of references and code specific to Python 2.4
• #5434: Web browser-like redirection Agent for twisted.web.client
• #5787: trial –order for selecting an order for running test cases
• #5795: Failure's startDebugMode should allow using a debugger other than pdb
• #5962: Clock.callLater(0, f) can lead to infinite loop
• #6042: twisted.protocols.test.test_basic doesn't following coding standard
• #6071: t.i.task.react's docstring doesn't mention the reactor argument
• #6166: Deprecate pickling support (__getstate__ and __setstate__) in twisted.internet._sslverfiy

Migration Report

I have completed the migration scripts for deploying the services currently running on cube. They have been run against our new machine, dornkirk which is currently running with a snapshot of data.

It can currently be accessed by putting

66.35.39.66     twistedmatrix.com speed.twistedmatrix.com

in /etc/hosts. Please tests it, and verify that things appear to be working, but be aware that any changes will be lost, when the transition occurs.

At some point Monday or Tuesday, there will be some downtime for mail and the mailing lists, as mail-service is migrated to the new machine. For those that have accounts on cube, your data will be copied to the new machine at this point.

On Wednesday, at about 10 MDT (16 UTC), there will be downtime of all twisted services, as live data is transfered over. This may last up-to a couple of hours.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

April end-month report

During the second half of the month, I have concentrated on the continuing development tools to automate the deployment of twisted infrastructure.

I've also spent some time developing some command-line tools for interacting with trac. It currently support view tickets, listing review tickets, and grabbing attachments (including applying them in git with an appropriate commit message). I plan to add support for doing reviews, as well.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

April mid-month Fellowship report

During the first half of the month, 43 tickets got some attention that they would not have received without the sponsored development. The result was 17 closed tickets and 24 other tickets unblocked for other developers to resume work on.

I set up an official git mirror that contains all the branches, including release branches and tags on github and documented my workflow on the wiki.

I've documened how our current infrastructure (cube), and with the help of Jonathan Stoppani, I've been working on scripts to automate the deployment of its successor (dornkirk).

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

The tickets I reviewed, that have now been closed:

• #3844 (Parse mIRC format codes)
• #4696 (client endpoint: process)
• #5386 (Get rid of references and code specific to Python 2.3)
• #5411 (ftp_NLST and ftp_LIST should only send data as "str")
• #5952 (Improve docstring for test_notTrapped in twisted.test.test_failure)
• #5955 (Move inlineCallbacks test from twisted.test.test_failure to somewhere better (test_defer?))
• #5989 (Standardize twisted names examples to use task.react)
• #6120 (twisted.web.http.Request.__repr__ is untested)
• #6240 (Merge `twisted/internet/_utilspy3.py` into `twisted/internet/utils.py`)
• #6284 (FTP.ftp_STOR() should handle IsADirectoryError error)
• #6350: Add a few unittest2 assertions to TestCase
• #6391 (Improve the format of http.Request.__repr__ to include eg class and object identity)
• #6392: Typo in test_irc
• #6400 (trial.test.test_directoryNotPackage is incorrect)
• #6405 (os.geteuid and os.getegid are called unnecessarily in SSHPublicKeyDatabase.checkKey)
• #6430 (test_loader.FileTest.test_filenameNotPython should create its test data)
• #6439 (Spelling mistakes in pb-intro.xhtml)

Other tickets I have reviewed:

• #2061: _sslverify.py: Wrong use of param caCerts in OpenSSLCertificateOptions
• #3701: Add a checker for ISSHPrivateKey which parameterizes the location of the keys
• #4891: twisted.web.http does not throttle pipelined requests
• #5228: twisted.test.test_paths.FilePathTestCase.test_getPermissions_Windows creates undeleteable file
• #5270: Add API to endpoints for connecting a protocol instance
• #5329: Replace deprecated string functions with str methods in t.w.p.irc
• #5435: twisted.web.client.RedirectAgent doesn't inform caller of final URI
• #5531: Add anonymous support to Conch SSHFactory
• #5532: AMP Command specification format
• #5787: trial –order for selecting an order for running test cases
• #5795: Failure's startDebugMode should allow using a debugger other than pdb
• #585: Include sendfile(2) support into defaultreactor
• #5951: Documentation for the best way to get a reference to a reactor reads as contradictory
• #6115: twisted.web.http.parseContentRange has no unit tests
• #6119: twisted.web.http.Request.parseCookies has incomplete test coverage
• #6166: Deprecate pickling support (__getstate__ and __setstate__) in twisted.internet._sslverfiy
• #6341: Replace usage of twisted.python.text and deprecate it
• #6342: Deprecate twisted.python.hashlib
• #6390: Using trial -j can result in wrong paths in sys.path
• #6393: Utility function to pretty-format list of things
• #6395: Delete twisted.words.protocols.msn
• #6402: twisted.words setup metadata is not up to date anymore
• #6426: test_mail.py failure
• #6437: twisted.trial.unittest.TestCase.flushWarnings may fail if run without .py files (with only .pyc or .pyo files)

Other tickets I have worked on:

• #6251 (Add a helper function for getting the body of an HTTP request.)
• #6348 (trial doesn't show `DeprecationWarnings` by default on 2.7)

Sponsored Development, October 2011

I'm happy to report that I've just completed another two weeks of sponsored Twisted development.  As usual, the issue tracker directed most of my work.  I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development.  The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on.  Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software.  Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure.  I:

• cleaned up our BuildBot master configuration and published it
• implemented improved error reporting for our API documentation build automation
• converted the subversion repository for the Twisted website to bzr and made it public
• moved some Windows build infrastructure to a new host with better network connectivity to try to reduce spurious build failures

For more details about a ticket, see <http://tm.tl/NNNN>.

The tickets I reviewed, with tickets now closed in bold:

#78 - abortConnection() method for transports

#2674 - t.i.defer.inlineCallbacks documentation issues

#2675 - Test timeout policy should be decided by runner

#3077 - twistd --help plugin list isn't alphabetized

#3078 - Dynamic ZSH tab completion for any commands using t.p.usage

#3350 - make each peer register as a producer in portforward

#3453 - twisted.web._auth.digest mostly ignores the value of the uri field

#3896 - Passing a unicode object to request.write corrupts the entire response

#3926 - twisted.positioning -- a better positioning framework

#4159 - Add pre-exec hook to spawnProcess

#4519 - Delay parsing of request body until Request.args is accessed

#4751 - t.c.telnet.ITelnetProtocol.unhandledSubnegotiation doesn't reflect reality

#4849 - Add stream compression to twisted.words.jabber

#5024 - MSI buildbot patches version to avoid breaking distutils MSI builder, but does it in the wrong place

#5040 - twisted.web.template is lacking patterns

#5044 - Client documentation should use explicit buildProtocol()s rather than magic "protocol = MyProtocol" on factories

#5085 - Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.connectTCP

#5139 - Replace usage of os.path.walk in favor of twisted.python.filepath.FilePath.walk

#5156 - no ambiguous antecedents (plugins documentation edition)

#5192 - 100-continue support for twisted.web.client.Agent

#5194 - log.msg can fail if someone removes an observer at the wrong time

#5250 - finger15.tac patch

#5252 - Manhole should support CTRL-A and CTRL-E for home/end

#5253 - Add support for input history persistence in twisted.conch.manhole

#5263 - add multipart/form-data support to twisted.web.client.Agent for easy form posts

#5267 - allow twisted.internet.ssl.[Private]Certificate (and friends) to be backed with TLSLite rather than pyOpenSSL

#5271 - trial test method timeout support is fragile

#5275 - t.w.template does not always properly escape comments

#5279 - pass mode to FilePath.createDirectory()

#5282 - ILogObserver, log.msg, log.err, and logging.html are insufficiently stern about log observer thread safety

#5283 - opt_user unused in manhole twistd plugin

#5285 - win32eventreactor misses disconnect events that happen too soon

#5286 - twisted.web.template documentation doesn't cover very basic use-case of rendering more than one of something

#5288 - twisted.web.template.Tag.fillSlots documentation doesn't mention it returns self

#5301 - test_tcp_internals.PlatformAssumptionsTestCase fails when we leak fds

#5312 - Create some release automation for building Sphinx documentation

And tickets I developed:

#581 - CopiedFailure.getTraceback should include the result of CopiedFailure.getErrorMessage even if unsafeTracebacks is not set

#745 - writeSequence not well-tested

#1946 - ErrorHolder is basically untested

#2838 - _dumbwin32proc.Process does not implement IProcessTransport completely

#4603 - names should handle unknown records

#4666 - t.i.task.Cooperator scheduler objects must return an IDelayedCall

#4671 - sometimes I want to have some constant values that are part of a set

#5040 - twisted.web.template is lacking patterns

#5062 - ProtocolWrapper masks the name of the wrapped protocol in the logs

#5075 - IntNStringReceiver copies too much data in dataReceived, causing AMP to be slow

#5140 - Deprecate twisted.scripts.tkunzip

#5158 - Installing on Pypy fails because of CPython-specific extension modules

#5233 - win32eventreactor doesn't notice TCP connection is lost in certain (hopefully rare) cases

#5237 - Add SFTP example

#5273 - Improve the API documentation for MultiService.addService to clarify its use in comparison to Service.setServiceParent

#5278 - When connecting with endpoints, the application factory used is neither started nor stopped, and irrelevant start/stop messages are logged instead

#5285 - win32eventreactor misses disconnect events that happen too soon

#5291 - twisted.trial TestCase should support assertDictEqual from Python 2.7 unittest

#5292 - Listening ports used with wrapping factories could log better start messages

#5293 - Remove mktap and corresponding twistd options (mktap is broken in trunk)

#5299 - udp.Port schedules connectionLost call in the global reactor

#5303 - Default transport implementation doesn't write out bytes immediately

#5304 - Win32Reactor.spawnProcess is redundant

#5308 - Make UDP ports identify themselves when stopping

#5316 - Incorrect API link in twisted.web.proxy.Proxy docstring

Unclaimed PayPal donations

Hello Twisted supporters,

If you recently tried to donate money to the Twisted Software Foundation using PayPal, then you might find that your donation has gone "unclaimed".  This is due to an unfortunate mixup in our PayPal address.  If you have a donation in this state, you should cancel it; we have no way to claim it.

If you'd like to donate, use the PayPal form on http://twistedmatrix.com/. We've updated it so future donations will go to the right place.  If you'd rather use Google Checkout, the form for donating with that service is also still available in the same place.

Fluidinfo sponsors Twisted!

Fluidinfo is Twisted's latest sponsor!  We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

    – Esteve Fernandez

       CTO, Fluidinfo Inc.