Wednesday, December 30, 2015

Twisted 15.5 Released

(This was queued but not actually posted, sorry everybody!)

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.5!

The sixth (!!) release in 2015 has quite a few goodies in it -- incrementalism is the name of the game here, and everything is just a little better than it was before. Some of the highlights of this release are:

  • Python 3.5 support on POSIX was added, and Python 2.6 support was dropped. We also only support x64 Python on Windows 7 now.
  • More than nine additional modules have been ported to Python 3, ranging from Twisted Web's Agent and downloadPage, twisted.python.logfile, and many others, as well as...
  • twistd is ported to Python 3, and its first plugin, web, is ported.
  • twisted.python.url, a new URL/IRI abstraction, has been introduced to answer the question "just what IS a URL" in Twisted, once and for all.
  • NPN and ALPN support has been added to Twisted's TLS implementation, paving the way for HTTP/2.
  • Conch now supports the DH group14-sha1 and group-exchange-sha256 key exchange algorithms, as well as hmac-sha2-256 and hmac-sha2-512 MAC algorithms. Conch also works nicer with newer OpenSSH implementations.
  • Twisted's IRC support now has a sendCommand() method, which enables the use of sending messages with tags.
  • 55+ closed tickets overall.
For more information, check the NEWS file (link provided below).

You can find the downloads at on PyPI (or alternatively our website) .
The NEWS file is also available at https://github.com/twisted/twisted/blob/twisted-15.5.0/NEWS.

Also worth noting is the two Twisted fellows -- Adi Roiban and myself -- who have been able to dedicate time to reviewing tickets and generally pushing things along in the process. We're funded by the Software Freedom Conservancy (through what is known as the "Twisted Software Foundation"), which is, in turn, funded by donators and sponsors -- potentially like you! If you would like to know how you can assist in the continued funding of the Fellowship program, see our website: https://twistedmatrix.com/trac/wiki/TwistedSoftwareFoundation#BenefitsofSponsorship

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown
Twisted Release Manager, Twisted Fellow

SFC Sponsored Development, Oct-Dec by Amber

Hi everyone,

Along with Adi, I'm supposed to be publishing these too :)

Thanks to our generous sponsors, the Software Freedom Conservancy (of which our part is colloquially known as the Twisted Software Foundation) was able to financially support me as one of the Twisted Fellows. My job is to review tickets, and assist with the Git migration -- this report is mostly about the former, but does have some promising news about the latter.

This groups the tickets under reviewed and triaged -- you can get the full details of my work here, from Trac (which includes both SFC funded ticket reviewing, and non-SFC funded work on code).

Tickets reviewed:

- #5388 (merged): IRI implementation
- #5911 (merged): Support the 'HttpOnly' flag when setting cookies
- #6833: Port twisted.protocols.amp to Python 3
- #7037 (merged): Write test for cftp._cbPutTargetAttrs or remove untested / unused code.
- #7993: Port twisted.web.wsgi to Python 3
- #8082: Document deprecation policy and how to implement a deprecation
- #8107: Add a custom pyflakes runner to ignore Twisted excepted files
- #8116: Deprecate LoopingCall.deferred
- #8121: Port twisted.protocols.htb to Python 3
- #8122 (merged): Final pyflakes cleanup
- #8131 (merged): Port twisted.python.roots to Python 3
- #8132: Port twisted.web.vhost to Python 3
- #8138 (merged): twisted.conch.endpoints._NewConnectionHelper leaks agent connection

Tickets triaged:

- #1436 (invalid): add getCookie/setCookie to web2.http_headers.Headers
- #1920 (invalid): dav.resource.py: RedirectResponse takes a full href, not a uri
- #2342 (invalid): scgi client doesn't handle content-length
- #2343 (invalid): createCGIEnvironment should refuse if path segments have a /
- #2544 (invalid): twisted.web2.channel.[s]cgi contain multiple uses of a function that doesn't exist.
- #3329 (invalid): HTTP's #(...) syntax allows null contents
- #4081 (wontfix): tap2rpm should allow setting RPM dependency information
- #4082 (wontfix): tap2rpm should read configuration from the .tac file it packages.
- #4230 (invalid): Correct exception text in twisted.web2.http_headers:HeaderHandler
- #4426 (wontfix): tap2deb should generate LSB headers for DependencyBasedBoot
- #5875 (wontfix): tap2rpm assumes $PREFIX is /usr
- #6081 (wontfix): Reject unicode in http_headers
- #6168 (wontfix): Enforce type (bytes) of header name and values in twisted.web.http_headers
- #6178 (closed): Port twisted.web.util to Python 3
- #6920 (wontfix): twisted.scripts.test.test_tap2deb.TestTap2DEB.test_basicOperation fails (on Gentoo?)
- #7497 (closed): twistd produces confusing behavior under Python3
- #8007 (closed): API docs try to load urchin.js over http
- #8036: Document how the new logging system should be used and tested in new code

Other
- With others, assembled and proposed a Git Migration Plan, which has now been approved.

Expect to see more in the coming weeks about the Git migration!

- Amber Brown
TSF Fellow & Twisted Release Manager

Tuesday, December 15, 2015

December 2015 first half - SFC Sponsored Development


This is my report for the work done in the first half of Decemeber as part of the 2015 Twisted Maintainer Fellowship program.

Important changes made in these weeks:


* The Git migration plan was sent for approval
* Port of the AMP protocol to Python 3.

Tickets reviewed and merged


* #4811 - @unittest.expectedFailure decorator breaks trial
* #5704 - mkdir -p for FilePath
* #6833 - Port twisted.protocols.amp to Python 3
* #7998 - twisted.conch.ssh.keys should be ported to Python 3
* #8115 - Some Twisted tests use the deprecated "assertEquals" and similar
* #8119 - Remove gadfly support / tests from ADBAPI
* #8125 - LoopingCall.withCount with interval 0
* #8136 - Remove the deprecated twisted.web.http.Request.headers/request_headers
* #8139 - None of the ADBAPI tests actually run

Tickets reviewed and not merged yet


* #6757 - SSL server endpoints don't provide a way to verify client certificates
* #7460 - Add HTTP 2.0 support to twisted.web
* #7993 - 'twisted.web.wsgi' should be ported to Python 3
* #8107 - Add a custom pyflakes runner to ignore Twisted excepted files
* #8124 - Update @deprecated to work with properties and __init__
* #8129 - twisted.web.http_headers.Headers should work on bytes and Unicode
* #8137 - twisted.web server breaks with non-IP addresses
* #8138 - twisted.conch.endpoints._NewConnectionHelper leaks agent connection


Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Tuesday, December 1, 2015

November 2015 week 3 and 4 - SFC Sponsored Development


This is my report for the work done in the last weeks of November as part of the 2015 Twisted Maintainer Fellowship program.

We continue to have a small waiting line for the review queue. In general the initial review response time was below 24 hours, and delayed mainly due to time zones.

Important changes made in these weeks:

* Release of Twisted 15.5.0
* Serial port support was ported to Python 3.
* SSH client and server now support SHA2 HMAC.

Tickets reviewed and merged

#6082 - Make test_updateWithKeywords work on Python 3
#6842 - Dead link in documentation of t.i._dumbwin32proc
#6978 - rst markup bugs in docs
#7668 - Links to Dive into Python in the testing documentation are dead
#7881 - Misleading example in Deferred Intro
#7944 - Typo in docs/core/howto/logger.rst
#7791 - Error in options howto doc
#8050 - Release Twisted 15.5
#8070 - Twisted web client IPV6 DNS lookup failed
#8099 - Port twisted.internet.serial to Python 3.
#8101 - Malformed HTTP Header causes exception in twisted.web.http server
#8102 - Malformed HTTP method causes exception in twisted.web.http server/Resource.render
#8104 - Fix documentation of default thread pool size
#8106 - Add Python 3.5 to our tox config.
#8108 - conch: support hmac-sha2-256 and hmac-sha2-512

Tickets reviewed and not merged yet

#5789 - Remove usage of UserDict
#6757 - SSL server endpoints don't provide a way to verify client certificates
#7050 - Remove long deprecated _bwHack parameter
#7460 - HTTP2 inital patch
#7879 - documentation incorrectly says that Agent does not do certificate validation in Agent
#7926 - TLSMemoryBIOProtocol.loseConnection does not work properly
#7998 - twisted.conch.ssh.keys should be ported to Python 3
#8009 - twisted.web.twcgi should be ported to Python 3
#8077 - Session cookie under python3 never found
#8115 - Some Twisted tests use the deprecated "assertEquals" and similar
#8125 - LoopingCall.withCount with interval 0
#8126 - Improvements to Twisted’s SMTP server
#8127 - Support a user defined facility when using syslog
#8128 - ESMTP class can't be usefully subclassed to add extensions
#8129 - twisted.web.http_headers.Headers should work on bytes and Unicode

Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Thursday, November 19, 2015

November 2015 week 1 and 2 - SFC Sponsored Development

This is my report for the work done in the first 2 weeks of November as part of the 2015 Twisted Maintainer Fellowship program.
Important changes made in these 2 weeks:
  • NPN and ALPN support for the TLS transport.
  • Conch SSH client with diffie-hellman-group-exchange-sha256 support and support for latest release of OpenSSH server.
  • Prepare twisted.web for HTTP/2 supoprt.
  • Start working on replacing PyCrypto usage with cryptography.
  • Twisted review queues continues to be small and to have a review response time of less than 2 days.

Tickets reviewed and merged:

  • #5976 - Remove assertRaises backport from Trial
  • #6980 - doc-build should not build documentation with warnings.
  • #7672 - Add diffie-hellman-group-exchange-sha256 key exchange algorithm to conch
  • #7860 - NPN and APNL support in Twisted.
  • #8068 - twisted.python.urlpath.URLPath doesn't respect the default path of '/'
  • #8072 - twisted.python.urlpath attributes are not mutables
  • #8091 - detox support
  • #8094 - Remove IStreamClientEndpointStringParser.
  • #8095 - tap2deb/tap2rpm tests should not produce deprecation warnings
  • #8096 - Jelly leaks set deprecation warnings.
  • #8097 - MSN leaked deprecation warnings.
  • #8100 - Use MSG_KEX_DH_GEX_REQUEST in conch SSH client

Tickets reviewed and not merged yet:

  • #6757 - Get client certificates for SSL server endpoints
  • #7413 - Migrate conch.ssh to PyCA cryptography.
  • #7998 - twisted.conch.ssh.keys Python 3 port
  • #8025 - Make Trial work on Windows+Python3
  • #8036 - Document the usage of the new logging system
  • #8054 - Twisted's README and other top-level documents should be in rst format
  • #8086 - eventFlatten not preserving the format
  • #8090 - Thread pool does not create enough workers to handle multiple invocations of calllInThread
  • #8099 - Port twisted.internet.serial to Python 3.
  • #8105 - t.protocol.basic.IntNStringReceiver.dataReceived inconsistency
Thanks to theSoftware Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Friday, November 6, 2015

October 2015 - SFC Sponsored Development

October 2015 - SFC Sponsored Development

This is my first post from a series of post dedicated to my work as part of the 2015-2016 Twisted Maintainer Fellowship program.

The Twisted project via Software Freedom Conservancy were kind enought to contract me as a consultant to help with the review queue and the Git migration.
I am sharing this Twisted Maintainer Fellowship program with Amber aka Hawkie aka hawkowl.

Below is the report for October 2015.

Tickets reviewed and merged:

  • #7994 - twisted.python.urlpath update to work with bytes
  • #7717 - Support for diffie-hellman-group14-sha1 in conch.ssh
  • #8042 - Python 3.5 support in Ubuntu 15.10
  • #8028 - Improve error handling in twisted.cred.checkers.FilePasswordDB.
  • #6197 - Port twisted.web.client.downloadPage to Python 3
  • #7650 - Support IPv6 address literals in IAgent implementations
Tickets reviewd and not yet merged:
  • #6757 - SSL server endpoints client certificate retrieval
  • #7926 - Improve TLSMemoryBIOProtocol.loseConnection
  • #8036 - Document the usage and testing of the new logging system in Twisted own code.
  • #7985 - Use sdist to build Twisted distributables.
  • #7860 - Add support for NPN and ALPN
  • #8025 - Have Trial on Windows with Python 3
  • #7826 - Python2.6 cleaup in Trial.
  • #5976 - Python2.6 cleanp in Trial.
  • #8068 - twisted.python.urlpath Python 3 regression.
For the Git migration part I have investigated the current infrastructure and created tickets for each part of the infrastructure which require Git migration work.

All Git migration work is planed to be deployed and documented using the twisted-infra/braid infrastructure help tool and developed using a Vagrant VM. All task are attached to the dedicated Git migration milestone.

No changed related to the Git migration were applied yet to the Twisted infrastructure.

I have also tried to engage with Twisted contributors to get some feedback regarding how we can improve the contribution and the review process.

During the last month, and after a very long time we manage to get the review queue to zero.

We don't have a way to measure the review response time, but during the last month we should also see better response times with review requests not waiting more than 3 or 4 days in the queue, many of which were replied in less than 34 hours.

Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Monday, September 14, 2015

Twisted Devlog - September 2015

Hello everyone! This is something I'm going to try a bit more often -- writing about what's going on with Twisted, what changes are coming, what to expect, and how you can take advantage of what we're doing.

More Python 3

Twisted 15.4 has been released, and with it, the first Trial on Python 3. This means that downstream projects can now use the same testing tools for both Python 2 and Python 3. It is stable enough to be used by the Twisted project on our buildbots, but bugs may exist; if you find something that doesn't work as it should, please let us know by filing a ticket on the issue tracker.

Twisted 15.5 (coming Oct/Nov) will also contain the first release of twistd (the Twisted Daemon runner) on Python 3. The only plugin shipping with it right now is web, with most base features (file serving, running user-specified Resources). WSGI support and distributed web serving has not yet made it in, but the WSGI support is coming soon.

Twisted's porting status is now at ~50% by 'covered lines' (including tests).

More Developer Tools

Developers of Twisted should find the new tox.ini in our source tree useful. It provides a bunch of environments for tox (a testing helper that automatically builds test virtualenvs). Run "tox -l" in your Twisted checkout to see what environments are available.

Removing Python 2.6 Support

Twisted 15.4 was the last release with Python 2.6 support. Now it is not a supported platform and support code is being removed, Twisted 15.5 will no longer be importable on Pythons before 2.7 (for 2.x -- it is unimportable on Py3 versions before 3.3 as well).

Supported Platforms

This year, a handful of new platforms have been added to the fold -- FreeBSD 10.1 (Python 2.7/3.4), RHEL7 (Python 2.7), OS X 10.10 (Python 2.7), Fedora 21/22 (Python 2.7/3.4), Ubuntu 15.04 (Python 2.7/3.4), and Debian 8 (Python 2.7/3.4).

The older platforms which have been removed are RHEL6 (only has 2.6), Debian 6 (only has 2.6), Fedora 17/18/19 (EOL OS), and OS X 10.6 (only has 2.6, EOL OS).

Python 3.5 has now been released, and we will aim to make that a supported platform as soon as it is available in a released OS or official Ubuntu PPA.

Packaging Modernisation

More work is being done on bringing Twisted's packaging into the 21.1st Century.

In Twisted 15.3, we stopped supporting Twisted being packaged as 'subprojects'. There is now only one Twisted package. This has simplified our build tools considerably and made it less confusing when installing Twisted, just giving the one option with the whole distribution, rather than several with fuzzily-defined boundaries and unclear requirements.

Twisted 15.1 was the first release which uses the Setuptools extra_requires feature, where extra dependencies can be installed using pip automatically. For example, to install Twisted and the dependencies required for TLS support, you can use "pip install twisted[tls]". A full list of the optional dependencies that can be installed in this manner can be found in our docs.

It's long been the case that Twisted needs a C compiler to be installed, but that may not be the case for much longer. There is a ticket which is working on moving Twisted's optional C Extensions to another package, _twistedextensions. The base Twisted package will then be installable without a C compiler, and users that require the C extensions provided in _twistedextensions will be able to get it on PyPI.

Since the base Twisted release will be pure-Python, we will now be able to start shipping wheels for both Python 2 (full release) and Python 3 (limited release, containing only the Py3 ported modules).

Additionally, .msi and .exe installers will no longer be released for Windows, making .whls the the only binary installation method for Twisted's C extensions. A CExt .whl will additionally be shipped for Python 2.7 on OS X 10.10 (64bit). No C Extension package will be shipped for Python 3 on any platform, until it contains Python 3 compatible C Extensions.

Call for Sponsors & Volunteers

Twisted's lifeblood is volunteers, either personal or corporate. More people power means that we can get more done, more eyes reviewing code, more ideas and solutions that might solve our unique problems.

Glyph has written more about fundraising for Twisted, and why you should do so here. If you would like to support financially directly, you can donate to Twisted through the Twisted Software Foundation/Software Freedom Conservancy, a 501(c)3 non-profit.

If you would like to volunteer your time to the project, please see our contributors documentation.

Until next time,

Amber Brown (HawkOwl)
Twisted Release Manager

Friday, September 4, 2015

Twisted 15.4 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.4, codenamed "Trial By Fire".

Twisted is continuing to forge ahead, with the highlights of this release being:
  • twisted.positioning, the rest of twisted.internet.endpoints, KQueueReactor (for real this time), and twisted.web.proxy/guard have all been ported to Python 3.
  • Trial has been ported to Python 3! This was made possible by a Python Software Foundation grant.
  • Twisted officially supports several more platforms: Py3.4 on FreeBSD, Py2.7/Py3.4 on Fedora 21/22, and Py2.7 on RHEL7.
  • Python 2.6 is no longer supported, and support for Debian 6, and RHEL6 has been removed because of this.
  • Support for the EOL'd platforms of Fedora 17/18/19 has been removed.
  • Twisted has moved to requiring setuptools for installation.
  • twisted.python.failure.Failure's __repr__ now includes the  exception message.
  • 19 tickets in total closed!
You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

Tuesday, August 4, 2015

Twisted 15.3 Released

On behalf of Twisted Matrix Labs, I am honoured to announce the release of Twisted 15.3.

We're marching confidently into the future, and this release demonstrates this:
  • 10+ modules ported to Python 3 (see NEWS for specifics)
  • Twisted Lore has been removed.
  • Twisted no longer releases as 'subprojects' -- there is only one Twisted, long may it reign.
  • twistd now uses cProfile (instead of Hotshot) as the default profiler.
  • 40+ more tickets closed overall.
You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

Sunday, June 7, 2015

Twisted Fellowship 2015: Call for proposals

On behalf of the Software Freedom Conservancy and the Twisted project I'm happy to announce that we're looking for a paid maintainer for the Twisted project.

Funding a software developer to work as a maintainer will help Twisted grow as a project, and enable Twisted's development community to increase their output of innovative code for the public's benefit. Twisted has strict coding, testing, documentation, and review standards, which ensures excellent code quality, continually improving documentation and code test coverage, and minimal regressions. Code reviews are historically a bottleneck for getting new code merged. A funded maintainer will help alleviate this bottleneck, and speed Twisted's development.

You can read more about the 2015 fellowship at https://twistedmatrix.com/trac/wiki/Fellowship2015

Sunday, May 24, 2015

Twisted 15.2.1 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.2.1.

This is a bugfix release for the 15.2 series that fixes a regression in the new logging framework.

You can find the downloads on PyPI (or alternatively on the Twisted Matrix Labs website).

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
HawkOwl

Monday, May 18, 2015

Twisted 15.2.0 Released

On behalf of Twisted Matrix Labs, I'm honoured to announce the release of Twisted 15.2.

Bringing not only headlining features but also a lot of incremental improvements, this release has got plenty to like:

  • twisted.logger has landed! This is a brand-new, feature-rich logging framework.
  • Python 3.4 is now a supported platform for all the Py3 ported modules.
  • twisted.trial.unittest.TestCase's assertEqual, assertTrue, and assertFalse methods now pass through the standard library's more informative failure messages.
  • twisted.python.filepath.FilePath now supports Unicode (text) paths properly, and includes as{Bytes,Text}Mode methods for interacting with APIs that require a text/bytes-only FilePath.
  • twisted.mail.smtp.sendmail now supports ESMTP and provides a high-level interface for sending mail.
  • The following parts of Twisted are now ported to Python 3:
    • twisted.internet.process
    • twisted.cred.credentials
    • twisted.python.modules
    • twisted.internet.kqreactor
    • twisted.internet.endpoints.ProcessEndpoint
    • twisted.web.static
  • Over 50 tickets closed since 15.1.

You can find the downloads on PyPI (or alternatively the Twisted website).

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
HawkOwl

Monday, April 13, 2015

Twisted 15.1.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.1.0 -- just in time for the PyCon sprints!

This is not a big release, but does have some nice-to-haves:

- You can now install Twisted's optional dependencies easier -- for example, pip install twisted[tls] installs Twisted with TLS support.
- twisted.web.static.File allows defining a custom resource for rendering forbidden pages.
- Twisted's MSN support is now deprecated.
- More documentation has been added on how Trial finds tests.
- ...and 26 other closed tickets containing bug fixes, feature enhancements, and documentation.

For more information, check the NEWS file (link provided below).

You can find the downloads on PyPI (or alternatively the Twisted website) . The NEWS file is also available for reading.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Hawkie Owl

Friday, January 30, 2015

Twisted 15.0.0 Released

Hi everyone!

On behalf of Twisted Matrix Laboratories, I am honoured to announce that Twisted 15.0.0 is here!

Highlights are:
  • SSLv3 is disabled by default by endpoints created by twisted.internet.endpoints.serverFromString and twisted.internet.endpoints.clientFromString.
  • inlineCallbacks now has introductory documentation, and now supports using the return statement with a value on Python 3.
  • twisted.web.client.Agent now supports using UNIX sockets.
  • ProcessEndpoint now has flow control, which makes it useful for many more protocols
  • A whole bunch of bug fixes and other improvements, with 70+ closed tickets.
You can find the downloads on PyPI (or alternatively the Twisted Matrix website). The full details of what’s new in the release is contained in the NEWS file.

As usual, many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code and documentation, and all the people building great things with Twisted!

Twisted Regards,
HawkOwl