Thursday, November 19, 2015

November 2015 week 1 and 2 - SFC Sponsored Development

This is my report for the work done in the first 2 weeks of November as part of the 2015 Twisted Maintainer Fellowship program.
Important changes made in these 2 weeks:
  • NPN and ALPN support for the TLS transport.
  • Conch SSH client with diffie-hellman-group-exchange-sha256 support and support for latest release of OpenSSH server.
  • Prepare twisted.web for HTTP/2 supoprt.
  • Start working on replacing PyCrypto usage with cryptography.
  • Twisted review queues continues to be small and to have a review response time of less than 2 days.

Tickets reviewed and merged:

  • #5976 - Remove assertRaises backport from Trial
  • #6980 - doc-build should not build documentation with warnings.
  • #7672 - Add diffie-hellman-group-exchange-sha256 key exchange algorithm to conch
  • #7860 - NPN and APNL support in Twisted.
  • #8068 - twisted.python.urlpath.URLPath doesn't respect the default path of '/'
  • #8072 - twisted.python.urlpath attributes are not mutables
  • #8091 - detox support
  • #8094 - Remove IStreamClientEndpointStringParser.
  • #8095 - tap2deb/tap2rpm tests should not produce deprecation warnings
  • #8096 - Jelly leaks set deprecation warnings.
  • #8097 - MSN leaked deprecation warnings.
  • #8100 - Use MSG_KEX_DH_GEX_REQUEST in conch SSH client

Tickets reviewed and not merged yet:

  • #6757 - Get client certificates for SSL server endpoints
  • #7413 - Migrate conch.ssh to PyCA cryptography.
  • #7998 - twisted.conch.ssh.keys Python 3 port
  • #8025 - Make Trial work on Windows+Python3
  • #8036 - Document the usage of the new logging system
  • #8054 - Twisted's README and other top-level documents should be in rst format
  • #8086 - eventFlatten not preserving the format
  • #8090 - Thread pool does not create enough workers to handle multiple invocations of calllInThread
  • #8099 - Port twisted.internet.serial to Python 3.
  • #8105 - t.protocol.basic.IntNStringReceiver.dataReceived inconsistency
Thanks to theSoftware Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Friday, November 6, 2015

October 2015 - SFC Sponsored Development

October 2015 - SFC Sponsored Development

This is my first post from a series of post dedicated to my work as part of the 2015-2016 Twisted Maintainer Fellowship program.

The Twisted project via Software Freedom Conservancy were kind enought to contract me as a consultant to help with the review queue and the Git migration.
I am sharing this Twisted Maintainer Fellowship program with Amber aka Hawkie aka hawkowl.

Below is the report for October 2015.

Tickets reviewed and merged:

  • #7994 - twisted.python.urlpath update to work with bytes
  • #7717 - Support for diffie-hellman-group14-sha1 in conch.ssh
  • #8042 - Python 3.5 support in Ubuntu 15.10
  • #8028 - Improve error handling in twisted.cred.checkers.FilePasswordDB.
  • #6197 - Port twisted.web.client.downloadPage to Python 3
  • #7650 - Support IPv6 address literals in IAgent implementations
Tickets reviewd and not yet merged:
  • #6757 - SSL server endpoints client certificate retrieval
  • #7926 - Improve TLSMemoryBIOProtocol.loseConnection
  • #8036 - Document the usage and testing of the new logging system in Twisted own code.
  • #7985 - Use sdist to build Twisted distributables.
  • #7860 - Add support for NPN and ALPN
  • #8025 - Have Trial on Windows with Python 3
  • #7826 - Python2.6 cleaup in Trial.
  • #5976 - Python2.6 cleanp in Trial.
  • #8068 - twisted.python.urlpath Python 3 regression.
For the Git migration part I have investigated the current infrastructure and created tickets for each part of the infrastructure which require Git migration work.

All Git migration work is planed to be deployed and documented using the twisted-infra/braid infrastructure help tool and developed using a Vagrant VM. All task are attached to the dedicated Git migration milestone.

No changed related to the Git migration were applied yet to the Twisted infrastructure.

I have also tried to engage with Twisted contributors to get some feedback regarding how we can improve the contribution and the review process.

During the last month, and after a very long time we manage to get the review queue to zero.

We don't have a way to measure the review response time, but during the last month we should also see better response times with review requests not waiting more than 3 or 4 days in the queue, many of which were replied in less than 34 hours.

Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Monday, September 14, 2015

Twisted Devlog - September 2015

Hello everyone! This is something I'm going to try a bit more often -- writing about what's going on with Twisted, what changes are coming, what to expect, and how you can take advantage of what we're doing.

More Python 3

Twisted 15.4 has been released, and with it, the first Trial on Python 3. This means that downstream projects can now use the same testing tools for both Python 2 and Python 3. It is stable enough to be used by the Twisted project on our buildbots, but bugs may exist; if you find something that doesn't work as it should, please let us know by filing a ticket on the issue tracker.

Twisted 15.5 (coming Oct/Nov) will also contain the first release of twistd (the Twisted Daemon runner) on Python 3. The only plugin shipping with it right now is web, with most base features (file serving, running user-specified Resources). WSGI support and distributed web serving has not yet made it in, but the WSGI support is coming soon.

Twisted's porting status is now at ~50% by 'covered lines' (including tests).

More Developer Tools

Developers of Twisted should find the new tox.ini in our source tree useful. It provides a bunch of environments for tox (a testing helper that automatically builds test virtualenvs). Run "tox -l" in your Twisted checkout to see what environments are available.

Removing Python 2.6 Support

Twisted 15.4 was the last release with Python 2.6 support. Now it is not a supported platform and support code is being removed, Twisted 15.5 will no longer be importable on Pythons before 2.7 (for 2.x -- it is unimportable on Py3 versions before 3.3 as well).

Supported Platforms

This year, a handful of new platforms have been added to the fold -- FreeBSD 10.1 (Python 2.7/3.4), RHEL7 (Python 2.7), OS X 10.10 (Python 2.7), Fedora 21/22 (Python 2.7/3.4), Ubuntu 15.04 (Python 2.7/3.4), and Debian 8 (Python 2.7/3.4).

The older platforms which have been removed are RHEL6 (only has 2.6), Debian 6 (only has 2.6), Fedora 17/18/19 (EOL OS), and OS X 10.6 (only has 2.6, EOL OS).

Python 3.5 has now been released, and we will aim to make that a supported platform as soon as it is available in a released OS or official Ubuntu PPA.

Packaging Modernisation

More work is being done on bringing Twisted's packaging into the 21.1st Century.

In Twisted 15.3, we stopped supporting Twisted being packaged as 'subprojects'. There is now only one Twisted package. This has simplified our build tools considerably and made it less confusing when installing Twisted, just giving the one option with the whole distribution, rather than several with fuzzily-defined boundaries and unclear requirements.

Twisted 15.1 was the first release which uses the Setuptools extra_requires feature, where extra dependencies can be installed using pip automatically. For example, to install Twisted and the dependencies required for TLS support, you can use "pip install twisted[tls]". A full list of the optional dependencies that can be installed in this manner can be found in our docs.

It's long been the case that Twisted needs a C compiler to be installed, but that may not be the case for much longer. There is a ticket which is working on moving Twisted's optional C Extensions to another package, _twistedextensions. The base Twisted package will then be installable without a C compiler, and users that require the C extensions provided in _twistedextensions will be able to get it on PyPI.

Since the base Twisted release will be pure-Python, we will now be able to start shipping wheels for both Python 2 (full release) and Python 3 (limited release, containing only the Py3 ported modules).

Additionally, .msi and .exe installers will no longer be released for Windows, making .whls the the only binary installation method for Twisted's C extensions. A CExt .whl will additionally be shipped for Python 2.7 on OS X 10.10 (64bit). No C Extension package will be shipped for Python 3 on any platform, until it contains Python 3 compatible C Extensions.

Call for Sponsors & Volunteers

Twisted's lifeblood is volunteers, either personal or corporate. More people power means that we can get more done, more eyes reviewing code, more ideas and solutions that might solve our unique problems.

Glyph has written more about fundraising for Twisted, and why you should do so here. If you would like to support financially directly, you can donate to Twisted through the Twisted Software Foundation/Software Freedom Conservancy, a 501(c)3 non-profit.

If you would like to volunteer your time to the project, please see our contributors documentation.

Until next time,

Amber Brown (HawkOwl)
Twisted Release Manager

Friday, September 4, 2015

Twisted 15.4 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.4, codenamed "Trial By Fire".

Twisted is continuing to forge ahead, with the highlights of this release being:
  • twisted.positioning, the rest of twisted.internet.endpoints, KQueueReactor (for real this time), and twisted.web.proxy/guard have all been ported to Python 3.
  • Trial has been ported to Python 3! This was made possible by a Python Software Foundation grant.
  • Twisted officially supports several more platforms: Py3.4 on FreeBSD, Py2.7/Py3.4 on Fedora 21/22, and Py2.7 on RHEL7.
  • Python 2.6 is no longer supported, and support for Debian 6, and RHEL6 has been removed because of this.
  • Support for the EOL'd platforms of Fedora 17/18/19 has been removed.
  • Twisted has moved to requiring setuptools for installation.
  • twisted.python.failure.Failure's __repr__ now includes the  exception message.
  • 19 tickets in total closed!
You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

Tuesday, August 4, 2015

Twisted 15.3 Released

On behalf of Twisted Matrix Labs, I am honoured to announce the release of Twisted 15.3.

We're marching confidently into the future, and this release demonstrates this:
  • 10+ modules ported to Python 3 (see NEWS for specifics)
  • Twisted Lore has been removed.
  • Twisted no longer releases as 'subprojects' -- there is only one Twisted, long may it reign.
  • twistd now uses cProfile (instead of Hotshot) as the default profiler.
  • 40+ more tickets closed overall.
You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

Sunday, June 7, 2015

Twisted Fellowship 2015: Call for proposals

On behalf of the Software Freedom Conservancy and the Twisted project I'm happy to announce that we're looking for a paid maintainer for the Twisted project.

Funding a software developer to work as a maintainer will help Twisted grow as a project, and enable Twisted's development community to increase their output of innovative code for the public's benefit. Twisted has strict coding, testing, documentation, and review standards, which ensures excellent code quality, continually improving documentation and code test coverage, and minimal regressions. Code reviews are historically a bottleneck for getting new code merged. A funded maintainer will help alleviate this bottleneck, and speed Twisted's development.

You can read more about the 2015 fellowship at https://twistedmatrix.com/trac/wiki/Fellowship2015

Sunday, May 24, 2015

Twisted 15.2.1 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.2.1.

This is a bugfix release for the 15.2 series that fixes a regression in the new logging framework.

You can find the downloads on PyPI (or alternatively on the Twisted Matrix Labs website).

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
HawkOwl