Yesterday I had the chance to talk with Jack Moffitt and Christopher Zorn of Chesspark about their company, their vision, and how Twisted played an integral role in their application and helped them to meet their goals.
The interview is on the site and the feed is available here.
We're still experimenting with podcasting and feed enclosures, so please be patient with us! If you have any issues, the media files are available at archive.org.
Tuesday, February 26, 2008
Monday, February 25, 2008
Twisted Talks at PyCon 2008
There will be two Twisted-related talks at PyCon this year:
"Tahoe: A Robust Distributed Secure Filesystem" -- Brian Warner, of Twisted Foolscap fame, discusses his company's distributed file system, built on Twisted. Tahoe is free, secure, decentralized, fault-tolerant filesystem. This filesystem is encrypted and spread over multiple peers in such a way that it remains available even when some of the peers are unavailable, malfunctioning, or malicious.
"Adventures in Stackless Python/Twisted Integration" -- Andrew Francis will be discussing the Web Service Business Process Execution Language, particularly he integrated Stackless Python and Twisted.
"Tahoe: A Robust Distributed Secure Filesystem" -- Brian Warner, of Twisted Foolscap fame, discusses his company's distributed file system, built on Twisted. Tahoe is free, secure, decentralized, fault-tolerant filesystem. This filesystem is encrypted and spread over multiple peers in such a way that it remains available even when some of the peers are unavailable, malfunctioning, or malicious.
"Adventures in Stackless Python/Twisted Integration" -- Andrew Francis will be discussing the Web Service Business Process Execution Language, particularly he integrated Stackless Python and Twisted.
Wednesday, February 20, 2008
RSS Flood Apology
Hey all,
A quick note to apologize for the RSS updates you'll be getting that are full of old news. I just finished sync'ing Twisted news to the labs blog (and tagging them with "news"). There weren't many items, so the pain of the RSS flood should be minimal. I offer apologies nonetheless :-)
The upshot of all this is that we now have a couple different RSS feeds for your viewing pleasure:
You can also view the news items in either of these three places:
A quick note to apologize for the RSS updates you'll be getting that are full of old news. I just finished sync'ing Twisted news to the labs blog (and tagging them with "news"). There weren't many items, so the pain of the RSS flood should be minimal. I offer apologies nonetheless :-)
The upshot of all this is that we now have a couple different RSS feeds for your viewing pleasure:
- Items of Twisted Interest (labs.twistedmatrix.com): http://feeds.feedburner.com/TwistedMatrixLaboratories
- News Items: http://feeds.launchpad.net/twisted/announcements.atom
You can also view the news items in either of these three places:
- http://twistedmatrix.com/trac/wiki/ArchivedNews
- https://launchpad.net/twisted/+announcements
- http://labs.twistedmatrix.com/search/label/news
Sunday, February 17, 2008
Twisted's filepath Module
Glyph recently blogged about some of the buried treasure in Twisted, the filepath module in particular. Since working at Divmod, I've made use of this module quite a bit, and thought I'd share some of the features that I've found most useful (and intuitively nice to use).
Assuming you've got Twisted installed, let's fire up a python interpreter and import the FilePath class and instantiate it with /tmp:
>>> from twisted.python.filepath import FilePath
>>> tmp = FilePath('/tmp')
Now let's test some basic operations:
>>> tmp.exists()
True
>>> bogus = tmp.child('bogus')
>>> bogus.exists()
False
Isn't that nice? I love not having to import and use the os.path.exists function; it's a method on the object representing a file or a path, as it should be. I also enjoy the convenience FilePath offers when it comes to creating paths:
>>> mydir = tmp.child('notbogus').child('anotherdir').child('mydir')
>>> mydir.exists()
False
>>> mydir.makedirs()
>>> mydir.restat()
>>> mydir.exists()
True
We had to call restat() so that the object would check the file system again (since we just made some changes). Now, for some files:
>>> for filename in ['test1.txt', 'test2.txt', 'test3.tst']:
... mydir.child(filename).touch()
...
>>> mydir.listdir()
['test1.txt', 'test2.txt', 'test3.tst']
And if you don't believe that, we can switch to shell:
lorien:oubiwann 20:51:58 $ ls -al /tmp/notbogus/anotherdir/mydir/
total 0
drwxr-xr-x 2 oubiwann wheel 170 Feb 17 20:51 .
drwxr-xr-x 3 oubiwann wheel 102 Feb 17 20:46 ..
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test1.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test2.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test3.tst
Reading and writing operations are the same as usual:
>>> myfile = mydir.child(filename)
>>> myfile.isdir()
False
>>> myfile.islink()
False
>>> myfile.isfile()
True
>>> fh = myfile.open('w+')
>>> fh.write('do the usual thing')
>>> fh.close()
>>> myfile.getsize()
18L
>>> myfile.open().read()
'do the usual thing'
But you can use shortcuts like this, too:
>>> myfile.getContent()
'do the usual thing'
>>> myfile.setContent('and now for somethibg completely different...')
>>> myfile.getContent()
'and now for somethibg completely different...'
Let's get the path and recheck the file size, just to make sure:
>>> myfile.path
'/tmp/notbogus/anotherdir/mydir/test3.tst'
>>> myfile.restat()
>>> myfile.getsize()
45L
Or, again from shell:
lorien:oubiwann 20:52:06 $ ls -al /tmp/notbogus/anotherdir/mydir/
total 8
drwxr-xr-x 2 oubiwann wheel 170 Feb 17 20:58 .
drwxr-xr-x 3 oubiwann wheel 102 Feb 17 20:46 ..
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test1.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test2.txt
-rw-r--r-- 1 oubiwann wheel 45 Feb 17 20:58 test3.tst
Hmm... for some reason, I really like this file and want to keep it. What do I do?
>>> docs = FilePath('/Users/oubiwann/Documents')
>>> myfile.moveTo(docs.child('special_file.txt'))
That's it! Nice, eh? Of course, we're careful, so we check to make sure it happened:
>>> myfile.exists()
False
>>> newfile = docs.child('special_file.txt')
>>> newfile.exists()
True
>>> newfile.path
'/Users/oubiwann/Documents/special_file.txt'
>>> newfile.getsize()
45L
>>> newfile.getContent()
'and now for somethibg completely different...'
To see more, check out the source, or at least do dir(FilePath) to see what other goodies this class has to offer, and enjoy :-)
Assuming you've got Twisted installed, let's fire up a python interpreter and import the FilePath class and instantiate it with /tmp:
>>> from twisted.python.filepath import FilePath
>>> tmp = FilePath('/tmp')
Now let's test some basic operations:
>>> tmp.exists()
True
>>> bogus = tmp.child('bogus')
>>> bogus.exists()
False
Isn't that nice? I love not having to import and use the os.path.exists function; it's a method on the object representing a file or a path, as it should be. I also enjoy the convenience FilePath offers when it comes to creating paths:
>>> mydir = tmp.child('notbogus').child('anotherdir').child('mydir')
>>> mydir.exists()
False
>>> mydir.makedirs()
>>> mydir.restat()
>>> mydir.exists()
True
We had to call restat() so that the object would check the file system again (since we just made some changes). Now, for some files:
>>> for filename in ['test1.txt', 'test2.txt', 'test3.tst']:
... mydir.child(filename).touch()
...
>>> mydir.listdir()
['test1.txt', 'test2.txt', 'test3.tst']
And if you don't believe that, we can switch to shell:
lorien:oubiwann 20:51:58 $ ls -al /tmp/notbogus/anotherdir/mydir/
total 0
drwxr-xr-x 2 oubiwann wheel 170 Feb 17 20:51 .
drwxr-xr-x 3 oubiwann wheel 102 Feb 17 20:46 ..
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test1.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test2.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test3.tst
Reading and writing operations are the same as usual:
>>> myfile = mydir.child(filename)
>>> myfile.isdir()
False
>>> myfile.islink()
False
>>> myfile.isfile()
True
>>> fh = myfile.open('w+')
>>> fh.write('do the usual thing')
>>> fh.close()
>>> myfile.getsize()
18L
>>> myfile.open().read()
'do the usual thing'
But you can use shortcuts like this, too:
>>> myfile.getContent()
'do the usual thing'
>>> myfile.setContent('and now for somethibg completely different...')
>>> myfile.getContent()
'and now for somethibg completely different...'
Let's get the path and recheck the file size, just to make sure:
>>> myfile.path
'/tmp/notbogus/anotherdir/mydir/test3.tst'
>>> myfile.restat()
>>> myfile.getsize()
45L
Or, again from shell:
lorien:oubiwann 20:52:06 $ ls -al /tmp/notbogus/anotherdir/mydir/
total 8
drwxr-xr-x 2 oubiwann wheel 170 Feb 17 20:58 .
drwxr-xr-x 3 oubiwann wheel 102 Feb 17 20:46 ..
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test1.txt
-rw-r--r-- 1 oubiwann wheel 0 Feb 17 20:51 test2.txt
-rw-r--r-- 1 oubiwann wheel 45 Feb 17 20:58 test3.tst
Hmm... for some reason, I really like this file and want to keep it. What do I do?
>>> docs = FilePath('/Users/oubiwann/Documents')
>>> myfile.moveTo(docs.child('special_file.txt'))
That's it! Nice, eh? Of course, we're careful, so we check to make sure it happened:
>>> myfile.exists()
False
>>> newfile = docs.child('special_file.txt')
>>> newfile.exists()
True
>>> newfile.path
'/Users/oubiwann/Documents/special_file.txt'
>>> newfile.getsize()
45L
>>> newfile.getContent()
'and now for somethibg completely different...'
To see more, check out the source, or at least do dir(FilePath) to see what other goodies this class has to offer, and enjoy :-)
Wednesday, February 13, 2008
Simple Python Web Server
Corey Goldberg blogged about making simple web interfaces. I was going to comment with another example of a simple web server, but his blog kept spitting back errors, so I'll post it here instead.
Corey's code was based on
Of course, it requires a bit more than just the standard library. ;) This has a couple extra nice features though - it does logging, for example, and it can be trivially daemonized, or run under the profiler or the debugger, or chrooted or as a different uid/gid. Plus the Foo resource can easily be relocated into a "real" program, since it is entirely distinct from the HTTP server part of the code.
There's an even simpler way to express this though, as an rpy instead of a tac:
Just drop this into a directory and run a Twisted web server pointed at it (eg,
Corey's code was based on
BaseHTTPServer, here's another way to do this (as a Twisted tac file):
from twisted.web import server, resource
from twisted.application import internet, service
class Foo(resource.Resource):
def render(self, request):
return "hello world"
root = resource.Resource()
root.putChild("foo", Foo())
application = service.Application("Tiny Web")
internet.TCPServer(8080, server.Site(root)).setServiceParent(application)
Of course, it requires a bit more than just the standard library. ;) This has a couple extra nice features though - it does logging, for example, and it can be trivially daemonized, or run under the profiler or the debugger, or chrooted or as a different uid/gid. Plus the Foo resource can easily be relocated into a "real" program, since it is entirely distinct from the HTTP server part of the code.
There's an even simpler way to express this though, as an rpy instead of a tac:
from twisted.web.resource import Resource
class Foo(Resource):
def render(self, request):
return "hello world"
resource = Foo()
Just drop this into a directory and run a Twisted web server pointed at it (eg,
twistd web --processor rpy=twisted.web.script.ResourceScript --path bar/). You can even change the code without restarting the server if you do it this way. rpys can be convenient for one-off hacks, but clearly it's not suitable for anything larger scale than that.
Wednesday, December 19, 2007
The future of Twisted releases
Early in 2007 the other Twisted developers in Cambridge and I had a meeting about what we should do about Twisted releases. Things had been a bit chaotic since the split of Twisted into multiple components, and releases were still done on an irrelegular basis.
At the meeting we decided that a time-based release would be ideal. Given that, we would move to a time-based versioning scheme somewhat similar to Ubuntu's; the next release will be Twisted 8.0, which is the first release in 2008. That is, the version numbers will be (year - 2000) point (incremental per-year number starting at 0) point (patch level, if any, starting at 0).
We also discussed our deprecation policy, which we still haven't finished documenting, and decided to make it explicitly time-based as well. An API that is to be deprecated will be marked as such and will be supported for N months or years before being a candidate for deletion. We still need to come up with a convention for how the deprecation is presented to the user, and what information the deprecation message will contain, not to mention the value of N.
But none of these changes were possible; releasing was too tedious a process, fraught with numerous hiccups and changes to the procedure on every release. It would be extremely stressful to try to do this with any rapid regularity, leaving the release manager an insane fool constantly muttering about tarballs and web servers.
So while we knew we needed to switch to a more automated release system, the project had stalled before it even got on its feet. Fortunately, Jonathan Lange visited last month and "inspired" us to organize this project. We sat in the board room with a whiteboard and listed everything that needed done in order for regular releases to be possible. Since then, progress has been slow but steady; at least once a week someone is committing changes towards these goals (thank you, therve).
But we need some more help. If you know how to write software test-first, then please help out. Come join #twisted or contact me to see how you can help. Hopefully 2008 will see many Twisted releases.
At the meeting we decided that a time-based release would be ideal. Given that, we would move to a time-based versioning scheme somewhat similar to Ubuntu's; the next release will be Twisted 8.0, which is the first release in 2008. That is, the version numbers will be (year - 2000) point (incremental per-year number starting at 0) point (patch level, if any, starting at 0).
We also discussed our deprecation policy, which we still haven't finished documenting, and decided to make it explicitly time-based as well. An API that is to be deprecated will be marked as such and will be supported for N months or years before being a candidate for deletion. We still need to come up with a convention for how the deprecation is presented to the user, and what information the deprecation message will contain, not to mention the value of N.
But none of these changes were possible; releasing was too tedious a process, fraught with numerous hiccups and changes to the procedure on every release. It would be extremely stressful to try to do this with any rapid regularity, leaving the release manager an insane fool constantly muttering about tarballs and web servers.
So while we knew we needed to switch to a more automated release system, the project had stalled before it even got on its feet. Fortunately, Jonathan Lange visited last month and "inspired" us to organize this project. We sat in the board room with a whiteboard and listed everything that needed done in order for regular releases to be possible. Since then, progress has been slow but steady; at least once a week someone is committing changes towards these goals (thank you, therve).
But we need some more help. If you know how to write software test-first, then please help out. Come join #twisted or contact me to see how you can help. Hopefully 2008 will see many Twisted releases.
Monday, December 10, 2007
Twisted BuildBot Upgrade
I updated Twisted's buildbot installation to 0.7.6 a few days ago. This after investigating setting up a buildbot for the PyPy project. Since I had a chance to look at 0.7.6 in a fresh environment for PyPy, I saw enough that it looked like an upgrade for Twisted would be safe. It was, more or less. There are still some problems (certain actions cause slaves to become lost or hung, the permissions aren't fine-grained enough, and there are some strange problems reporting history which I don't fully understand, some links are generated wrong). However, the new HTML status views which are available in 0.7.6 may be worth this. After I got the basics working, I exploded the "one box per builder" view into this one. This is approaching the view we've wanted for a long time, and it only took about an hour of hacking to accomplish it (unfortunately the code is all untested, so it's basically garbage, but it'll serve for now I suppose).
An improvement which would be nice is to have builds aligned vertically so that each column represents a particular build across all the builders. It would also be nice if the default view only included trunk views, and if it were easier to adjust the number of columns, and if there were more useful links in the boxes, and if the "current status" area had a bit more information, and if the builder names linked to the builder information pages.
However, it's progress.
An improvement which would be nice is to have builds aligned vertically so that each column represents a particular build across all the builders. It would also be nice if the default view only included trunk views, and if it were easier to adjust the number of columns, and if there were more useful links in the boxes, and if the "current status" area had a bit more information, and if the builder names linked to the builder information pages.
However, it's progress.
Subscribe to:
Posts (Atom)