Mailman 3 November 2011 - Twisted

[Twisted-Python] Boston Twisted Sprint: November Edition
by Tenth 09 Nov '11

09 Nov '11

In an attempt to get back into the habit of holding Monthly sprints planned somewhat more in advance, I'm pleased to announce that we'll be hosting another Twisted Sprint at the Smarterer offices on Sunday, November 13th, from about 1:00 to 9:00 PM. If you'd like to attend but aren't on the contact list, let me know! We prefer that sprinters RSVP for food planning purposes (and to receive more detailed instructions about getting into the building, which is locked down on weekends). Thanks, - Dave

2 1

[Twisted-Python] kqueue reactor / ticket #1918
by Tobias Oberstein 08 Nov '11

08 Nov '11

> Is there even a kqueue-based Twisted reactor? > > Yes... ish. More detail here: <http://twistedmatrix.com/trac/ticket/1918>. Hi Glyph, first, thanks as always! The pykqueue 1.3 and 2.0 referenced in that ticket seem to be no longer maintained. On the other hand, Python has built-in support for kqueue since Python 2.6 in the select module. I have adjusted the Twisted kqreactor to select.kqueue and made a package: https://github.com/oberstet/txkqreactor I have tested that reactor successfully with Autobahn WebSockets on FreeBSD 8.2 (running as a VirtualBox x86 VM). When I'm home later, I can check out Mac. Doing the Autobahn tests helped me finding the only non-canonical change required https://github.com/oberstet/txkqreactor/commit/27302b4f643cd3967289ff11d811… Is there any regression/test/stresstest I could run to verify the stuff actually works? The Autobahn WS tests do quite some stuff, but they do not test concurrency and high connections numbers .. Cheers, Tobias

6 14

[Twisted-Python] Global reactor unit tests in the Twisted test suite
by exarkun＠twistedmatrix.com 08 Nov '11

08 Nov '11

Hello, I'd like for us to decide that we will introduce no new unit tests into Twisted's test suite which use the global reactor. These tests require extra complexity in trial - to support the Deferred, to check for left over event sources (incompletely implemented), and to clean up left over event sources (incompletely implemented). The tests themselves are also complicated by the need to clean up those event sources. The tests only exercise functionality against one reactor at a time which leads to additional challenges for buildbot. If a test is for reactor functionality with multiple implementations, the ReactorBuilder-style tests are better. If a test is for implementation details of a particular reactor, I think the necessary parts of that reactor should just be invoked directly - on a new instance. For anything that's not a test for reactor functionality, no reactor should be involved at all (protocol implementations, etc). I've mentioned these ideas to various people at various times, but they don't seem to be catching on, so I'd like to hear why and come to some conclusion about how we're going to write tests in the future. Jean-Paul

12 34

[Twisted-Python] [ANN] txWS 0.6.2
by Corbin Simpson 07 Nov '11

07 Nov '11

Hi, I figured I should announce this, so: txWS 0.6.2 has been made available. You can grab it from PyPI (http://pypi.python.org/pypi/txWS) or from source on Github (https://github.com/MostAwesomeDude/txWS). txWS is a small, single-module library for augmenting Twisted with WebSockets support. Unlike other approaches, txWS has a minimalist philosophy and only provides enough WebSockets machinery to wrap server factories. Notably, txWS does not provide a WebSockets client; we believe that it is harmful to the ecosystem of the Internet to do so. (Additionally, I'm only paid for the server-side stuff.) Feedback (i.e. criticism, insults) is welcome; this library has seen exactly zero usage outside OSUOSL and we hope to continue to contribute to the Twisted world. ~ C. Oregon State University Open Source Lab -- When the facts change, I change my mind. What do you do, sir? ~ Keynes Corbin Simpson <MostAwesomeDude(a)gmail.com>

1 0

[Twisted-Python] Website visual improvements
by Jonathan Jacobs 07 Nov '11

07 Nov '11

Hello, I recently volunteered to make some visual improvements to the Twisted Trac website. I received a lot of good feedback and suggestions from the IRC channel, most of which I took to heart and fine-tuned the changes. I think everyone is generally happy with the changes, I'm happy with the changes and I'd like help with moving things forward to eventually deploy these changes. The branch is currently up for review on Launchpad[1], the merge proposal contains links to some[2] static[3] pages demonstrating the style changes. (It is worth noting that there are some quirks that are as a result of "File -> Save As" in Firefox, I wouldn't worry about these.) I am willing to help with the final deployment of the site changes as well as integrating the new styling with: * the Sphinx documentation and documentation tools; * the Lore documentation (which I see thijs recently did some work on) if that is necessary; * and the pydoctor API documentation. -- Jonathan [1] < https://code.launchpad.net/~jjacobs/twisted-website/visual-design-update/+m…<https://code.launchpad.net/%7Ejjacobs/twisted-website/visual-design-update/…> > [2] <https://dl.dropbox.com/u/4030134/Twisted/Twisted.htm> [3] <https://dl.dropbox.com/u/4030134/Twisted/5047.htm>

5 9

[Twisted-Python] TDD: How to test that a Timeout Exception occurs within the timeout period
by Tom Sheffler 07 Nov '11

07 Nov '11

I am investigating timeout properties of the FTP server module. I have written the following trial test, that when functioning properly will throw a PortConnectionError with a "DTPFactory timeout" I would like to use something like self.failUnlessRaises() but the test is also constructed using defer.deferredGenerator, and it doesn't seem possible to apply the test for an exception to a Generator-based test. This is a useful type of test to be able to write: to assert that an exceptional condition *will* occur within a timeout period. Failure is if the exceptional condition is *never* raised. Suggestions for attacking these types of properties is welcome. FYI: Here is my test, along with the trial directive that if it runs for 60 seconds it is a failure. But first, here is the ERROR traceback that a "successful" run of this test produces. =============================================================================== [ERROR] Traceback (most recent call last): Failure: twisted.protocols.ftp.PortConnectionError: DTPFactory timeout test_ftp_timeouts.TimeoutsFTPServerTestCase.test_FtpControlChannelTimesOut ------------------------------------------------------------------------------- def test_FtpControlChannelTimesOut(self): """Test that after the FTP timeout, the control channel closes itself.""" # Set the timeout to something small, but greater than DTPTimeout self.serverProtocol.timeOut = 15 # Login wfd = defer.waitForDeferred(self._anonymousLogin()) yield wfd wfd.getResult() log.msg("Login Anonymous") # Wait for N seconds, protocol timeOut not fired yet wait1 = defer.waitForDeferred(self.mywait(2)) yield wait1 # Should get here and not fail log.msg("FTP is ok here") # Issue a PASV command, and extract the host and port from the response pasvCmd = defer.waitForDeferred(self.client.queueStringCommand('PASV')) yield pasvCmd responseLines = pasvCmd.getResult() log.msg("PASV responseLines", responseLines) host, port = ftp.decodeHostPort(responseLines[-1][4:]) # Wait for N seconds, protocol timeOut will fire wait2 = defer.waitForDeferred(self.mywait(20)) yield wait2 log.msg("FTP should have timed out and left reactor clean") test_FtpControlChannelTimesOut = defer.deferredGenerator(test_FtpControlChannelTimesOut) # test_FtpControlChannelTimesOut.skip = "skipping" test_FtpControlChannelTimesOut.timeout = 60 # is a real error if it takes this long

2 1

[Twisted-Python] FTP Proxy
by Einar S. Idsø 07 Nov '11

07 Nov '11

Hi, I need to develop a customised FTP proxy for one of our systems, and would like to consider Twisted for this project. The t.p.ftp module seems to have the server and the client more or less ready to use, so I think I am good to start, but just wanted to check with you guys if there are any gotchas that I should be aware of. The system is intended to work as follows: - Users authenticate directly with the proxy, not the backend-server, using a Drupal site for authentication (either through some sort of API-call or by accessing the database directly - to be decided) - The client-part of the proxy connects with a master-user/password to one or more backend FTP-servers running in passive mode behind firewalls. - The master-user has access to the union of all file and directory sets that each individual user has access to. - The root folder presented to the user should contain either all the available hosts, or a virtualised overview of all main directories on all the hosts that the user may access. - Any file access or directory listing will be filtered against an ACL to allow only certain files and directories to be accessed by the user. Is there anything I should be aware of that may be a showstopper or otherwise overly complicate this project by using t.p.ftp, or am I good to go? Will producers/consumers be an appropriate way of transferring data between the client-part and the server-part of the proxy, and is t.p.pcp something I should look into? Any suggestions, especially mentions of/links to similar projects, would be most helpful and appreciated. Cheers, Einar S. Idsř

2 1

[Twisted-Python] Docs HTML suggestion/request for moving between versions.
by Phil Mayers 06 Nov '11

06 Nov '11

Lately, I've taken to finding Twisted docs by writing: twisted.internet.task.Cooperator ...into the Chrome bar'o'magic and hitting return to get the Google hit. This is great, except that the resultant Google hits almost always point me to the 8.2.0 HTML docs. 2nd hit is (currently) 11.0 docs. Obviously it's tricky to force Google to put "current" at the top! But could we maybe have a little <select> dropdown in the top-right that shows the version, and lets you pick a new one and have the page reload? It strikes me that this would be useful in general, for flicking back and forward between two versions of the docs page for a given method/API.

3 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 05 Nov '11

05 Nov '11

1 0

[Twisted-Python] adbapi / Postgres : LISTEN/NOTIFY and RealDictCursor
by Tobias Oberstein 03 Nov '11

03 Nov '11

I am using Twisted adbapi, psycopg2 and PostgreSQL which works really great. Now, there are 2 features I'd like to use in this context, which I've been using before without Twisted. I'm missing those. Would be glad for any hints .. If there is work to do to make those features happen, and it's doable for non-Twisted-gods, then I'd also be willing to invest time/effort ... Thanks, Tobias LISTEN/NOTIFY ============ Is a Postgres>=9 feature which allows one to asynchronously notify a database client from _within_ the database (i.e. from pgPLSQL). http://www.postgresql.org/docs/9.1/static/sql-notify.html http://initd.org/psycopg/docs/advanced.html#asynchronous-notifications On the Python side, it'll do a select() on the TCP underlying the connection to Postgres. adbapi assumes DBAPI, and above is a non-standard extension. Now, what would be a sensible approach to use that with Twisted? Own thread? How do I integrate that into a standard Twisted application service hierarchy? Or somehow extend adbapi? psycopg2.extras.RealDictCursor ======================== We have a convention that all database access must be exclusively via stored procedures with all IN and all OUT parameters named. Outside the context of Twisted, we use that in conjunction with # Real Dict Cursor (returns a Dict which can be referenced via named bracket access, or offset) dict_cursor = db.cursor(cursor_factory = psycopg2.extras.RealDictCursor) dict_cursor.execute("SELECT first_name, last_name, email FROM user.get_user(%s)", (user_id,)) my_row = dict_cursor.fetchone() => {'first_name': 'John', 'last_name': 'Doe', 'email': 'john(a)doe.com'} Is there a way to do something like this when using self.dbpool.runInteraction or otherwise? Setting the "cursor factory" to be used? On the other hand, the txn given to the fun provided with runInteraction already is somehow dressed up, right?

6 10